Harems should only be there if the members value add to the story or if your story is primarily a male wish fulfillment fantasy. Harem members should contribute to story progression and/or MC development. Otherwise you could end up with one dimensional characters whose only role is to be the MC's harem member. It looks really shallow outside of a male wish fulfillment story.
Also keep in mind that a harem doesn't mean that the MC can't have more than one sexual partner. Having sex with a female is different from making them part of a harem. In the same way that not everyone who an MC fights would be considered part of their rivals list.