I actually have somewhat the same issue with my novel where readers shout "there should be a harem tag" or "please no harem"
Though I really don't think about it much as I on;y read the comments to see what they think about the novel. Since we authors don't have control over our own tags or even remove the incorrect ones, i can't really do anything about what tags have been put there.
To me, when i deem a novel i read a harem novel, it means that the MC has trouble choosing between the various choices or straight out chooses more than one. (that's my own judgement of considering one a harem novel anyway) Since mine does not have that problem... (at least I haven't really decided yet?) Even if there are a generous amount of female characters, the MC's focus is completely on something else.
I don't know if this is going to be helpful to you or not but what I do is just write what I already planned while ironing out how to get there. Remember your original thoughts on what is about to happen and do not allow the comments to sway your own opinion because no matter what, some people will not like what you decide. Don't let the comments influence what you write ESPECIALLY the main choices/parts. I'm not saying don't listen to the comments but compromising with "what they are asking for" is a good way for one to lose the intended way.
Take note that most of the people reading your novel may not even be actively commenting. or if they do comment, it's the usual xp sect. Always remember that you have more readers in the dark than those actually trying to impact the way you tell your story. If your getting swayed by the comments, you can always stop reading for a few chapters and just come back at it afterwards. complete the thoughts you have on your mind goes because you are first and foremost the creator of the fiction they are enjoying. the fact that they are following your novel means that they like the content you started in the first place.
Just my two cents. hope it helps