- Edited
It's not the matter of realism. I don't think the lack of realism is the problem - in fact, harems are realistic. Look at the Middle East, or even in history. Nobles had concubines. Sultans have harems of women. Comparing harems to fantasy or flying pigs or magic is just absurd because harems actually do exist in reality. So if you think the issue people have with harems is "lack of realism", you're sorely mistaken.
The problem with harem is the sort of values inherent within it. I wouldn't go so far as to say it "promotes" them, but it seems to naturalize the attitude that women are objects or trophies to be added into a man's collection. The harem genre treats women (or men as the case may be, depending on who the protagonist and gender) as objects and with no respect at all. Like these characters exist simply to love the main character. It's degrading and insulting to the poor characters. It's not the matter of realism, but rather the degrading depiction of men/women. Most of all, it's simply poorly written and horribly executed. Women fall for a single guy for no reason other than because he's nice?
By the way, the affair path is considered harem. Harem simply refers to any or all stories where the protagonst has multiple relationships with members of the opposite sex (or same sex, depending on the genre). The affair path is yet another reason why people hate harems. Because the protagonist is, quite simply, a scum. Harem doesn't necessarily mean the main character ends up with multiple women by the end of the story. There's a reason why manga like Nisekoi are considered harem. The harem genre simply refers to a story where a single person is surrounded by multiple (usually 3 or more, 2 is just love triangles and drama, and isn't considered harem) characters who are romantically in love with him/her. Whether he/she ends up with only one of them is irrelevant.
Now if you have a series like Kimi no Iru Machi, that's not exactly harem because it's not like there are multiple women in love with Haruto at once, unlike in Nisekoi where you have 4 or more heroines competing for Raku's affections. But Kimi no Iru Machi is not an affair story either - just because Haruto dates Asuka before getting back with Yuzuki doesn't mean he had an affair. It's a realistic portrayal of life where normal people (unlike harem protagonists) have several relationships throughout their life. Most people will form relationships, break up, and then date another person again. That's not an affair. However, the affair that SecretWolf mentions is precisely the tripe produced out of harem novels where all the women in the story want to sleep with the male protagonist because he...I don't know. Sometimes he's not even a chad but just an indecisive scum who goes with the flow. Yet they are all attracted to him for some reason. Another example of poor execution. It's not realism, but if the readers can't be convinced that the girls like the guy according to the rules established by the world in that story, then they will find the whole thing ridiculous.
Now, will men who read harem novels suddenly turn into cheating tyrants who abuse women or cruelly lead them on? I hope not. Such a claim is as dumb as nonsensical claims about gamers being more prone to murdering people with guns just because they play shooters. Or increase in violence just because people watch action movies. However, when you start seeing readers' comments where they call women "sluts" for not being virgins (and thus have no right to be part of the harem) or accusing men of being "simps" just because they respect women and accede to their requests, you get the feeling that there is some sort of negative influence at play here. As in, the normalization of insidious attitudes that are far subtler than "he's going to become a murderer because he plays Call of Duty!"