I think, I have said it earlier also and I'll say the same again. We can't please everyone. We are not supposed to please everyone. We are not here to please everyone. Everyone has a different perspective and liking. I don't exactly remember if I have read it in the reviews or the comments in your story that a bunch of people do not like the protagonist wearing specs. I personally love people with specs. I find it really cool.
You get what I am trying to say? You can't really be giving up on your story because some people do not like it the way it is. That way, you are snatching the read from the people who are actually looking for the same stuff. You are snatching their chance of reading who are in need of it. Believe it or not, sooner or later, one always finds people, who can see the things the way one wants them to see - the story and the characters here.