I can tell you why, especially regarding the adventurer's guilds in Japanese stories. 1. Dungeons and Dragons is a very popular RPG in Japan, and has influenced most RPGs on consoles and PCs. 2. It's a very convenient way or system for the main character to get a job, earn money, and trade supplies (so you don't have to think too hard on how he/she survives in this new world). 3. They copy from each other.
As for the medieval setting, it's mostly because they just copy off aesthetics from each other, and they don't really conduct proper research. Not that they need to - why would readers care about historical accuracy in a fictional fantasy? As long as it serves the story, the setting can be however convenient they want it to be. Nobody wants to read about pollution or unhygienic stories. You're reading a fantasy story for entertainment here, not a history textbook. Also...everything can be explained away by magic.
I also have a few gripes about urban cultivation stories. Why the hell would you use an urban setting if you're going to have everybody commit murder blatantly in broad daylight, and not have the characters bound by the laws of modern society? I know it's fiction and fantasy, but the setting has to serve the plot, not be dismissed entirely. I'm not expecting realism or accuracy (for example, I don't need the writers to know specific legal codes), but it kind of defeats the purpose of setting the story in a modern, urban world if you are going to ignore the modern, urban laws that should come with the setting. While you probably can expect people to not know a single thing about the harsh reality of medieval history, you can't expect readers to not know that murder has been criminalized in modern, urban society...
