I've seen mainstream websites and random people alike come up with elaborate theories about height. Sometimes they try to link height with childhood confidence, or conflate tallness with authority because our parents and teachers were taller than us.
The Occam's Razor explanation is much more succinct. If you stand someone with dwarfism next to a person with gigantism, the latter is more intimidating and therefore taken more seriously. That's it. It's the same reason women complain about not being taken as seriously as men, because females are smaller and weaker on average. The only difference is that this society gets mad if you don't take the average woman seriously.
