One should also not underestimate X's image compared to Zero's. X and Zero are both Combat Reploids of the highest and deadliest caliber, but they have both very different images, particularly in the public eye.
Zero is looked at as a monster, a demon with unparalleled combat skill who kills and slaughters his enemies without hesitation or compassion. He projects a cold, hard, unfeeling personality, and definitely strikes a VERY imposing silhouette, with his crimson colored armor, broad shoulders, long blonde mane, and spiked helmet.
He is also of shady origins, much like X, being a highly advanced relic of a bygone century, however with no known creator, and a bloodstained discovery, in which he was a rampaging Maverick. Of course, that last part is not entirely common knowledge, but by the time of the Elf Wars, where all on Zero was laid bare with the discovery of his origins, I'm sure it wouldn't leave a good taste in people's mouths, much less in Zero's own.
X on the other hand, has pretty much the opposite effect. He acts and emotes on the same level as a a Human, is known to be very friendly, very compassionate and caring for others, and is a serene blue color, with a much sleeker, rounded design. He is known to have been created by the Father of Robotics, Dr. Light, and is known for disliking violence and having a tendency to hesitate in combat, or try to negotiate whenever possible.
His combat style also, leads to a difference in view. whereas Zero is a close ranged melee fighter who is frightening with a sword and is right in the middle of the blood soaked front lines, X is a long distanced shooter, who is distanced from the death.
Then there is the fact that Zero has been used various times as the object of catastrophe. He was the motivation behind the Eurasia colony drop that nearly decimated the Earth, which resulted in a virus with an eerily similar energy reading as him, (which also took his form) and the cause and motivations behind the Nightmare Virus and Gate's insanity. His body was used to create the Devil Reploid, Omega, and the Dark Elf, both of which arose in response to a way to deal with Mavericks caused by the Sigma Virus, which was discovered to have come from Zero all along, and spread by him, also making him the reason for Sigma's going Maverick and becoming the worst terrorist madman the world ever saw.
It's pretty obvious why people would look at him kinda funny, when pretty much because of him, 60% of Humanity and 90% of Reploids were wiped out by an apocalyptic war that also ravaged the planet and left most of it a wasteland.
X meanwhile, through all this, became a politician and was beloved by all for his personality and heroic exploits, and therefore once the war was over, everyone united under him and became the Leader of what was left of the world.
I mean, sure, he would have preferred Zero stay along with him, but he doesn't have NEARLY the same issues Zero had after all was said and done.