as much I love X4, I can't help but wonder what the other character was doing throughout the entire game.
X4 is confusing...(
Downtime)
But yeah, that's the big story contradiction we wound up with in X5. X4 in itself is made so that one hero or the other takes care of things alone, yet the future storyline clearly takes both scenarios into account.
thats what we mean when we say "Zero" was supposed to be X. the design was supposed to be X.
Well, "supposed to" implies that Inafune was in charge, which he wasn't. But, yeah, more or less. The name "X" would have gone to whatever design was chosen. I don't think that was decided when they were actually doing the designs, though, as Inafune simply uses the term "new MegaMan" in MMXOCW.
While Inafune personally envisioned Zero as the "new MegaMan", he knew all along that he'd be submitting it to his boss as a sub-character. It allowed him more freedom, creating an original design and an overall "darker" hero type, without others (his boss in particular) criticizing it for not fitting the "MegaMan" image. Hence, Inafune submitted Kaji's "new MegaMan" (which is what Inafune told him he'd be drawing from the start) to his boss as the main character.
Noteworthy here is that the whole "Zero was Inafune's new MegaMan, but wasn't very MegaManly" thing refers to personality as well, not merely appearance. Inafune refers to a "hardcore" personality for Zero versus the "nice guy image" which the new MegaMan would inevitably be stuck with a lot within the pages of MMXOCW.
Basically, I see the whole thing as this: When they were making the X-series, Inafune would have liked to make a game with the general atmosphere of the Zero-series.
hoo boy, Hypershell's gonna have a field day here. 
I can understand what you mean, but separating their stories completely is not the answer either. If anything, what X5 and 6 did. both playable in the same story, just you only pick one character. Separating them is a bad idea because the story states that they are good as a team. they are the best of partners, as Zero puts it in Zero 1. So giving them completely separate story lines in a game really contradicts that.
My issues with taking the hero solo are more story-driven than gameplay-driven; many of my thoughts mirror what you posted.
That said, yeah, I *LIKE* the Scramble Change. But I understand how some would consider that too easy. Thing is, the main series kinda treats it like a tack-on (albeit necessary due to the "2 out of 3" character selection). Xtreme2, changing between your ONLY two characters, often plays off of their individual strengths and the benefits of teamwork between them. The second fortress boss and Sigma's final form are ideal examples. You can also use Zero to "sequence break" the power-ups with X and better take advantage of the power-up system (assuming one has no attachment to Hadouken/Shoryuken, anyway), meanwhile X contributes greatly when you come to the occasional stage enemy at a difficult attack angle. There is a strong benefit to tag-teaming with them, and yet it is never an outright necessity (Sigma less so, though IIRC you can simply wait for him to switch); an excellent balance.
Another idea would be to introduce the duo in a 2-player simultaneous game.
YES!!!Seriously, how many of us HAVEN'T been fantasizing about that for damn near the entire series? Pre-release translation errors with both X7 (double hero) and X8 (double attack) didn't help any.
Better yet, do this with an X4 remake, that way we'd resolve the above issue.