How far back do these pre-OoT sages go? Even in OoT the Master Sword sleeps inside of the Temple of Time. And in my recent replay through OoT I got the distinct impression there was indeed a previous hero of time.
SSB Brawl notes the Sages forging the Master Sword in the TP Sage trophy. Wind Waker also establishes a connection between the Sages and the Master Sword's power to repel evil. So the Sages go at least as far back as the Master Sword does, but since we don't know how far that is, it's a giant question mark. The Master Sword seems to be fairly well-known even in OoT, though, seeings how Navi recognizes it.
Ganon's failure, as hinted by TP not having the Gerudo, would lead to the Gerudo's disappearance. It's a mystery where they went, but some we have no exact confirmation of all of them being gone, they could re-establish themselves.
The Gerudo Desert in Twilight Princess is the location of the Arbiter's Grounds, where the Mirror of Twilight is kept. The Sages also reside there, and of course this is the same area where Ganondorf's execution was to be held. So, presumably, the Gerudo cleared out, and the royal family's reach extended there over the course of the OoT generation, before Ganondorf was executed. Whether they left of their own accord or were removed forcibly isn't known, but considering what the Mirror of Twilight was there for, it's likely that if any Gerudo were captured, they were assimilated into the Twili.
Him coming twice from the Gerudo isn't odd, it's just that in neither case, there's a recollection of the former. Which there would have to be if the Gerudo never vanished between FSA<>OoT. The fact that they gave birth to the King of Darkness, twice, and with Twinrova being the mother still... you'd think they'd remember...
What do the Twinrova possibly have to do with FSA? They appear only in OoT, MM, and Oracles. And in Majora they're actually friendly (granted that's Termina and not Hyrule).
Considering the Gerudo only have one male to them born every century, your argument could be pretty well applied no matter which one comes first. Furthermore, the FSA Ganon isn't all that infamous. The Gerudo hate him for violating their laws and entering the forbidden pyramid, but what happened to him after that isn't general knowledge. Ganon doesn't make his presence known until after Vaati is defeated, very near the end of the game. Hyrule as a whole isn't aware of what happened the way they are in OoT.
If he indeed made his way into the Sacred Realm, by however the Four Sword got there, there's the full Triforce right there.
But this is why the lack of a Triforce mention in FSA would be so odd in the first place. The ONLY way FSA is linked to the Sacred Realm is if the Four Sword Sanctuary, where Vaati and later Ganon are sealed, is part of it. But if that's the case, and Ganon had access to it all this time (shadow Link emerges from there), why stop at releasing Vaati? It doesn't make any sense.
With Ganon dead as of WW, and revived as of FSA, it's very much a matter of convenience and the path of least resistance.
That's just it, Ganon being revived in FSA is fanon. There IS no revival mentioned in the game. There is nothing to indicate that FSA Ganon performed any acts of evil before he entered the forbidden Gerudo pyramid, the "ancient demon" likely refers to the power of the trident, which is the focus of Ganon's power in FSA anyway.
One of the six maidens dismisses the name "Ganondorf" as being merely a desert nomad, unable to understand why forest monsters would worship him. They don't recognize the shorter name "Ganon" at all. Clearly, the FSA Ganondorf has not been previously established as the king of darkness. So the argument of FSA Ganondorf being a revival of a previous Ganondorf is extremely weak, if even the maidens don't know him. To add to this, the Gerudo conversations indicate that the theft of the trident, the furthest back "evil act" to be associated with FSA Ganon, is recent. Some of them claim to not know what his "plans", present-tense, are. Others believe him to have died failing to reach the pyramid. The road to the desert pyramid was blocked following Ganondorf's intrusion, and you have to speak to the guards in order to pass.