Loading times aren't always just about the CD or the CD drive speed. The console needs time in order to pull the data from the correct part and load it into the Playstation's RAM. With a cartridge, there aren't load times because the console can access the data instantaneously.
It's like if you're looking for something in your house. Imagine that a cartridge would be just like an organized shed in your backyard and you can find anything because you have everything organized and it's within arm's reach. With a CD, imagine a normal house that can have several rooms, but you can put more things within the house verses the cartridge shed. You will actually have to get up, remember where you put it, go to that room, look and acquire it, then bring it back to where you were.