I dont have time to baby
Lemme try.
While you can choose to only see one "Point of View" in a Virtual Boy, you must remember that these games were made specifically with the purpose and the ability to let you play a game using two "Points of View".
While a Polygonal game has an unfixed camera which your brain can use to determine depth with clue such as depth and layering of object, the 2D games of a Virtual Boy fail considerably at this and depend on the use of the stereoscopic vision to let your brain determine depth.
These are polygonal games in motion. Your brain can realise a certain amount of depth from how things change as they move.


This is a non-polygonal game using only one Point of View. Sans things like Mode-7 Scaling and rotating, you don't have much or any clues as to depth beyond the shading or static size of objects.



These games even have a vast array of shades to let your brain know what is darker or lighter. The Virtual Boy has none of that.
So, what it comes down to is a very simple monochrome game with simple graphics that, through stereoscopy, lets your brain realise depth. And this is its main and only focus. When you remove one of those Points of View you are removing half your visual experience. You are now looking at an extremely off center 2D gameboy game that seems oddly obsessed with isometrics.
For an audio experience. Go find some Beatles Music(Pre-White Album) and remove one of the channels.
Well, gee, that's perfectly fine. Isn't it? Taking a stereo sound and just observing one channel sure isn't detrimental to the experience at all.