Just a comment on the main topic of this thread.
I think that what the poster meant of an "operating system" is something more like a "Window Manager". Now, for windows there is something called SphereXP (
http://www.hamar.sk/sphere/screenshots.htm) which uses some kind of 3D. Then, there is the Sun's Java desktop that was presented some time ago that promised some kind of 3D functionality.
Now, that is quite possible. Just create a normal linux distribution CD, and then, over X11 (the graphical interface) you could make a window manager (something as KDE or GNOME or XFCE or Enlightment or Explorer in MSwindows) that was built on CrystalSpace.
Of course one of the main issues with 3D driven user interfaces is the functionallity of it, I am still waiting to see a 3D interface which really provides USEFUL functionality and is not only eye-candy CPU-crunching 3D graphics.
About the person that said it would be a waste of resources, there is not too much problem on that, you just need to make the window manager thread to a low priority and the OS kernel will do the rest.
Oh!, the other issue with 3D engines is the input. So far, the mouse only gives you movement on 2 dimensions, only mice with scroll wheel can have some sort of 3D. And, of course the inherent problem that our current output devices are 2D (monitors =op).
Hope someone is enlighted with this.