Setting up an augmented-reality experience requires more than just putting a collection of virtual objects into the world. The user’s experience is important: how do the visuals, the controls, and the sound contribute to the overall impression that we want to convey?
For the HoloLens Games Institute virtual tour app, I wanted the visual elements of the app to evoke a futuristic, high-technology impression. Since the Games Institute is looking toward “understanding the future of games,” a futuristic style to the tour fits well with the tour’s goals.
To help evoke a futuristic impression, I wrote a shader for the HoloLens-generated environment mesh that is intentionally reminiscent of Star Trek holodeck technology. (I’m thinking here of the Next Generation holodeck, but using yellow for the grid lines seemed a bit dated. These days, high technology seems more like a blue thing—blue LEDs, Blu-Rays, blue web page themes, etc.)
It is interesting to note that what I’m doing here is the opposite of what Steve Jobs did for the iPhone: rather than striving to make the technology invisible, I pushed to make it visible, and visible in a way that would quickly evoke an impression of optimistic futurism and high technology. What Steve Jobs strove for was a zen-like simplicity, and an invisibility of underlying means—what I’m working toward here is visibility of underlying means, a presentation of underlying reality in a way that makes sense to the user.
The shader itself is not too complex: it uses world-space coordinates to produce blue-coloured pixels wherever the environment mesh intersects a set of planes in 3D space. Non-blue pixels were coloured black, which visually render as transparent. (What you’ll notice in the captured photos is that black pixels are visible due to the way the HoloLens does AR photo capture, but the black pixels don’t show up when wearing the headset.)
One of the major issues with presenting the environment mesh visually is that it’s ugly. The HoloLens environment meshes are captured at runtime, and unlike pre-built models (which can be polished quite extensively by hand), the environment meshes are typically incomplete, have unsightly bumps, sharp edges, and strange normals, and are poorly optimized.
The Star Trek shader actually hides some of the defects. It ignores normals entirely, and defects rendered in black pixels are invisible to the user. The only defects that actually show up are those that intersect one of the 3D planes.
Effects on the Player
The effect is a grid pattern that appears to be overlaid on top of real-world surfaces. It looks like a hi-tech scan of the room geometry, which is exactly what it is. There are a few extra-looking and oddly-angled lines and other visual defects, but I find that the visual defects actually add a bit a character to what would otherwise be a very hygenic-looking scan of the space.
Virtual objects appear to be on top of the grid (see virtual poster in the second photo). Not applying the holodeck shader to virtual objects allows them to be clearly distinguished from the real-world environment. This is important because the person using the app may only have a few minutes with it, and have little former experience with games or virtual reality of any kind.
Getting the results to look better would probably involve some fairly substantial work put into a mesh cleaning system, but that kind of effort doesn’t fit well within the time and budgetary constraints of the project. I don’t plan on improving the environment-mesh visuals further for this project, and will instead be focusing on other areas of development.