The Altered Reality Why
Altered Reality: n. Experiences where technology affects the state or quality of objects or information to having existence or substance. This primarily includes techniques that affect how the human senses of sight, sound, and touch can be manipulated to deliver information and to convince a user that information exists in their current reality. Virtual, Augmented and Mixed realities and 360° media are included in Altered Reality.
Remember 3D? When uttered, 3D tends to trigger anger for the movie ticket inflating annoyance we know too well. It should also remind you of the technology industry wide trend to add the gimmick to any tech product. TV’s, phones, projectors, all were graced with the ability to detect depth in an image. The consumer technology community has passed on, admitting it never became a differentiator in the marketplace to push more device sales. But a takeaway from that moment was listening to a common refrain from consumers: why? Why should I put these glasses on when they make it harder to see the visuals and adds arguably little to the experience?
Enter Virtual Reality. It too has enjoyed the thrilling heights of the hype curve and currently is enjoying that initial dip of a reality crash. After solving many problems technically this year (including less nausea inducing experiences and hand controllers), it still is plagued by the “why?” question.
There is a distinction of how we see altered reality experiences in the innovation space. The usual reasons to explore a technology for Medical Education rarely involve profit. Success for tech like VR in education does not resemble success for the gaming industry. We are concerned with creating experiences that encourage learning and better decision making, specifically creating better trained physicians using tools that augment their wisdom. Altered Reality Technologies can bring mobility and responsiveness (two affordances that have transformed using the web into an everyday break of dawn of habit) to simulation. Students can now experience in the anatomy lab anywhere.
We are challenged with creating experiences that quickly answer the why a user should take their valuable time to enter an altered reality.
Trips to Altered Realities
IIME’s innovation space has now demonstrated VR and AR experiences to dozens of educators. We also recently conducted a section in the 4th Year elective Medicine 2050 where students explored leading edge VR / AR experiences that use proximity services technology like iBeacon. These simulations ranged from Science education to media creation.
Some insights:
-
Seeing is believing. You cannot describe the experience of being in Virtual Reality, even when showing a 2D screen. The reaction when people realize this difference has been enjoyable.
-
Watching someone else enjoy a virtual world is beyond boring. We need new ways for spectators to view someone in virtual reality or augmented reality.
-
It can take years to master manipulating objects in 3D space in multiple interfaces with habituated ease. It takes minutes in virtual reality with all types of users. The HTC Vive leans on comfort with our vision and senses to create a natural interface.
-
You can capture performances, like instruction or testing, in VR.
-
VR and AR are not inherently competing technologies. They have very different affordances and can be made to compliment each other in an experience together.
Content is King
A large part of year one exploration of this technology includes building capacity for creating content in this new medium. Here are some experiments on this front.
Aframe
Aframe, a HTML framework created by Mozilla for WebVR, has many new updates that includes support for VR controllers like the ones in available in the HTC Vive. What does that actually mean? You can go to a website, click on a button to send that Aframe site to your headset, walk around in the space and interact using your controllers.
Here is an Aframe demo you can view on any headset (from Cardboard to the Vive) that shows surface scans of livers.
Photos into 3D Objects
Physicians often lament how students have less and less time in the anatomy lab or exposure to pathological specimens. Dr. Amy Rapkiewicz, a fellow of NYU IIME, is creating an archive of novel pathological tissue in the form of 3D objects that can be viewed using VR and AR technologies like the Vive or the Hololens.
This is an example of a recent Liver with fissures I created taking 64 still images with a Samsung S7 smartphone. The goal of this experiment was to imagine a use case where a physician only had a smartphone at their disposal; what kind of asset could we create using photogrammetry software? The results can be stunning.
The process can be time consuming and sometimes requires the use of third party servers to process. This particular subject was done locally using Agisoft Photoscan and took 4 hours to solve. For faster scans in minutes, we have used the Structure Sensor Depth sensor with an Ipad.
Take a look at more experiments in 3D scanning using different techniques of actual tissue.
360° Video
360° video is exploding because of the ubiquity of affordable cameras. Click and move this 360° time lapse video of our innovation space.
We are assisting the School of Medicine anatomy lab to create 360° 4k video of dissections. With these videos, a learner can control the view and zoom in for more detail of the subject. Click the link in the embed to see this ability on mobile.