How Extended Reality enhances our capabilities, storytelling, and collaboration

ai_extendedrealities

AI_ExtendedRealities

Extended Realities Become Reality

Augmented and virtual reality systems are rapidly evolving from emerging technologies into multi-billion dollar industries. But creating solutions that either immerse users in a wholly artificial environment (VR) or augment their environments (AR) with mixed reality (MR) components is a major challenge. In order to ensure clarity, context, sensory balance and that VR/AR systems meaningfully enhance users’ environmental capabilities and experiences, such systems must be designed with human users, not technology, at their center. frog designs and builds Extended Reality (XR) experiences that increase the efficiency and agency of users in their work environments, delight and engage consumers through participatory encounters, and infuse environments with machine intelligence to create platforms that naturally foster deeper connections between the user and their surroundings.
New and existing clients are increasingly looking to frog for guidance and innovative design solutions that utilize XR hardware and software platforms. While we’ve developed a wide variety of XR implementations and uses, our clients have seen the most successful outcomes when they see XR as an enabler for other experiences and processes. The three most common enabling methods use Extended Reality:
1) As a means to enable human superpowers 
2) As a method of facilitating compelling storytelling 
3) As a tool to accelerate the product development cycle.
Here’s a peek into how frog XR solutions have helped our clients.

When we talk about enabling human superpowers, we mean using Extended Reality technology as an intelligent, robust, and insight-driven aide that gives people the ability to do more, better. XR allows for the creation of layered realities that can both respond to and inform human users, efficiently and effectively increasing their agency within the connected environment. 
 
In our partnerships with healthcare clients, frog has demonstrated the potential of XR systems to expand and improve therapeutic options and outcomes.  One such proof of concept project aimed to develop a new pain management mechanism for individuals undergoing burn care therapy. While opioids are the mainstay of pain management, current pain control protocols are inadequate for many burn patients. Alternative measures such as distraction, however, are known to be effective. With this in mind, the frog team set out to create a <a href="https://www.frog.co/designmind/vr-care-for-burn-patients">low-cost headset and virtual reality game</a> suitable for burn care patients. The flexibility of XR technology allowed for design solutions that <a href="https://www.frog.co/designmind/virtual-reality-vrcare-pain-reduction">accommodate burn patients’ needs and mobility constraints</a> during treatment while also providing an open-source gaming concept that gives other developers the opportunity to create their own experiences for patient use. In addition to expanding the range of marketable healthcare technology, this application of Extended Reality demonstrates the viability of safer, more affordable means of pain management. By immersing patients in a virtual reality experience, we can extend their innate pain management capabilities, allowing them to take advantage of the brain’s ability to divert attention away from sensations like pain and consciously refocus it on other information-processing activities.
Current Landscape
PwC&#8217;s “<a href="https://www.pwc.com/us/en/services/consulting/technology/emerging-technology/assets/pwc-understanding-the-effectiveness-of-soft-skills-training-in-the-enterprise-a-study.pdf">The Effectiveness of Virtual Reality Soft Skills Training in the Enterprise</a>” study focused on inclusive leadership training in one of three methods—classroom, e-learning, and v-learning. The study found that VR learners were 275 percent more confident to act on what they learned in the training, four times faster to train than classroom learners and four times more focused than e-learners. Additionally, these VR learners felt 3.75 times more emotionally connected to the material than classroom learners. 
Future Considerations
Extended Reality experiences must be engaging, usable, and provide continual value. Introducing XR purely for the sake of innovative technology has the potential to create over-orchestrated solutions and new hurdles, hindering capabilities rather than enhancing them. Any XR solution should help deliver on the core promise of the business and enhance the overall user experience.
As technology merges more seamlessly with individual users, data collection and privacy practices will become increasingly important. Collections of human metadata from connected XR devices could be exploited for marketing purposes, while VR hardware enhancements such as eye tracking (for foveated rendering or interpupillary adjustments) might also allow for unwanted data collection. Especially as more data migrates to the cloud, systems that collect and protect user data must be carefully considered.

Enabling Human Superpowers 

frog believes that storytelling is key for the success of any new venture or design solution, regardless of whether we’re working with a cultural institution, a new startup or a Fortune 100 conglomerate. If we can’t convince a customer of the value inherent in a new experience, or if the corporate board can&#8217;t see the benefits of a new product design, then innovative initiatives may not succeed. Extended Reality enables organizations to tell compelling stories by immersing the audience in narratives set in dynamic virtual environments. 
 
frog worked with TouchTunes to redesign their <a href="https://www.frog.co/work/reinventing-the-touchtunes-digital-jukebox-for-next-gen-in-venue-entertainment">digital jukebox system</a>, combining its hardware, software and services into a seamless, connected experience that helped establish TouchTunes as a leader in digital entertainment services for venues across North America. In order to communicate the intent and features of the redesign, frog built a VR demonstration of the new product that the client could walk through and interact with. The virtual experience allowed the stakeholder teams to personally engage the new product at scale and told a visceral story about the future environments the product would live in. 
 
The San Francisco Museum of Modern Art, one of the most influential cultural institutions in the US, is a recognized leader in using technology to engage both art lovers and new audiences. For its Spring 2018 special exhibition, Rene Magritte: The Fifth Season, SFMoMA asked frog to help them create an “<a href="https://www.frog.co/work/augmented-reality-meets-fine-art">interpretive gallery</a>” in which museum visitors could experience “the paradoxes at work in reality” that Magritte explored in his late-career work. This immersive space drew the audience into a spatial and participatory narrative by compelling them to solve augmented reality puzzles based on motifs found in Magritte’s paintings, engaging viewers in a completely different way than exhibitions of the past.
Current Landscape
Extended Reality adaptations to traditional storytelling formats like film, video games and journalism have been key to the rise of XR experiences in recent years. For example, the New York Times has embedded augmented reality elements to enhance reader context in stories covering topics such as the Olympics and <a href="https://www.nytimes.com/spotlight/augmented-reality">COVID-19</a>. In the latter, readers can visualize the recommended social distancing range by placing a ring around themselves using an augmented reality overlay. Societal expectations for these types of immersive and convergent storytelling experiences will continue to grow as long as they enhance comprehension and provide value to the end user. 
Future Considerations
Game engines and real-time rendering methods are evolving rapidly. Epic Games’ <a href="https://www.unrealengine.com/">Unreal Engine</a>–the foundation of the hugely popular game Fortnite–is one of the key technologies being used to <a href="https://www.unrealengine.com/en-US/features/vr-ar-and-mr-xr-support">build new XR experiences</a> and speed the production of hit shows like <a href="https://www.youtube.com/watch?v=gUnxzVOs3rk">The Mandalorian</a> on Disney+. With Unreal Engine 5, Epic Games achieved a major breakthrough in real-time rendering, enabling the <a href="https://vimeo.com/417882964">visualization of billions of polygons</a> in each frame. Such advances will speed the convergence of film, game design and XR, unleashing immersive, cinema-quality and hyper-realistic XR product storytelling experiences in the future. The physics and lighting built into these game-engine driven XR visualizations will allow consumers to explore products in highly dynamic and customizable ways, enhancing comprehension of fit, feel and scale of everything from clothing to automobiles.

Facilitating Compelling Storytelling

Designing tangible new products for our clients has been core to frog’s legacy since the founding of the company 50 years ago. Close collaboration between cross-disciplinary team members and rapid iteration during the product design process have been key to frog’s success and to our ability to bring new products to market quickly. Extended Reality tools and processes have even further enhanced and improved the way we work. We use XR systems in diverse ways, from testing new design simulations with future customers to iterating industrial design and architecture at scale.
Recently frog designed a state-of-the-art collaborative lab for one of the leading genome research groups in the world. Their lead investigator and founder wanted to &#8220;create a signature, intelligent, sentient work environment where people come to do the best work of their life.&#8221; frog designed new workflows, collaboration tools, rituals and spaces to support and evoke that ethos, and then built a full-scale VR mock-up of the entire space, complete with functioning prototypes of the workbench, storage systems and lighting. By sharing our design with the client in VR, we used XR technology in the same collaborative spirit that the lab was intended to embody.
In 2019 frog helped an autonomous scooter company design a vehicle that transcends the 3-wheeled vehicle stereotype into a vehicle that people will love to drive. We were asked to evaluate the client’s current autonomous scooter prototype and develop a design that would be more competitive in today’s market. Because building a vehicle appearance model requires a lot of time, money and resources, we instead used VR models to assess our vehicle designs on a daily basis, leveraging the latest technology to review the form, iterate quickly and save time and money for the client. 
Current Landscape
The current landscape of XR product design requires that designers and developers reconcile competing AR and VR hardware platforms, a limited selection of collaboration software, and a low degree of interoperability. Current CAD software is slowly adopting XR systems as another output for display and collaboration, but improvements to XR software and hardware standards are necessary before we are able to collaborate with teams more holistically in a less restrictive workflow. The <a href="https://www.khronos.org/news/press/multiple-conformant-openxr-implementations-ship-bringing-to-life-the-dream-of-portable-xr-applications">OpenXR runtime standards</a> are currently contributing to greater hardware and software interoperability, which will help make XR collaboration more easily achievable. This means that present day XR collaboration applications can be easily developed to run across a range of hardware from VR and AR headsets to mobile AR devices. This could have profound implications given our new remote-working norms and varying levels of access to XR hardware for distributed employees. 
Future Considerations
In the future we will be able work on complex design problems untethered from the large, physical 2D screens we use today. Some design processes will take place in virtual 3D spaces from beginning to end, with entire teams joining from their preferred work location using technology no bigger than a pair of eyeglasses. Continued hardware evolution will allow designers to flip between XR modes at will. AR will allow for enhanced in-person collaboration, while VR will allow us to change context and simulate design outcomes without switching between hardware or software.
Extended Reality—whether augmenting our daily lived experiences to allow us to work in a more familiar and connected environment, enabling us to imagine the possibilities of something new within our own home, or refocusing our consciousness to experience the absence of pain–enables us as humans to forge connections and experience the environments around us in a more informed, efficient, and empowered manner. With the additional opportunities in using Extended Reality to aid in creating more efficient product delivery methods or enhance and engage us through new means of storytelling, the opportunities seem almost boundless in what it will enable us to do next. The important thing is to always remember that this technology alone doesn’t solve anything. Only by exploring this emerging tech through the lens of human-centered design can we arrive at solutions that add true value to the people that need them.

Implementing Faster Product Development Cycles

Alexis Du Mond Puchek

Charles Yust

Authors

frog_author_alexis_puchek

frog_author_Alexis_Puchek

charlesyust-300x300

CharlesYust-300x300

frog and our partners joined a robust digital crowd at the first ever virtual SF Design Week to discuss the ethos, benefits and assumptions around inclusive design


sfdw_recap-1

SFDW_Recap (1)

Designing for Inclusive Experiences

For the first ever remote San Francisco Design Week, our team addressed the important issue of how to design for inclusive experiences. Together with our partners Corey Timpson and Sina Bahram, Charles Yust and Tammy Baird discussed the ethos of inclusive design, plus how we can keep pushing ourselves to learn from our past work to ensure we’re continuously moving forward and making better design decisions.

frog and our partners joined a robust digital crowd at the first ever virtual SF Design Week to discuss the ethos, benefits and assumptions around inclusive design

<h2>What is inclusive design?</h2>
This question seems simple enough to answer — design that is inclusive for all. But so often, “inclusive design” only meets the bare minimum without truly making experiences or products available and accessible to the widest possible audience. The goal of inclusive design is to actively fight against these biases to make experiences as welcoming and accessible as possible for as many people as possible.
In order to better communicate the goals and ethos of inclusive design, it’s helpful to define some key terms: 
<ul>
<li style="font-weight: 400;">Accessibility: The ability of all people to use a product, place, service regardless of disability.</li>
<li style="font-weight: 400;">Disability: The consequence of an impairment that may be physical, cognitive, mental, sensory, etc.</li>
<li style="font-weight: 400;">Othering: The act and consequence of making someone or a group of people feel essentially different.</li>
<li style="font-weight: 400;">Inclusive Design: Inclusive design is a design process, not restricted to interfaces or technologies in which a product, service, or environment is optimized for a specific person with specific needs.</li>
</ul>
With this in mind, Sina points out that it’s imperative for designers to make the distinction between the medical vs. social model of disability. The prevailing medical model is to say the individual is disabled, meaning there is something to fix or correct in the individual. But for so many, what we consider a “disability” cannot be cured or fixed, Sina, notes. Rather, we should be adhering to the social model of disability, which posits that it’s the environment that is disabling, not the individual that’s disabled.
&nbsp;
<h2>Ethos of Inclusive Design</h2>
Starting from this assumption that it’s not the individual that’s disabled, but the environment that’s disabling, we must ask ourselves as designers, what can we do about it? As Sina notes, “inclusive design is a design process that realizes our fellow humans are going to interact with the things we put into the world in their own special way. Let’s not put barriers and constraints in front of them to do that.” 
This is a truly human-centered approach to design, as it puts the needs of all humans up front and center, rather than assuming the “average” or “norm” will suffice. In order to build inclusive experiences or products, it’s imperative that we start to make this switch. “Rather than designing and developing something and then figuring out how to make it accessible, we design with a consideration to all audiences and all vectors of human difference from the outset so that we don’t have to make compromises down the road,” says Corey. 
Some key takeaways from Sina and Corey’s inclusive design practice: 
Inclusion and exclusion: Exclusion results from mismatched design intentions and user intentions. Inclusive designers take responsibility for understanding how design intent affects, responds to, and facilitates user intentions. Resolving points of exclusion yields not only inclusion, but immersion and innovations.
Inclusive ecosystems: Users today exist in both physical and digital space, and we can no longer consider them in isolation. As Corey says, “All of the pieces have to work and be inclusive for the entire ecosystem to be inclusive.” 
Characteristics of Inclusive Design: Inclusive design provides multiple entry points for people with a variety of preferences, learning styles, and levels of ability. As Corey says, this is crucially important because the more variety we provide, “the more opportunity we’re giving people to enter the content and experience, and the more we can build engagement.” 
<ul>
<li style="font-weight: 400;">Mixed interaction design (passive, active, interactive)</li>
<li style="font-weight: 400;">Analogue and digital blends (mixed and trans media)</li>
<li style="font-weight: 400;">Stylistic variety</li>
<li style="font-weight: 400;">Immersion</li>
<li style="font-weight: 400;">Multi-sensory affordances</li>
<li style="font-weight: 400;">Strategic redundancies</li>
<li style="font-weight: 400;">Deliberate decision making</li>
</ul>
&nbsp;
<h2>Hindsight is 20/20 </h2>
In order to better understand how we can continue to push forward with inclusive design practices in our future, we took some time to learn from our past. One past project we used to assess where we can improve inclusivity in design is our work with SFMOMA on the <a href="https://www.frog.co/work/augmented-reality-meets-fine-art" target="_blank" rel="noopener">Magritte Interpretive Gallery</a>. 
Key questions we set out to answer through this dialogue were: 
<ul>
<li style="font-weight: 400;">How can question norms and previous notions of inclusivity?</li>
<li style="font-weight: 400;">Can we be more empathetic in our design process?</li>
<li style="font-weight: 400;">How might we ensure a broader representation on our teams and across our research participants?</li>
</ul>
The interpretive gallery was an immersive augmented reality space designed in collaboration with a team of curators, digital staff, marketing and visitor experience specialists from SFMOMA to capture René Magritte’s artistic intent and create an interpretation of his motifs in an engaging interactive experience that wasn’t dependent on any personal devices. The design of the experience and software needed to accommodate visitors of all ages, heights, skin colors and whether they were wearing glasses in order to make it inclusive of the museum’s entire audience. 
Looking back at this project, we considered how we might design an even more inclusive version of it in the future by asking: 
<ul>
<li style="font-weight: 400;">How could we use audio to tell the story of the experience?</li>
<li style="font-weight: 400;">How could we cue visitors to “hidden” features or content layers that might be more obvious to people of varying abilities?</li>
<li style="font-weight: 400;">How would these enhancements have given the audience even more insights into the motifs and mystery inherent in Magritte’s work?</li>
</ul>
As Sina pointed out, every design problem has a set of constraints and any good interactive exhibition encourages people to discover or uncover something. It’s essentially a puzzle. The trick for designing experiences to be truly inclusive is to look at the puzzle through many different modalities so that it’s not exclusionary based on a single modality. For instance, if the “delight” that’s uncovered is seeing an image reflection move based on the user’s own body, then perhaps there is a haptic equivalent that would engage a different sense. The addition of this haptic or sound element wouldn’t take away from users engaging with the visual element, but it would create a whole new layer for more people to be able to engage more deeply with the experience.

Here are 5 key points to making sure processes around product design and development are inclusive from the outset: 
<ul>
<li style="font-weight: 400;">Hire diverse teams and encourage a wide range of perspectives</li>
<li style="font-weight: 400;">Recruit a diverse set of research participants</li>
<li style="font-weight: 400;">Design for experiences that are not othering and are inclusive from the outset</li>
<li style="font-weight: 400;">Leverage technology in a way that makes experiences  more inclusive </li>
<li style="font-weight: 400;">Consider product representation, marketing and outreach</li>
</ul>
The real takeaway around inclusive design is that it not only enables greater access to products, services and experiences, but expands the horizon of innovation, creativity and design for all, regardless of ability.

Tammy Baird

tammybaird_nov2018_bw_square-1

tammybaird_nov2018_bw_square (1)

A SXSW panel discussion on how designers, artists, curators and storytellers of all kinds can come together to leverage technology in the service of digital play.

sfmoma_1-1-1-2

SFMOMA_1-1-1

Art Museums as Spaces of Digital Play

For most, “fun” is not at the top of the list when it comes to adjectives ascribed to art museums. While kids museums have “fun” interactive installations that are designed to teach through discovery and play, there are few critical examples of interactive play for art museums. On the whole, these institutions tend to project authority rather than empathy or emotional connection. However, museums’ aspiration to grow attendance beyond traditional art audiences has begun to create opportunities for playful engagement almost unthinkable a decade ago. Digital experiences, particularly AR, have played a key role in this development. As a designer, I have found in my own work with frog that focusing on delight over didactic forms of education can have powerful effects on fostering understanding and engagement with people of all ages. Along with my fellow panelists, we’ll examine trends in the newly emerging field of interactive design in fine art and take a deep dive on findings from the immersive <a href="https://www.frog.co/work/sf-moma">Interpretive Gallery</a> created by SFMOMA and frog in 2018 for the “Magritte: The Fifth Season” exhibition.
Ahead of our conversation, I spoke to my fellow panelist Sarah Brin, a creative producer specializing in play, new media, and accessibility and Digital Storytelling Program Manager at Meow Wolf, about how we can begin to create these spaces of digital play in museums. At SXSW we will be joined by Chad Coerver, Chief Content Officer at SFMOMA.
&nbsp;
Chuck Yust: Why do you think it’s important for museums and cultural institutions to embrace new forms of engagement and/or emerging technology (like AR/VR) in order to reach new audiences?
Sarah Brin: There are at least a couple of really important reasons! One would be that because these are emerging forms, we can’t let industry exclusively shape that discourse. Artists and cultural institutions need to play a part in determining how we use technologies, who uses them, and what we use them for. And while these technologies are definitely not perfect, creative producers have a responsibility to engage with them now to make sure we’re building the kind of cultural future that we want for ourselves.
And a second is that the visual languages of art historical genres like for example, Abstract Expressionism, are often unfamiliar to many contemporary audiences. The visual language of AR and VR, especially in gaming contexts, is much more ubiquitous and recognizable to many. Artists and arts organizations can (and have) use this somewhat common aesthetic to engage audiences who might be otherwise alienated by traditional arts contexts.
&nbsp;
Chuck Yust: What is an example of a stellar digital or interactive installation or industry trend you’ve seen in cultural institutions thus far?
Sarah Brin: About 8 or so years ago, museums started to catch “videogame fever,” and a lot of games-focused exhibitions started popping up at art museums all over the place. But they really varied in quality and had a tendency to emphasize aesthetics as the primary artistic value.
But lately, we’re starting to see a lot more nuanced, focused, brilliantly-curated games projects that, in one way or another, focus on experience as a criterion of understanding the work. Design/Play/Disrupt at the V&amp;A is a great example. ACMI in Melbourne is really thoughtful about this too. I’d be remiss if I didn’t highlight PlaySFMOMA, the museum’s ongoing initiative supporting artist-made games, as a great example too. Although, admittedly I’m very biased because I had the great privilege of working with the museum a bunch on shaping the program!
&nbsp;
Chuck Yust: What does “focusing on delight” (more than didactic forms of education) mean to you?
Sarah Brin: I was an in-gallery educator for a really long time, and that was really valuable for me in terms of figuring the various headspaces people are in when they come to museums. And of course there are the people who’ve been coming to museums their whole lives and feel very comfortable there–but that’s not most people. And to many, art museums can be really intimidating; they’re quiet, they’re pristine, they’re full of really expensive stuff that you can’t touch and might not understand.
For me, delight is a way to reduce some of that pressure and anxiety. It looks like extending an invitation, like making people feel like they’re not only welcome, but that they’re filling in a missing piece of the museum. It looks like an opportunity for surprise and play! I think a lot about comedy when I think about museums, and there’s this idea that a lot of theorists reference, that laughter is the sound of an audience congratulating themselves for understanding a joke. To me, a delightful art experience combines those feelings of validation with making playful (or even joyous) connections to and re-contextualizations of lived experience.
&nbsp;
<h6 style="text-align: left;">About the Panelists</h6>
— Sarah Brin is a creative producer specializing in play, new media, and accessibility. She currently works as the Digital Storytelling Program Manager at Meow Wolf. Before joining Meow Wolf, Sarah has developed programs, exhibitions, and publications for more traditional arts institutions including the Hammer Museum, British Arts Council, the Museum of Contemporary Art Los Angeles, the MAK Center for Art and Architecture and elsewhere. Sarah specializes in organizational development and the implementation of new initiatives and technologies within arts organizations. Some of her prior projects include supporting the development of PlaySFMOMA, the artists in residence program at the Pier 9 Workshop, and an EU research grant focused on playful museum experiences.
— Chad Coerver has served as chief content officer for the San Francisco Museum of Modern Art (SFMOMA) since 2012. He is responsible for establishing the overall direction, tone, and appearance of SFMOMA-produced public media, exploring new platforms and creative approaches in the effort to broaden and engage the museum’s audiences. He also leads SFMOMA’s Content Strategy and Digital Engagement division, which comprises more than 30 staff members across Web + Digital Platforms, Interpretive Media, Community Engagement, Publications, and the Design Studio. Coerver lectures frequently on content strategy and organizational change, recently appearing at the conferences of the Association of Art Museum Directors, Museums and the Web, the American Alliance of Museums and National Museum Publishing Seminar. A former specialist in Renaissance Art, Coerver holds an MPhil in art history from Yale University and a BA in art history from Duke University.
— As a Principal Design Technologist at frog Charles Yust leads client engagements, develops software, conducts research, and builds prototypes of new products with a focus on emerging technology. He is frog’s AR/VR tech capability lead and has built mixed reality applications across a range of platforms and spaces that include low-cost pain distraction for burn patients in VR (VR Care) and the first version of the IKEA Place ARKit app announced at WWDC 2017. Prior to frog he received an MFA with honors in Design and Technology from Parsons School of Design and worked for two award-winning design consultancies in NYC building immersive and gestural interactive environments for leading museums and clients throughout the US. He lives in San Francisco and is a self-taught pancake artist (Instagram <a href="https://www.instagram.com/prandomcakes/?hl=en">@prandomcakes</a>)
Visit the <a href="https://schedule.sxsw.com/2019/events/PP91358">SXSW schedule</a> for more information on “Art Museums as Spaces of Digital Play.”

frog is a global design and innovation firm. We transform businesses at scale by creating systems of brand, product and service that deliver a distinctly better experience. We strive to touch hearts and move markets. Our passion is to transform ideas into realities. We partner with clients to anticipate the future, evolve organizations and advance the human experience.

Author

Best practices for developing low-cost VR to support burn patients receiving wound care.

designing-virtual-reality-for-pain-reduction_vrcare_-640x360

Designing-Virtual-Reality-for-Pain-Reduction_VRCARE_-640x360

Designing Virtual Reality for Pain Reduction

Imagine suffering a serious burn. The pain is unlike any you have ever felt. As you are rushed to the hospital for stabilization, in the throes of one of the worst moments of your life, you are only at the beginning of an agonizing journey.
After your initial treatment, you arrive at a hospital burn center for weeks or months of ongoing care. Especially in the beginning, this treatment can be just as excruciating as the initial injury. It includes removing the bandages from the burns and scrubbing them, before applying ointment and replacing the dressing. The process can last for up to two hours and it must take place every day, sometimes twice a day, in order to protect your burned flesh from infection. In addition, you will likely need operations. The pain is relentless, and even as it slowly improves, mental scars remain.
The acute injury of a burn — followed by the days, weeks, or months of frequent wound care and dressing changes — is one of the most painful things a patient can experience. Opioid pain medications, the mainstay of pain management, have well known short and long-term side effects. Despite these medications, pain often remains inadequately controlled in burn patients.
Alternative measures for pain control, including distraction, are known to be effective. Numerous studies have demonstrated that the immersive environment of virtual reality is a highly effective method of distraction for burn patients receiving wound care. However, virtual reality is not widely used in clinical practice outside of research studies, because many of these systems are customized and expensive. Emerging systems such as the Samsung Gear VR and Oculus Rift dramatically increase access to virtual reality, but even these may be too expensive for clinical settings, particularly if they are only used with a single-patient to minimize infection control concerns.
Google Cardboard presents an opportunity to pair nearly ubiquitous smartphones with an inexpensive cardboard box and lenses, in order to create a virtual reality system suitable for healthcare. However, existing Google Cardboard headsets are not designed for a moist burn wound care environment. Furthermore, most existing games are designed with the assumption that the user is upright and able to move about in an environment allowing for six degrees of freedom, which is not feasible in patients receiving burn wound care.
At frog we saw this as a design challenge: understand the opportunities and constraints involved with burn wound care and design a low-cost headset and VR game experience suitable for these conditions. Together with Dr. Brian Pridgen, a healthcare design fellow at frog and a Stanford Plastic and Reconstructive Surgery resident, we approached this challenge as a multidisciplinary team. The outcome of this passion project will be an inexpensive headset concept that is extremely affordable, easy to assemble, and feasible for use in the burn wound care environment. In addition to the headset, we are developing an open-source game concept, “Ēpiónē”, for use by burn patients. Our hope is that it can serve as a platform or template that other developers could use to build additional VR experiences for patients. The game will allow the patient to remain appropriately positioned for burn wound care while being immersed in a distracting virtual reality environment.
While working on this concept, the project team assembled a list of design principles that we share in detail below. The principles emerged through design and development of both the hardware and software, and they take into consideration the design constraints we documented through on-site research, interviews with nurses and patients, and user testing. The principles may be generally applicable to VR use in healthcare applications, but they were created with burn patients in mind. In the spirit of making healthcare solutions accessible as widely as possible, we’re sharing them below for anyone who is designing or building solutions in this space.

<h2 style="text-align: center;">Headset</h2>
A number of design constraints dictated the need to explore a custom VR headset solution. It needed to be very low-cost, built using a durable and water-resistant material, and comfortable for the patient. Ideally the design would be easy to assemble from pieces that would allow it to be flat-packed for efficient distribution. The following are specific principles that guided our headset design.
Cheap to manufacture – Hospitals have limited funding, and relying on insurance companies for funding creates a barrier. If headsets are $30 or less (we aimed for $10 or less), hospitals are more likely absorb the cost for the patient benefit. If the cost is low enough, the headsets can be provided to each patient, minimizing infection control concerns that might otherwise be a barrier to adoption.
Water Resistant – Before removal, the patient’s bandages are soaked with slow moving warm water from a hose, which is controlled by a nurse. There may be some indirect contact with water. It is okay if the headset gets wet, but the device inside should be protected.
Patient Comfort – The most painful part of the cleaning procedure can last up to two hours. The patients are often lying down, but may sit up as well. Materials, weight, form, and adjustability to head sizes should be carefully considered.
Accessible to nurses – It should be easy for a nurse to access the device inside the headset while the patient is wearing it, without having to remove the entire headset.
Device Agnostic – Since each patient may receive a headset, it should accommodate a wide range of devices. A desired range might include devices with 4-inch to 6-inch screens.
Intuitive assembly and operation – Assembly and phone insertion should require as little instruction as possible. Devices of any size must be properly aligned/centered to avoid double vision while looking through lenses. A guideline on the headset, combined with alignment cues in the app could help with centering the device.
Eco-friendly – Where possible, environmentally friendly materials should be used to reduce environmental impact. Easily replaceable parts should be considered to avoid having to replace the entire headset.
Delightful – Burn patients undergo severe physical pain and emotional stress. Any way to induce positive feelings throughout the entire experience — from assembly to after the patients return home — should be explored. Consider personalization.

<h2 style="text-align: center;">Software</h2>
The most important design principles for the game software address the lack of functional VR applications that work when someone is lying down, as many burn patients do while undergoing wound care. As the patient prepares for their procedure, they may have little opportunity to start the game experience themselves because they are connected to IVs, monitoring devices, and oxygen, so they will require assistance from nurses. The game experience must also be forgiving to allow for different types of players that have varying levels of familiarity with VR, and games in general.
Admin for Nurses or Doctors – An admin mode should allow nurses to adjust game settings, volume, brightness and alignment before the procedure starts. After assisting with these settings, the staff should be able to easily initiate the VR experience as they align the device in the headset, so the patient can avoid exiting gameplay during the procedure.
Allow for supine position – Most VR games will not play correctly if the player is lying down, because the device assumes the player is standing/sitting and looking up.
Calibration – Burn patients are often lying down, which the device’s accelerometer must accommodate. Once the phone has been inserted in the headset, the game should allow the user a moment to hold their head still while pointed forward, in order to establish a reference point for rotation.
Accommodate limited head mobility with head-tracking-only controls – Patients lying down can only comfortably move their head slightly to the left and right. Vertical up/down head movement is more restricted.
Age-friendly – Patients will be of all ages, demographics, and interests. Game content, including visual style and story, should be as inclusive as possible.
Challenging for all experience levels – Patients may have no video game experience, or a lot. Pain medication may also impair the patient’s ability to concentrate. After the initial difficulty level is selected from the admin menu, difficulty should continue to increase over time.
Reduce eye strain – Long exposure to bright colors will cause eye fatigue, so limit use of bright colors in the game world. Setting hardware brightness correctly before gameplay is key.
Forgiving gameplay – Gameplay should be forgiving across all difficulty levels. Players should be given second chances to correct their mistakes (which may be out of their control due to wincing from pain, or a nurse asking them to move). A regenerating shield is an example of a useful game mechanic that accomplishes this goal, and even produces unpredictable periods of suspense that add to the interest of the game.
Audio – Place sounds in physical locations around the player position. Make sure audio includes appropriate and immersive sound effects and background music loops. The audio should be nearer to higher frequencies since low bass tones are sometimes inaudible on standard mobile device speakers.

<h2 style="text-align: center;">Conclusion</h2>
frog assembled these principles during active development and testing of the VR Care headset and custom game experience, which is <a href="https://github.com/frog/vr-care/">available here as an APK file and an open source Unity package</a>. The headset is now being used as part of an ongoing study with burn patients. This work is an example of the power of partnerships between medicine and design, with the potential to produce solutions that increase access to affordable virtual reality for patients. We hope that this work will serve as a model for future medicine-design partnerships to identify clinical needs, design practical solutions, and bring these solutions back to patients and the healthcare system. Feel free to leave us questions in the comments section and <a href="https://www.frog.co/contact">click here to get in touch about working with frog</a>.

Andrew Haskin

Brian Pridgen

Brian is a Visiting Healthcare Specialist at frog and a plastic surgery resident at Stanford. While at frog, he has developed his interest in merging human centered design with novel applications of technology to improve healthcare delivery and patient care.

Charles has over 15 years of experience implementing compelling digital experiences across a range of platforms at various scales, from mobile and web applications to spatial installations. He conducts research, develops software in a range of languages, and builds interactive prototypes utilizing cutting-edge technologies.

Andrew is an interaction designer in San Francisco who excels at finding order and meaning in the complex and abstract. His raw enthusiasm for the power of design is infectious as he blends craft and process-oriented pragmatism to execute ideas that innovate and disrupt.

	Global		EN
	France		FR

	Global		EN
	France		FR

Charles Yust

Featured

Designing Virtual Reality for Pain Reduction

Articles

Our Studios

Careers