Color in Layers: From Pepper’s Ghost to Augmented Reality

Michael J. Murdoch
Munsell Color Science Laboratory at the Rochester Institute of Technology

Abstract
Augmented reality (AR) is a tantalizing, developing technology that promises to seamlessly blend virtual, computer-generated objects into the real world. AR is already being used in industrial and medical applications, has enjoyed its first hype cycle with gaming, and will likely impact education and retail applications, among others. One type of AR, optical see-through AR (OST-AR), is implemented in a goggle- or glasses-like headset using an optical combiner such as a beam splitter to create a transparent display where virtual content can be overlaid on the real world visible behind it. Because the transparent display is unable to block out the real-world background, the appearance of AR content can vary, from apparently solid to transparent or ghostly.

Ghostly images via glass beam splitters have an interesting history in theatre, popularized in the mid-nineteenth century in London during the era of phantasmagoria. Attributed to John Henry Pepper, Pepper’s Ghost is the reflected image of an off-stage, spotlighted actor who appears glowing and transparent, able to interact with and even physically pass through the other actors on the stage. The ethereal effect was a sensation at the time, offering a surprising visual experience that seemed inexplicable except by magic.

AR is similarly compelling in part because it is novel and challenging to explain. However, transparent reflections are actually no surprise at all, as they accompany everyday windows and other transparent, glass-like objects. How do we interpret the reflections in a transparent object? In a window, we usually ignore the reflections, but sometimes pay attention to them. In water, reflections similarly vary: we may see all the way to the sandy bottom below, we may see the reflected sky and hills, or we may see the medium itself: blue, aqua, white, ever-changing. In an object such as a cut diamond, we are mesmerized by the reflecting sparkles and apparent multitude of facets.

Our recent research has shown that in OST-AR display systems, which function similarly to Pepper’s Ghost, viewers perceive a blend of layers: the virtual AR “ghost” foreground and the real-world background. As with glass windows, when looking at the reflected ghost, the background layer is ignored to some extent; when focusing on the background, the ghost layer is ignored instead. We model the relative impact of the AR foreground layer and the real-world background layer as a perceptually-weighted sum. However, because the discounting of layers is inconsistent, depending on the visual task and the attention of the viewer, the perceptual weighting varies. Current work continues to measure the magnitude of discounting effects in layered transparent AR environments with the goal to model them robustly. Such a model will enable proper accounting for perceptual effects, which will help ensure that AR content can be reliably, naturally, and comfortably delivered to viewers.

Acknowledgment
This material is based upon work supported by the National Science Foundation under Grant No. 1942755.

Bio:  Michael J. Murdoch is an Associate Professor in the Munsell Color Science Laboratory at the Rochester Institute of Technology, where he teaches topics including colorimetry, psychophysics, lighting, and imaging. He leads a research project on color appearance in augmented reality (AR) funded by the U.S. National Science Foundation, and additionally conducts research on displays and temporally dynamic LED lighting. Through his career, Mike has enjoyed working on color system modeling and human-centered design for solid state lighting, LCD, and OLED display systems, previously working at Kodak Research and Philips Research. He holds a BS in Chemical Engineering from Cornell, MS in Computer Science from RIT, and PhD in Human-Technology Interaction from Eindhoven University of Technology in The Netherlands.

Michael’s website