Virtual Touch: Inside Technology That Makes VR Feel Real
In a series of experiments in the 1980s, Roland Johansson, a professor of physiology at Sweden’s University of Umea, demonstrated just how important touch is. In one, he filmed a woman picking up a match and lighting it in just a few seconds. Then he numbed her thumb, index finger and middle finger. Even though she could see the match, she couldn’t feel where the match was touching each of her fingers or when it was touching the matchbox, and that same simple task took 25 seconds.
Touch is the most underrated of all of our senses. From walking to sitting to sleeping, you are constantly using your sense of touch to adjust your motions so that you complete the task you’re working on, whether that’s swiping your tablet screen to read this story, adjusting the chair you’re sitting on or picking up a mug to take a sip of coffee.
“Most of us have experienced a temporary loss of touch – perhaps when your leg falls asleep, or when we have local anesthetic for a dental procedure,” says Katherine Kuchenbecker, an associate professor of computer science, mechanical engineering and applied mechanics at the University of Pennsylvania, who is recognized as one of the leading authorities on haptics, or the science of touch. “That’s what current virtual reality is like: you can’t feel anything.”
After getting her PhD in robotics at Stanford, Kuchenbecker was awarded a grant from the National Science Foundation to develop a way for virtual robots to “glance around any scene and accurately estimate how it would feel to grasp or step on all of the visible surfaces.” She did this using something she’s called “haptic photography – a way to capture how the surfaces feel to the touch and then recreate them.” In September 2014, she partnered with tech entrepreneur Steven Domenikos to create a compact device that introduces the sense of touch onto the digital world. Together they founded Tactai, with Domenikos as CEO and Kuchenbecker as Chief Science Officer. After a couple years and a few million dollars in funding and grants, that device – the Tactai Touch – is almost ready for prime time.
“The lack of touch breaks the suspension of disbelief in VR,” says Kuchenbecker. “Your brain knows what to expect when you touch something.” Kuchenbecker’s work centers on the touch-based interaction between a person and objects that may be real, far away – touched through a remote-controlled robot – or entirely virtual.
Kuchenbecker’s lab has been working for years to create realistic haptic interfaces for virtual reality. These research-based systems are usually expensive, complicated, delicate – maybe even a little dangerous – but they let the user touch virtual objects that feel very similar to the real objects we all touch every day. Most people still haven’t been exposed to how good haptic experiences can be and how much touch feedback can add to an interaction.
But now that’s set to change. All of that research with expensive and complicated equipment led Kuchenbecker to focus on existing or low-priced technology that will open up touch to the masses. Tactai has created a prototype touch controller to work with existing virtual reality platforms like Oculus Rift and Samsung Gear VR, though since touch is independent of the headset, any existing or future device will be able to use the technology.
The Tactai Touch controller looks like a high-tech thimble and clips onto your finger. It can be worn on any and every finger, but only one index finger is required to trick your brain into feeling things. “Most of our experience comes through our fingertips, with the index finger driving touch,” Kuchenbecker says. How users employ Tactai Touch will depend on what they’re doing. For example, shopping in VR can be done optimally with one index finger, but opening a virtual water bottle in VR would require a device on each index finger.
“Three devices per hand is a good option, and five devices per hand [one per finger] is an even more exciting configuration, where you could move all of your fingers independently and feel what they each are interacting with,” Kuchenbecker says. “That might be nice when petting a virtual cat or playing a virtual piano. We will be working to determine what configurations make the most sense for the different applications we are pursuing.”
Kuchenbecker says that when you pick up an object like a pencil in real life you have pressure on two fingers, but if you’re wearing only one Tactai, the brain tricks itself into feeling pressure in the thumb even though you’re not wearing the device on that finger.
I was able to test out this technology at the Kuchenbecker’s U Penn lab. Sliding the Touch onto your index finger, it fits snugly but comfortably. It’s not unlike clipping on a pulse oximeter on your finger at the doctor’s office, which measures your heart rate and the level of oxygen in your blood.
Once my finger was hooked up, a bulky prototype headset was secured and Domenikos appeared in a white virtual reality room talking about the importance of touch. I looked to the right and saw a fully inflated football sitting on a counter next to a slightly deflated football. Using Tactai Touch, I can not only feel the detailed, worn leather texture of the football, but also the indentations of the slightly deflated football compared to the tightness of the air-packed football.
“I’ve always pushed my team for solutions that could get to the market in the next few years,” Kuchenbecker says. “Our solution is more powered by unique data and software, and the hardware is pieced together with off-the-shelf components.”
Kuchenbecker explains how the sense of touch is an extremely complex a network of cells that detect different types of physical contact – some tell you how hard you are pressing on an elevator button, while others react mainly when the interaction changes, like at the start or end of the touch or when something slips in your grasp. All of these sensations are constantly streaming to your brain. Even though the information is coming from a lot of different sources, your brain weaves it together into a consistent touch experience that you’re able to use to decide which piece of fabric feels nicest or type without looking at your fingers.
With that in mind, the Tactai software constantly reads where your finger is in space using sensors such as a camera on the headset or an external measurement system. Each time it has a new measurement, it figures out where your real finger is located in the virtual environment. If your finger is in virtual free space, the software commands the platform inside the device to move away from your fingertip, so you don’t feel anything. And if your finger is inside a virtual object, the software calculates a proprietary waveform – or Dynamic Tactile Wave – that makes the platform inside the device press into your finger and vibrate in a way that matches what you would feel if you moved your finger the same way over the surface of the corresponding real object. These calculations happen many dozen times per second so that the experience of the user is seamless.
The magic inside the controller comes in the form of two actuators, or mechanical systems. The first moves a platform in and out of contact with your finger and lets you know when you’re touching an object. Kuchenbecker says when you first barely touch something in VR, you will both see your virtual finger touch the object’s surface and feel the start of contact on the surface of your real finger. Even though the Tactai Touch can’t physically stop your finger from moving inside the object, the sensations on your fingertip give you the illusion that there is a physical object there. You can move your finger away and feel the platform leave your fingertip and then touch again and feel it again.
“As you press deeper into a solid object, the pressure on your finger increases accordingly even if it is nothing really there,” says Kuchenbecker. “Essentially, if the VR technology provides just enough information to suggest the presence of an object, your brain fills in the gaps of what isn’t there.”
“We employ pseudo-haptics [manipulating what you see in VR to match what we want you to feel] by never showing your hand going inside a virtual object says Kuchenbecker. “Instead, we make the finger stop right at the surface, while you are feeling the contact, to help persuade your brain that the object is there. In essence, we are distorting the displayed movement of the user’s hand to help strengthen the illusion created by the haptic feedback.”
The second actuator is a small vibration motor. Unlike the vibration motors in a cell phone, this one functions more like an audio speaker. It can output vibrations at a wide range of frequencies, to create vibrations that match those you feel every day when interacting with real objects. Kuchenbecker says that vibrations are how the brain recognizes texture.
“Your fingers can feel up to 1,000 hertz, which is a lower frequency than ears can hear,” Kuchenbecker says. “This enables the user to feel the textures when touching different virtual objects like rough stones and smooth paper and woven fabric.”
Kuchenbecker explains that if you drag your fingertip across different surfaces in a quiet room, you can hear the sound they produce. Corduroy would sound different from rough stone, terracotta, or silk, and you could also easily feel the difference between these materials. Similar to how you can hear the difference between different voices or different notes in a musical scale, your brain can separate tactile vibrations into the different frequencies that they contain. Each material has its own particular combination of frequencies for a given way that you are touching it (slow and soft, for example), and your brain quickly learns this response pattern from just a couple of seconds of interaction.
“Our technology recreates the blend of vibration frequencies for the way you are moving, so that we can fool you into thinking you are touching a real object even though it is VR,” says Kuchenbecker.
But what’s perhaps most impressive about Tactai’s technology is that it’s already affordable, with current prototypes costing less than $12 to make. And Domenikos says that if a large technology or consumer electronics company licenses their technology and manufactures it at a large scale, it could push costs even further down. Theoretically, Domenikos believes a tech giant could earn money on content and give away the device for below cost or even free, like Google has with Google Cardboard.
Backed by Tactai, Kuchenbecker and her lab have also begun to create a library of touch sensation data. By gathering less than a minute of data on an object like a table using a scanner, anyone can capture the important characteristics and put them into the library, meaning there will be plenty of content that works with Tactai. “We also take existing content and auto-recognize how it looks so you can reach out and touch it,” says Ilann Derche, head of device development at Tactai. “It takes minutes to convert existing video game objects to add touch. We recognize the object based on visual experience and apply textures from our database.”
Using proprietary algorithms, each feeling in the library is assigned Haptic Object Properties (HOPs). “Each HOP currently includes the softness of the surface, its texture, how difficult it is to move along the surface [friction], and how it feels when you tap on the surface,” explains Derche. “This is the key part of the process that results in making the object feel real, so that the user can touch, feel, grasp and interact with it.” The HOPs library currently has over 100 objects in it.
But there’s also another opportunity for this touch controller beyond existing headsets. Derche says Tactai is working on a virtual movie theater, where you can watch a video and pause it to check out a new pair of Nike shoes by feeling the texture of the shoes, lifting them in your hand, and even playing with the laces. And the company could reward consumers for doing that by offering a coupon. Tactai could also help educators, adding touch capabilities to the 2D and 3D multimedia tools that have become mainstream in classrooms.
And unlike the devices that researchers have been experiencing for several years, Tactai Touch isn’t a sci-fi wish. The technology is nearly ready for mass production today, just as companies are laying the groundwork for consumer VR platforms.
The power of touch is something we all take for granted, but it’s the fact that it’s so ingrained in everything we do that makes true touch in the virtual world a huge step forward. Just as some early adopters don’t want to leave VR games and universes – even at this early stage – Kuchenbecker believes Tactai Touch will make the illusion of virtual reality more like real life. It will allow shoppers to feel what that t-shirt they saw online feels like before clicking “buy,” or let gamers grip their goalie stick for better control as they attempt to keep the puck out of the net in a virtual hockey game. Tactai might finally bring the missing link to VR: true interaction.