One of the most amazing VR experiences I’ve ever had was being able to touch an object that doesn’t exist. Immersing yourself in a virtual world with a headset that provides video and audio is one thing, adding a touch is something transformative; the physical interaction brings you into this virtual world in a much more direct way. But, as with everything related to virtual or augmented reality, all this is extremely demanding on the equipment. And especially in the case of augmented reality, all this equipment can get in the way.

This hardware layer is a problem for virtual touch because it just seems impossible to bypass it. Video and audio have ways to make themselves transparent, such as through glasses, contact lenses, or bone conduction, which allow you to experience the real world with an extra layer of reality added on top – augmented reality. However, with touch, there are no good options for transparency, because there must be some physical object touching, say, your fingers for you to actually feel the sensations. It’s usually some kind of glove, and whether or not the glove transmits a tactile signal, you’re still wearing it all the time, which isn’t transparent at all.

Researchers at the University of Chicago have found a clever way around this problem. In a paper to be presented at the 2023 Conference on Human Factors in Computing Systems, or CHI 2023 (where it will also receive the best paper award), they demonstrate a wearable system that can generate tactile sensations in the lower body. your fingers and palm without interfering with the equipment, instead hacking at the nerves on the back of your hand.

This method is based on the so-called “reflected sensation”, where stimulation of your body in one place is felt in another place – this is similar to the case when you accidentally hit your elbow on something, but feel a tingle through your fingers because the signal traveled along your nerves through your arm. In some places, including the fingers, the referred sensation can be determined with a reasonable degree of accuracy. With a signal electrode on the top of the finger and a ground electrode closer to the wrist, separate parts of each finger can be stimulated, creating 11 individually controlled tactile zones on five fingers and the palm.

However, simply by snapping your fingers, you will not be able to receive these signals. only on the bottom (palm) side of the fingers, not on the top (back) side where the signal comes from. Fortunately, the asymmetric arrangement of the nerves in our hands makes this possible. The back of the hand stimulation technique works because your palms and fingertips are much more sensitive than the back of your hands, thanks to about 60 times more mechanoreceptors on the palm side. Thus, if you are using an electrode to stimulate the back of one of your fingers, the sensitivity on the front is so much higher that you will feel it much more even if the electrode is in direct contact with the back. side. The researchers were able to find enough stimulation intensity to trigger the nerves in the pads of the fingers while remaining below the threshold of detection on the back of the fingers, neatly solving the problem.

We asked Pedro López, head of the Human-Computer Integration Laboratory at the University of Chicago, to describe what it is like to use this system:

Electrotactile stimulation is not like real touch. But on contact, it feels like touching my skin, which is pretty realistic. When touching the bouldering wall you see in the video, you feel your hand touching the wall exactly where you would expect it to. Surprisingly, you feel it in completely different points of the palm, and not where the electrodes are – in virtual reality, when I see my hand touching the wall, my brain expects the sensation at several points in front, and this is exactly what I feel. It feels good on contact, and the moment I grab the grip for the first time, it feels right. There are still some improvements to feel constant pressure, and if I keep grabbing the grip for a minute, I start to notice that it’s not the same as a real grip.

In a couple of user studies, most participants reported that more than 90 percent of tactile sensations were felt on the palmar side of their hand, despite electrodes placed on the back. And the location orientation was pretty reliable too. In mixed reality manipulation tasks that included clay work and virtual rock climbing, participants noted that “the sensation itself was warm and tingly” and “I felt exactly what I expected to feel.” Perhaps most importantly, this system allows you to use your hands for other purposes at the same time without affecting the VR/AR experience. As Lopez explains, it can be as simple as being able to use a keyboard or setting up a virtual reality headset, or as difficult as DJing:

While you see a lot of VR in video, with this new level of hands-free haptics, we think we can uncover new uses for haptics that go beyond VR/AR. For example, we think that tactile sensations can be used in educational applications (for example, when playing the piano or drums) or even in complex interactions. For example, in the video, you can see me DJing, and the digital DJ system helps me – while I grab the faders and turntable, I feel tactile notifications that tell me when to release the record in order to achieve the best mix. No DJ would wear gloves just to enjoy that extra feedback, but this way we feel the trade-off is much more balanced.

Much more can be done to improve this technology. Lopez says that in addition to being able to sense contacts, they already have a system set up to create a sense of texture, which is achieved by sending continuous signals (rather than short pulses) to the electrodes.

Electro-tactile feedback of the whole hand without obstruction to the palm side of the handcreated by Yudai Tanaka, Alan Shen, Andy Kong and Pedro Lopez of the University of Chicago, will be on display at CHI 2023 in Hamburg.