Simple Tricks and Nonsense

Friday, 2 February 2024

Upon launching Apple’s Encounter Dinosaurs app for Vision Pro — an immersive 3D experience — you see a butterfly a few feet in front of you. As it flutters about the space in front of you, if you reach out your hand, the butterfly will land on your extended finger. When I first experienced this demo back in June at WWDC, reps from Apple said — half boasting, half warning (both halves well-warranted) — that when the butterfly lands on your finger, some people actually feel it. They feel a physical tickle on their finger. I didn’t feel that, but some of my colleagues in the media who got the same demo did. They perceived a physical sensation from what is, ultimately, an optical illusion.

When an illusion is realistic enough, your brain accepts it as real, even though it’s not — and even though you know it’s not.

One generally underappreciated aspect of the iPhone is the accuracy of its finger-tracking for touch. From the beginning — the original iPhone on the day it launched — when you slide your finger on an iPhone to scroll, the scrolling tracks your finger movement exactly as though you were sliding a piece of paper. The delight began right from “Slide to Unlock”, because the knob you slid to unlock the iPhone tracked your finger precisely. It didn’t feel like you were issuing an indirect shortcut gesture; it felt (and to this day, of course, still feels) like you were directly manipulating whatever it was you were sliding or scrolling on screen. The bouncing when you scroll to the end of a view. The stretching when you pull down from the top of a view. It felt (and feels) real, to some degree.¹

The experience of manipulating the user interface in VisionOS conveys something like this too. It’s not direct manipulation, though. You don’t reach out and grab a window bar to move the window. You just look at the bar under the window, pinch your finger and thumb, and move your hand to move the window. It is as easy to pull a window toward you, or push it away, as it is to move it side to side. The same is true for resizing windows from the grabber that appears when you look at any corner, and it’s also true for using two hands — both making the finger-thumb pinch — to zoom the content (like, say, a photo) within a window. There’s no lag and there is tremendous precision. You can arrange, position, stack, and resize windows exactly the way you want them.

One comparison would be to a maestro conducting an orchestra. But that doesn’t convey the sense of precision in VisionOS — the sense of fine control through indirect manipulation. What it feels like is using the Force.

One of the first times we see any character use the Force in Star Wars² is when Obi-Wan Kenobi uses the Jedi mind trick to convince the droid-hunting stormtroopers on the outskirts of Mos Eisley that R2-D2 and C-3PO are not, in fact, the droids they’re looking for. He even pinches his finger and thumb together while gesturing with his hand. It couldn’t be more apt an image.⁴

In the very scene preceding Kenobi’s mind trick, we see Darth Vader using the Force to choke his vainglorious colleague Admiral Motti, with ... a pinching gesture: “I find your lack of faith disturbing.”

Kenobi, conjuring an audio distraction to escape the attention of two stormtroopers:

George Lucas didn’t conceive of telekinesis, but his portrayal of it made it seem real, and defined how my and subsequent generations imagined they would invoke such an ability if they could. And now, in a sense — as hokey as this sounds — I feel like I can. Pulling virtual application windows through each other. Pushing them back against the wall. Expanding them. Shrinking them. Moving one outside my living room through a real glass window. Activating buttons and flipping switches just by looking at them, and gesturing.

And, like the butterfly from Encounter Dinosaurs that some people can feel landing on their outstretched finger, I swear to you, I can almost feel the telekinetic connection with UI elements in VisionOS. It’s a hint, a whiff, of tension — between not just my hands and the virtual elements I’m manipulating, but between my mind and those elements. Just the vaguest sensation of tension emanating from my forehead, like a taut thread of ultrafine string connecting my mind to the window I’m moving, or button I’m pressing, or photo I’m stretching.

With Vision Pro, the gestures are necessary, because the device can’t actually read your mind. (At least not in this first-generation version.) In Star Wars, the gestures exist not because they’re required to perform mystical feats of mentalism, but to make those actions cinematic. It is utterly natural to express ourselves via our hands, and to read meaning from the hand gestures of those we see.⁵ As good a job as the original film does conveying this, the most evocative shot exemplifying the use of a pinching gesture to convey telekinetic Force ability comes from 2016’s Rogue One, when Vader — who, whatever his flaws, never loses his sense of humor — offers Orson Krennic, father of the Death Star, this helpful advice: “Be careful not to choke on your aspirations, director.”

Setting aside potential trademark complaints from their friends at Disney, this is what Apple should have saved the term “Force Touch” for.