By John Gruber
Become smarter in just 5 minutes. Subscribe to Morning Brew.
Three weeks ago, writing for The Guardian, Alex Hern reported:
Apple contractors regularly hear confidential medical information, drug deals, and recordings of couples having sex, as part of their job providing quality control, or “grading”, the company’s Siri voice assistant, the Guardian has learned.
Although Apple does not explicitly disclose it in its consumer-facing privacy documentation, a small proportion of Siri recordings are passed on to contractors working for the company around the world. They are tasked with grading the responses on a variety of factors, including whether the activation of the voice assistant was deliberate or accidental, whether the query was something Siri could be expected to help with and whether Siri’s response was appropriate.
Apple says the data “is used to help Siri and dictation … understand you better and recognise what you say”.
But the company does not explicitly state that that work is undertaken by humans who listen to the pseudonymised recordings.
I pooh-poohed this story at first, mostly on the grounds that I thought we knew about this, and that the recordings were only saved from users who had consented to it. I was mistaken. This is a privacy fiasco, and a betrayal of Siri users’ trust.
A week later, Apple issued statements to TechCrunch and The Verge stating that it was suspending this “grading” program. From Matthew Panzarino’s report at TechCrunch:
Apple says it will review the process that it uses, called grading, to determine whether Siri is hearing queries correctly, or being invoked by mistake.
In addition, it will be issuing a software update in the future that will let Siri users choose whether they participate in the grading process or not.
My reading of this is that until last week, if you used Siri in any way, your recordings might be used in this “grading” process. If I graded Apple on the privacy and trust implications of this, I’d give them an F. I don’t think it’s debatable whether users of any voice assistant should have their recordings listened to or even reviewed (in text form) by human employees without their express consent. But especially users of Siri, given Apple’s prominent position as a privacy focused company. Apple literally advertises on the basis of its user-focused privacy policies — but apparently the billboards should have read “What happens on your iPhone stays on your iPhone, except for some of your Siri recordings, which we listen to.”
Apple did not comment on whether, in addition to pausing the program where contractors listen to Siri voice recordings, it would also stop actually saving those recordings on its servers. Currently the company says it keeps recordings for six months before removing identifying information from a copy that it could keep for two years or more.
Until the opt-in process is crystal clear, Apple should delete all existing recordings and confirm that it is no longer saving them. I don’t even know where to start with the fact that until this story broke, they were keeping copies with identifying information for six months. This defies everyone’s expectations of privacy for a voice assistant.
We should expect Apple to lead the industry on this front, but in fact, they’re far behind. Amazon has a FAQ written in plain language that explains how Alexa works, and how to view your voice recordings from Alexa-powered devices. You can review them in the Alexa app in Settings: Alexa Privacy (a pretty obvious location) or on the web. That settings page also has an option: “Use Voice Recordings to Improve Amazon Services and to Develop New Features”. I think Amazon should make clear that with this turned on, some of your recordings may be listened to by Amazon employees, but it’s not too hard to surmise that’s what’s going on.
Apple offers no such setting, and offers absolutely no way to know which, if any, of our Siri recordings have been saved for review by employees. This is something we should have explicit, precise control over, but instead it’s a completely black box we have no control over or insight into whatsoever.
From a privacy perspective, there are two fundamental types of Siri interactions: purposeful and accidental. Purposeful interactions are when you press the side button or say “Hey Siri” with the intention of invoking Siri. Accidental interactions occur when the button is pressed too long accidentally, or when a device incorrectly hears “Hey Siri” even though you said no such thing. All recorded Siri interactions should be treated by Apple with extraordinary care, but accidental invocations, when identified, should be deleted immediately unless the user has expressly agreed to allow it — each and every time. Having Apple contractors listen to random conversations or audio is the nightmare scenario for an always-listening voice assistant.
Compare and contrast with iOS’s transcript feature for voicemail. At the bottom of each transcription, iOS asks whether the transcription was “useful” or “not useful”. Tap on either of those and you get a very explicit prompt:
Help Improve Transcriptions?
Would you like to submit this voicemail to Apple to improve transcription accuracy?
Recordings will only be used to improve the quality of speech recognition in Apple products.
Do not submit recordings if you believe the speaker would be uncomfortable with you submitting the content to Apple.
The two buttons at the bottom of the prompt: Cancel and Submit. You must address this same prompt every single time you flag a transcription as useful or not useful. Every time. That’s how you do it.
In addition to being correctly respectful of privacy, the voicemail transcription feature also puts the user in control. So when a voicemail is transcribed poorly, you can flag it and submit it to Apple. That would be a great feature for Siri — when an interaction goes poorly, and we know the interaction was innocuous in terms of revealing anything private, we should be able to flag it and submit it to Apple. I firmly believe that Siri has gotten far more useful and far more accurate in the last few years, but clearly it’s still very far from perfect. I’d be happy to help Apple by submitting failed interactions on a per-interaction basis. Apple needs to stop pretending Siri is perfect.
I’ll give the final word to Steve Jobs, speaking about privacy back in 2010 at Kara Swisher and Walt Mossberg’s D8 conference:
“Privacy means people know what they’re signing up for, in plain English and repeatedly. I believe people are smart and some people want to share more data than other people do. Ask them. Ask them every time. Make them tell you to stop asking them if they get tired of your asking them. Let them know precisely what you’re going to do with their data.”
I can’t say it any better than that. ★
Last week was a good week for privacy. Or was it?
It took an article I almost didn’t publish and tens of thousands of people saying they were creeped out, but Superhuman admitted they were wrong and reduced the danger that their surveillance pixels introduce. Good on Rahul Vohra and team for that.
I will say, however, that I’m a little surprised how quickly some people are rolling over and giving Superhuman credit for fixing a problem that they didn’t actually fix. From tech press articles implying that the company quickly closed all of its privacy issues, to friends sending me nice notes, I don’t think people are paying close enough attention here. This is not “Mission Accomplished” for ethical product design or privacy — at all.
If you haven’t been following this saga from earlier this month, it’s well worth your time to read the whole thing, including Davidson’s original post and Superhuman CEO Rahul Vohra’s genuinely thoughtful — but ultimately unsatisfying — response.
Basically, Superhuman is an invitation-only Gmail front-end whose users seem to genuinely love it. But they embed tracking pixels in emails by default, and use these pixels to show the sender when (and until last week, where, which is truly fucked up) the recipient views them. They call them “read receipts”, and functionally they do work like read receipts, insofar as they indicate when you read a message. But real email read receipts are under the recipient’s control, and they’re a simple binary flag, read or unread — they don’t tell the sender how many times or when you view a message.
I know that mailing list software generally includes tracking pixels. I don’t think that’s ethical either. On a personal level, though, with Superhuman, tracking when and how many times a recipient views a message is simply absurdly wrong.
It’s also something the vast, overwhelming majority of people don’t even realize is possible. I’ve told the basic Superhuman tracking story to a few people over the last few weeks, and asked whether they realized this was possible; all of them expressed shock and many of them outrage as well. Email should be private, and most people assume, incorrectly, that it is. You have to be a web developer of some sort to understand how this is possible. Email is supposed to be like paper mail — you send it, they get it, and you have no idea whether they read it or not. It bounces back to you if they never even receive it, say, because you addressed it incorrectly. The original conception of email is completely private.
But also, the original conception of email is that messages are plain text. No fonts, no styles, just plain text, with optional attachments. But those attachments are embedded in the message, not pulled from a server when the message is viewed.
Once we allowed email clients to act as de facto web browsers, loading remote content from servers when messages are viewed, we opened up not just a can of worms but an entire case of canned worms. Every privacy exploit for a web browser is now a privacy exploit for email. But it’s worse, because people naturally assume that email is completely private.
Read receipts should be under the control of the recipient, not the sender. Full stop. The strength of email is that it is open and decentralized, but that’s email’s weakness too. No closed messaging platform that I’m aware of allows for read receipts that are controlled by the sender, not the recipient.
I think Superhuman should be ashamed of themselves for building this feature in the first place — particularly the geo-tracking. But ultimately, email clients should defend against this. The fact that this nonconsensual tracking is even possible should be treated as a serious bug in all email clients. Apple Mail — both on Mac and iOS — allows you to disable loading of remote images as a preference, but that breaks most graphically rich emails. Mail clients should allow remote images but load them anonymously, through a proxy server perhaps. I’m sure it’s a tricky problem to solve, but I’m convinced it can be solved.
Email should be every bit as private as people assume that it is. ★
The fallout from Zoom’s massive webcam vulnerability continues. In a report published today, security researcher Karan Lyons shows that the same flaw — which gave attackers easy access to laptop cameras and microphones — affects RingCentral, which is used by over 350,000 businesses, as well as Zhumu, essentially the Chinese version of Zoom.
On July 16, Apple confirmed that it had released another silent update to Macs patching the vulnerability affecting Zoom’s partner apps. The update, which went out this morning, requires no user action, but may take some time to roll out to all impacted Macs. Lyons tweeted that Apple’s latest update takes action on 11 different apps, all vulnerable to the Zoom webcam flaw.
So here’s an interesting question. I’ve been using the phrase “nonconsensual technology” to describe Zoom’s invisible web server that remained installed and running even after you deleted the Zoom app. But when Apple first issued a silent, emergency system update to remove Zoom’s software, a few DF readers emailed or tweeted to ask: Isn’t this “nonconsensual technology” too?
Clearly, the answer sounds like yes at first. Users get no indication of the update, and “requires no user action” makes it sound like it’s mandatory. But there is a setting to control this, allowing Mac users to disable the automatic installation of such updates. On MacOS 10.14 Mojave, it’s in System Prefs → Software Update → Advanced (screenshot); on 10.13 High Sierra, it’s in System Prefs → App Store (screenshot). In both versions, the checkbox is labeled “Install system data files and security updates”, and resides at the bottom of the section that controls what gets installed automatically.
This option is enabled by default — even if you choose to install regular system updates manually — which is why the vast majority of Mac users are getting these “silent” updates automatically. But if you disable this option, even these silent updates won’t be installed automatically. I confirmed this with an Apple spokesperson, who emphasized that Apple only issues such updates “extremely judiciously”. Any pending security updates will be installed the next time you manually update software.
I think Apple has struck a nearly perfect balance here, between doing what’s right for most users (installing these rare emergency updates automatically) and doing what’s right for power users who really do want to control when updates — even essential ones — are installed. I also think Apple is doing the right thing by going to the press and explaining when they issue such updates. If I could tweak anything, it would be to have these updates show up in the regular list of pending software updates if you have “Install system data files and security updates” turned off. ★