Simon Willison on the Privacy/Security Risks of Personalized Siri, vis-à-vis Prompt Injection

Simon Willison:

These new Apple Intelligence features involve Siri responding to requests to access information in applications and then performing actions on the user’s behalf.

This is the worst possible combination for prompt injection attacks! Any time an LLM-based system has access to private data, tools it can call and exposure to potentially malicious instructions (like emails and text messages from untrusted strangers) there’s a significant risk that an attacker might subvert those tools and use them to damage or exfiltrate a user’s data.

I published this piece about the risk of prompt injection to personal digital assistants back in November 2023, and nothing has changed since then to make me think this is any less of an open problem.

Prompt injection seems to be a problem that LLM providers can mitigate, but cannot completely solve. They can tighten the lid, but they can’t completely seal it. But with your private information, the lid needs to be provably sealed — an airtight seal, not a “well, don’t turn it upside down or shake it” seal. So a pessimistic way to look at this personalized Siri imbroglio is that Apple cannot afford to get this wrong, but the nature of LLMs’ susceptibility to prompt injection might mean it’s impossible to ever get right. And if it is possible, it will require groundbreaking achievements. It’s not enough for Apple to “catch up”. They have to solve a vexing problem — as yet unsolved by OpenAI, Google, or any other leading AI lab — to deliver what they’ve already promised.

So Apple had promised for this year — and oft promoted — an entire set of features that they not only have now acknowledged will not ship this year, but which they might, in fact, never be able to ship. Makes me wonder how many people inside Apple were voicing these concerns a year ago, and why they lost the debate to start promising these features last June and advertising them in September.

★ Saturday, 8 March 2025