On-Device AI Craves RAM

Ron Amadeo, writing for Ars Technica on a purported leak of a trio of Pixel 9 phones:

Rozetked says (through translation) that the phone is “similar in size to the iPhone 15 Pro.” It runs a Tensor G4 SoC, of course, and — here’s a noteworthy spec — has a whopping 16GB of RAM according to the bootloader screen. The Pixel 8 Pro tops out at 12GB.

Anything could change between prototype and product, especially for RAM, which is usually scaled up and down in various phone tiers. A jump in RAM is something we were expecting though. As part of Google’s new AI-focused era, it wants generative AI models turned on 24/7 for some use cases. Google said as much in a recent in-house podcast, pointing to some features like a new version of Smart Reply built right into the keyboard, which “requires the models to be RAM-resident” — in other words, loaded all the time. Google’s desire to keep generative AI models in memory means less RAM for your operating system to actually do operating system things, and one solution to that is to just add more RAM. So how much RAM is enough? At one point Google said the smaller Pixel 8’s 8GB of RAM was too much of a “hardware limitation” for this approach. Google PR also recently told us the company still hasn’t enabled generative AI smart reply on Pixel 8 Pro by default with its 12GB of RAM, so expect these RAM numbers to start shooting up.

That last link is to a story positing that Google’s Gemini Nano runs on the Pixel 8 Pro but not the regular Pixel because the Pro has more RAM (12 vs. 8 GB).

Comparing iPhone RAM to Android RAM has never been apples-to-apples (same goes for battery capacity), but still, it’s hard not to wonder whether Apple’s on-device AI plans are hamstrung by the relatively stingy amounts of RAM on iPhones. Here’s a list from 9to5Mac showing the RAM in each iPhone going back to the original (which had just 128 MB!). iOS 17 supports models dating back to 2018’s iPhone XS and XR (4 and 3 GB of RAM, respectively). If iOS 18 drops those models, the new baseline will be the iPhones 11 and 11 Pro, which all sport 4 GB. The most RAM on any iPhone to date is 8 GB in the 15 Pro models, but 8 GB is what Google deemed insufficient for the Pixel 8 to run Gemini Nano.

Might some iOS 18 on-device AI features be limited to newer models with more RAM?

Update: Follow-up post regarding base Mac memory.

★ Monday, 22 April 2024