I spent the last several months building a photo journal that runs a full multimodal LLM — Gemma 4 E4B — entirely on an iPhone. No cloud inference, no accounts, no analytics, full privacy; the App Store privacy label is "Data Not Collected," because there is nowhere for anything to go. This post is about the engineering: what actually fits on a phone in 2026, what breaks, and the findings I haven't seen published anywhere else.
The memory wall comes first
Everything on iOS starts with jetsam: the OS kills apps that exceed roughly half of device RAM. Apple's increased-memory-limit entitlement raises the ceiling, but not past what the OS keeps for itself. So on an 8 GB phone the real working budget is maybe 5–6 GB — for weights, KV cache, the vision encoder's transient spikes, and the entire app.
This budget decided the model. Gemma 4 E4B is a MatFormer — ~8B params with an effective 4B active — and the .litertlm package is 3.66 GB (3,659,530,240 bytes; I've stared at that number a lot, see "shipping weights" below). It fits on the 8 GB class with the entitlement. It fails on anything less, which is why the app gates to A17 Pro-class devices and later.
The vision encoder fires the jetsam trap specifically: image encoding is a memory spike, not a plateau. If you're going to die, you die mid-encode. So the app budgets for the spike, not the average.
The runtime fork: llama.cpp vs LiteRT-LM
I came in loyal to llama.cpp — it's the same engine I run on the Mac, one runtime everywhere, GGUF, done. In reality: the full multimodal GGUF config (Q4_K_XL + F16 mmproj) is ~3.6 GB for the smaller E2B model, and the projector alone is nearly a gigabyte. On a 6 GB phone it jetsammed; trimming quants bought margin, but barely, and E4B multimodal was out of reach entirely. That was a setback; I moved on.
Google's LiteRT-LM path packages the same family mobile-first. That was promising. The whole E4B multimodal stack came in one 3.66 GB .litertlm. That won the day — with trade-offs that became half of this post because the mobile-optimized path is also the less-documented one.
The GPU backend is not optional
On the CPU, prefill was unusable for anything interactive: first token on a long prompt took ~14 seconds. The Metal GPU backend brought that to ~1.4s — about 10× on prefill — and that made the product viable. Decode speed matters less than everyone thinks for this workload; prefill latency is what the user notices. Every single ask re-feeds context.
The numbers, from the app's own journal — medians over 790 real generations on my iPhone 17, GPU path, E4B:
| What | Median |
|---|---|
| First token, text-only prompt (CPU → GPU) | ~14 s → ~1.4 s |
| Photograph + prompt → first word (vision prefill) | 3.5 s |
| Decode | ~16 tok/s |
| A whole two-sentence photo reading | 6.6 s |
| A conversational turn with a ~1,000-token tool result in context | 12–15 s |
Not a benchmark harness — production telemetry, which is the number that actually matters.
Findings I haven't seen published
1. The runtime's sampler defaults are greedy — and your config may be silently discarded
Two separate discoveries here.
First: LiteRT-LM's default sampling is effectively topK=1 — greedy — and I found the randomSeed parameter is ignored. I verified by generating with different seeds and hashing outputs: md5-identical generations, run after run.
Second, and nastier: the Flutter wrapper I use (flutter_gemma / flutter_gemma_litertlm) silently dropped the sampler config entirely on iOS in versions ≤1.1.0 — temperature, topK, seed, all discarded at the boundary. I proved it to myself the blunt way: identical output at temperature 0.3, 0.6, and 5.0. If temperature 5.0 doesn't change your output, your temperature isn't reaching the runtime. Fixed upstream in 1.3.1 (native v0.14; I ship on v0.16 today). If you're building on any wrapper stack: test that your sampling params actually arrive — generate at absurd temperature and diff. Doing this and in ten minutes it explained weeks of "why does this model repeat itself so deterministically."
2. A 4B model cannot reliably retype digits — so I stopped letting it
This is the finding I'd most want another on-device builder to have.
E4B, asked to repeat dates from context — not compute, retype — garbles digit sequences in ways no repair layer I built could chase: "2022" comes back "20222" (doubling), "October 20–22" becomes "October 209-22" (insertion), digit groups split and merge. The error shapes are too numerous to fix via pattern-matching. Words, meanwhile, come back clean. The failure is in retyping digit sequences, not in knowing the date.
The fix I chose: digits never cross the model boundary, in either direction. Dates are converted to words before the prompt ("October fifteenth, two thousand twenty-two"), the model speaks words, and a deterministic layer converts worded dates back to digits for the screen. Anything unparseable stays worded — visibly odd beats silently wrong.
Two traps sprouted from this fix:
- Repetition is the real failure mode, not digits per se. "Twenty twenty-two" came back "twenty twenty twenty-two" — the doubled word is the same defect as the doubled digit. Year forms with no repeated adjacent words ("two thousand twenty-two", "nineteen hundred nineteen") are safe; forms with internal repetition are not.
- Translation is composition. In Spanish, the model rendered a year as "dos mil novecientos veinticuatro" — 2,924. A model can copy in-context words safely in any language. However, if it must translate it must also compose, and composition is where small models garble. So the worded dates are generated in the user's language before the prompt — the model copies, never converts.
I ended up with a per-language date grammar (generate + parse, tested for round-trips and the no-repeated-words invariant) for every UI language. Tedious, deterministic, and it closed the bug class entirely.
3. The engine can wedge unrecoverably — design for the brick, not the retry
Many times in weeks of heavy use, a generation entered prefill and simply never returned. No token, no error — and the subsequent cancel I put in also never returned. A native thread stuck mid-call is unsalvageable from the host language: there is no in-process recovery, and trying otherwise (retries, re-inits) just queues more work behind a corpse. The first occurrence poisoned the entire generation queue: every surface in the app waited forever on an engine that looked idle.
The design that worked: a teardown deadline. If a cancelled generation's teardown doesn't return within 8 seconds, the engine is declared wedged — permanently, for this process. Every queued and future request fails fast with an explicit "restart required," the UI says so once, and a restart is clean.
4. Throughput shapes the product more than intelligence does
A photo "reading" (multimodal: image + prompt → a few sentences) costs ~13 seconds on the GPU path. A 2,500-photo library is therefore an overnight job, not a progress bar — so the architecture became brief-first, two-phase: every photo gets a fast two-sentence brief (whole library searchable in hours), then a second pass rewrites them as full readings, newest-first, forever in the background. Reading runs while charging by default.
5. The failures that matter are model × data — and unit tests can't see them
Two days before launch, "tell me about my trip to Japan" — an ask that had worked for months — started refusing, reliably. Nothing in the routing code had changed in ten days. What changed was the data: the library had finished reading. A coverage gate (answer from a computed card while a scope is under a third read; hand it to the model once readings are deep) graduated the scope, and graduation handed a whole ask class to the free model — whose travel-assistant prior refuses trip narratives. Every component behaved to spec. The failure was emergent.
Three things I'd tell anyone shipping a small model behind a router:
- *Unit tests are modelless, so they pin design decisions*, not outcomes.** My suite literally pinned that exact ask to "the model narrates" — the bet was formalized, and when it lost, the tests went on passing.
- A 4B shapes quotes; it must never filter rows or do arithmetic. Every time I let the model decide which data, it eventually chose none. The rules live in code; the model gets the voice. (Corollary: coaching the model out of a behavior with prompt text never held — persona + facts + trust worked, rulebooks didn't.)
- **Run a battery through the real path, with the real model, on the real data.** Mine is ~150 asks (plus a second, natively authored Spanish set) driven through the live app on the phone — router, gate, tools, voice — auto-scored only for structure (refusal on a scoped ask, empty, token salad, digit garble, timeout, route drift). About an hour on an iPhone 17, unattended. Route drift — the observed path differing from the annotation — is the approach that catches the graduation flip.
Shipping 3.66 GB of weights
App Review won't let you bundle it (4 GB uncompressed app cap, and every update would re-ship it), so the model downloads on first run. I serve it from Cloudflare R2 — zero egress fees, which matters when every install pulls 3.66 GB — with range requests so resumed downloads work. Verify your mirror byte-for-byte and mind the license terms when redistributing weights (Gemma's terms require passing conditions to recipients — the "Built with Gemma" notice on the About screen).
Was it worth it?
The whole point was a question: can you get the "ask your own life anything" experience — the thing everyone assumes needs a datacenter — with the photos never leaving the phone? On 2026 hardware, with a 4B MatFormer, a GPU backend, and a lot of respect for the failure modes above: yes. The colloquy answers questions like "show me mountains in my travels through Europe?" from 2,500 photographs in seconds, with the receipts, offline in airplane mode.
The app is Amoli (amoli.app) — a private photo journal, on the App Store: https://apps.apple.com/app/amoli/id6794453443. But the reason I wrote this up is the findings: greedy defaults, discarded sampler configs, the digit problem, wedge-proof teardown, and the model × data failure class are all things I wish someone had published before I found them the slow way.
The app is built on Juice, my open-source Flutter framework — state in blocs, logic in use cases, and the juice_llm packages that drive the on-device engine. It's all on pub.dev if you're building something similar.
Happy to answer anything about the stack.