project · android · shared runtime

gi-hub — one language model, shared by every app on the phone

A single APK hosts the inference runtime and lends it out over a bound AIDL service. One model on disk, one RAM footprint, one KV cache — and any number of consumer apps.

The first version of Activity Hub loaded model weights inside the app. That works perfectly until you want a second app to do the same thing — and then you are holding two copies of a gigabyte-plus model in RAM on a mid-range phone, paying two cold loads, keeping two KV caches warm.

So the runtime moved out. gi-hub is an APK whose only job is to own the model and answer questions on other apps' behalf.

                    ┌──────────────────────────────────────┐
                    │            gi-hub  (APK)             │
 activity-hub  ──┐  │  ┌────────────────────────────────┐  │
 local-expenses ─┼─►│  │       InferenceService         │  │
 next consumer ──┘  │  │       (foreground, bound)      │  │
                    │  │                                │  │
                    │  │  IInferenceService  streaming  │  │
                    │  │                     + grammar  │  │
                    │  │  ISimpleInference   init /     │  │
                    │  │                     generate / │  │
                    │  │                     shutdown   │  │
                    │  │                                │  │
                    │  │  engines:                      │  │
                    │  │    LiteRT-LM → Gemma 4 E2B-it  │  │
                    │  │    llama.cpp → Qwen 2.5 1.5B   │  │
                    │  └────────────────────────────────┘  │
                    │   one model · one ~1.3 GB footprint  │
                    │   one KV cache                       │
                    └──────────────────────────────────────┘

Consumers bind over Binder IPC and stream tokens across the process boundary. They ship no model code, no weights, and no native inference library. On the client side the whole surface is a typed AsyncIterable.

Getting it from 70 seconds to 7

The first working round trip took about seventy seconds for a single planner prompt on a Pixel 7a. That is not a slow feature, it is a broken one. Four phases later it was seven.

Steady-state means the model is already resident; a cold load adds another 4–10 s. The test prompt mirrors the real workload — about 220 input tokens of chat template plus tool catalogue plus question, expecting ~50 tokens of JSON back.

Why the "better" quantisation was slower

Q4_0 beats Q4_K_M on ARM, which is the opposite of what the naming suggests. ggml's K-quant formats do not have the hand-tuned NEON and sdot kernels that Q4_0 has on arm64, so the baseline was running a generic dequantise-to-FP16 path.

But the fix only lands with the right build flags, and this is the part that cost real time: KleidiAI needs +dotprod in the arch flag, not just =ON. ggml's CMake string-matches ARCH_FLAGS looking for +dotprod / +i8mm, and silently skips compiling the accelerated kernel sources if it doesn't find them. So you enable ARM's GEMM accelerator, measure no change, and reasonably conclude it doesn't help on your workload. It does — it was never in the binary.

Its sibling is a trap in the other direction: +i8mm is a foot-gun on a big.LITTLE Tensor G2, where the core types don't share the same instruction support and the scheduler will happily land you on a core that doesn't want it.

The Vulkan dead end

Phase 3 looks like progress — 30 s down to 18 s — and it is the most misleading number in the table. Vulkan on a Mali-G710 is a bandwidth wash, not a kernel-efficiency win. Memory bandwidth is the ceiling on this class of SoC, and moving matmuls to the GPU does not raise it. Chasing further GPU tuning would have been weeks for nothing; swapping runtime and model was what actually moved the number.

The related finding: multi-token prediction only wins on GPU, never on CPU. On the CPU path it costs more than it saves.

Four things Android did not want me to do

The model file the native runtime cannot read

You adb push the Gemma weights to /sdcard/Android/data/com.gi.hub/files/, because on an unrooted device that is the only writable path. Then LiteRT-LM's native open(2) returns EACCES.

Android 11+ mounts that directory over FUSE for user-space visibility, and native code operating in a different SELinux context cannot traverse it the way the JVM can. The fix is unglamorous: on first load(), stream the file into context.filesDir — real internal storage — then delete the external copy. It is a one-time ~2.6 GB read-and-write costing 10–20 s on a Pixel 7a, and every load after that is free.

AIDL Parcelables are byte-compatible, not source-compatible

The AIDL files are forward declarations; the real Parcelable classes are Kotlin under com.gi.hub.api.*, and every consumer APK must ship byte-compatible copies in the same package path — not merely the same shape.

Fields are positional in the parcel. Append a field with a default and old consumers still parse fine, because the parcel ends short and the default fills in. Insert one in the middle and you get BadParcelableException in every consumer until each is rebuilt. The rule that fell out of that: only append, never insert — and bump INTERFACE_VERSION even for appends so consumers can detect what they are talking to.

Adjacent trap: getInterfaceVersion() is reserved by stable-AIDL, so it is not available as a name for your own versioning.

Cross-app Binder needs a shared signing key

com.gi.hub.permission.BIND_INFERENCE is declared protectionLevel="signature", so the OS grants it only to apps signed with the same key as the declarer. Consumers and gi-hub must therefore share a keystore.

In debug this works without any setup, which is precisely the danger — Android ships one debug keystore per machine, so everything signed locally matches by accident. The first release build where one APK is signed with a different key, every cross-app bind fails at the OS level with no error a user could act on.

The native module is generated, not written

expo prebuild --clean regenerates the whole android/ tree, so anything hand-edited there is already gone. The AIDL files, the mirrored Parcelables and the React Native bridge module are all written into place by an Expo config plugin on every prebuild, from a read-only mirror of the gi-hub source. The plugin hard-fails with a pointer to the sync script if that mirror is missing — a loud failure being much cheaper than a stale interface.

Design calls worth defending

Engine selection is global, not per-consumer. The service hosts exactly one engine at a time and the user picks it from gi-hub's admin screen — consumer apps do not choose. Two engines resident at once would defeat the entire reason the service exists, and per-app engine choice would mean every consumer shipping its own opinion about which model is best, fragmenting the shared-runtime story. The AIDL call is still there for diagnostics; it is just not exposed in consumer UIs.

Idle-unload is five minutes. When the last client unbinds, a timer starts; if it fires with no rebind, the engine unloads and 1.3–2.6 GB comes back. Thirty seconds recovers RAM fast but makes every background-then-foreground cycle pay a 4–10 s cold load. Thirty minutes never goes cold but holds RAM through long idle gaps. Five minutes keeps the common "back to the app" cycle warm, and there is a manual unload button for people who want their memory back sooner.

Grammars are a hint, not a guarantee. llama.cpp enforces GBNF hard. Gemma 4 through LiteRT-LM does not honour it at all. Since the runtime is swappable, no caller may assume constrained output — so JSON extraction and repair at the orchestrator level is the real safety net on both engines, and the grammar is an optimisation where it happens to work.

Cancellation lands on token boundaries. stream.cancel() over Binder is not instant; it takes effect at the next token. Any UI that pretends otherwise will look broken.

Where it stands

Both AIDL surfaces are implemented with a real JNI bridge to llama.cpp alongside the LiteRT-LM path, and two consumers bind to it today: Activity Hub for its Ask Anything pipeline, and Local Expenses for transaction extraction and promotion classification.

The thing I did not expect going in is how little of the work was model work. Almost all of it was process boundaries, file permissions, build-flag archaeology and versioning discipline — ordinary systems engineering that happens to have a language model at the far end.