On-device Inference Articles

Isolated-Process Inference for Android On-Device AI

Use a separate Android process and AIDL to isolate model memory and native crashes while preserving reliable inference lifecycle management.

Observability for Android On-Device AI Inference

Trace each on-device inference, build multi-dimensional performance profiles, and detect production regressions across fragmented Android hardware.

ONNX Runtime on Android: Export, Quantize, and Run Models

An end-to-end guide to ONNX Runtime for Android, from export and integration through INT8 acceleration and operator compatibility.

Deploying ExecuTorch Models on Android

A practical ExecuTorch path from native PyTorch export to `.pte` deployment on Android, including AOT delegates and runtime trade-offs.

Android On-Device LLM Latency: From Tap to First Token

Break down and optimize the full Android on-device LLM path, from user input and model loading to prefill, decode, and first-token rendering.

Multi-Model Orchestration for Android On-Device AI

Design Pipe-style workflows for multiple Android models, with routing, fallback, model management, and coroutine scheduling.

Choosing an Android On-Device AI Runtime: LiteRT to ExecuTorch

A production-oriented comparison of LiteRT, MediaPipe, ExecuTorch, ONNX Runtime, and llama.cpp for Android inference.

On-Device LLM Tokenization and Decoding on Android

How BPE tokenization, Top-K, and Top-P sampling shape quality, latency, and memory use in Android on-device AI.

Android Multimodal On-device AI: Gemini Nano, Image Tokens, Streaming, and Compose

A practical Android multimodal AI walkthrough covering Gemini Nano Multimodality, AICore model loading, image preprocessing, ViT tokenization, streaming inference, Compose rendering, memory, and thermals.

Designing an On-Device LLM Inference Scheduler: Priority Queues and Backpressure in Practice

This article shows how to build a scheduling layer above an on-device inference engine, using priority queues, preemption, and backpressure to avoid OOMs, unpredictable latency, and out-of-order results.

Android Local LLM Inference: LiteRT, MediaPipe, Quantization, and Production Trade-offs

A practical guide to Android local LLM inference across LiteRT, ONNX Runtime Mobile, MediaPipe LLM Inference, INT4 quantization, GPU delegates, KV cache memory, and device fallback.

Android On-device LLM Context Window Engineering

A practical Android on-device LLM context management strategy covering layered prompt compression, summary caching, dialog state machines, and token budget allocation.

Inside Android TTS: From TextToSpeech API to On-Device Vocoders

A full-stack look at Android TTS, including engine binding, synthesis callbacks, on-device HiFi-GAN vocoders, streaming playback, and TTFA tuning.

Android ML Kit in Practice: Vision Pipelines and CameraX Integration

A practical guide to ML Kit for on-device vision, covering detection pipelines, CameraX integration, multi-model orchestration, and inference optimization.

Android AICore and Gemini Nano: System Services, Safety Filters, and LoRA Adaptation

A deep dive into Google AICore for Gemini Nano on Android, covering APEX delivery, permission isolation, model distribution, safety filtering, session management, and LoRA adapters.