"On-device AI on Android" usually means a Pixel or a flagship Samsung in a developer's pocket. The Indian mass-market device profile is different: phones such as the Redmi 14C, the Realme C-series and Samsung's Galaxy A1x models, commonly with 4GB of RAM (sometimes padded with "virtual RAM" that is really storage), entry-level MediaTek, Unisoc or Snapdragon 4-series chips, and storage that fills up every month. Models that run beautifully on a Pixel can be killed by the system on these phones. Here is the constraint set to engineer against.
Working budgets for this device class
These are planning targets rather than platform limits. Measure on your own test devices and adjust.
- Memory: keep peak model memory to a few hundred MB at most; 150–300MB is a sensible target. The OS, your app and other background apps share the rest of 4GB, and Android's low-memory killer will end your process mid-inference if you push it.
- Download size: aim for a model file in the tens of MB, not hundreds. Users on these phones uninstall apps to free storage.
- Cold start: aim for well under a second from tapping the feature to the first result, loading the model lazily and off the main thread.
- Battery: sustained ML use should not drain the battery faster than everyday activities such as video playback, or users will notice your app in the battery settings.
What survives these constraints
- Quantised vision models for OCR, document detection and simple classification. INT8 MobileNet and EfficientNet-Lite class models work well.
- Small embedding models for on-device semantic search over the user's own content. Compact sentence-embedding models quantised to INT8 fit the budget.
- Small classifiers and detectors for utility tasks: language detection, content safety filters, simple intent classification.
- Speech-to-text via platform APIs. Android's speech recogniser can work offline on many devices for supported languages. Check availability on your target devices before shipping your own model.
What does not work on this profile
On-device LLMs for free-form generation. The 1–3B parameter models that run on flagships either don't fit in memory alongside your app, generate too slowly to be usable, or heat the phone. Platform-provided on-device models, where they exist, are currently limited to higher-end devices. Use the cloud for generation and on-device models for everything else. The hybrid pattern is practical; the all-on-device approach is not, for this device class, today.
Quantisation and runtime choices
- INT8 post-training quantisation is the usual starting point. It stores weights in a quarter of the space FP32 needs, usually with small accuracy loss for vision and classification models. Validate accuracy on your own test set after quantising.
- LiteRT (formerly TensorFlow Lite) is the most widely supported runtime across Android chipsets. Treat the XNNPACK CPU path as the baseline; GPU delegates help on some chips and not others. NNAPI is deprecated from Android 15, so don't build new work around it. See the LiteRT documentation.
- ONNX Runtime Mobile is a better fit if you also ship iOS and want one model format.
- MediaPipe is the fastest path for face detection, hand tracking, pose estimation and similar ready-made tasks.
- MLC LLM and similar on-device LLM runtimes are worth evaluating only for flagship-tier devices, not this mass-market profile.
Memory discipline in the app
- Load the model on first use, not at app start, and release it when the feature closes.
- Respond to
onTrimMemory()callbacks by freeing model buffers. - Check
ActivityManager.isLowRamDevice()and serve a smaller model or a cloud fallback on low-RAM devices. - Download larger models after install, on Wi-Fi, with a checksum and resume support, rather than bloating the app package. Android's guide to managing your app's memory covers the platform side.
Testing on the right devices
If your test rack is an iPhone 15 Pro and a Pixel 9, you know little about how your app behaves for most mass-market Indian users. Build a rack of entry-level phones, for example the Redmi 14C, Samsung Galaxy A15, Realme C61 and itel A70, covering MediaTek, Unisoc and Snapdragon chips. Test with other apps open and the phone warm, not freshly rebooted. Performance and out-of-memory behaviour under those conditions is what counts.
Battery behaviour matters as much as latency
Android shows users which apps use the most battery, and people check it when a phone runs flat early. An app near the top of that list becomes a candidate for uninstalling. Measure energy per task with Android's power profiling tools, batch inference where you can, avoid keeping the model warm in the background, and never run inference on camera frames you don't need.
A worked example (hypothetical)
Say a lending app wants on-device document capture: detect the ID card's edges, check for blur, and read the card number before upload. A quantised detector and a small OCR model, each in the tens of MB, fit the budget and reduce failed uploads. Judging whether an applicant's free-text explanation is coherent is a generation-and-reasoning task, so send it to a cloud model rather than trying to squeeze an LLM onto a 4GB phone.
How we approach this at Velura Labs
Our Mobile App Development service tunes on-device AI for the mass-market device profile, not just flagships. Pair it with Custom LLM Applications when you need hybrid on-device and cloud inference. Read our Flutter vs React Native comparison for the framework decision and Bharat design patterns for the broader UX framing. Talk to us if retention drops sharply on entry-level Android; an ML feature is a likely culprit.
Velura Labs delivers this for teams across the United States — Seattle (Washington), San Francisco and Los Angeles (California), Austin and Dallas (Texas), and New York — as well as Europe (Paris, Milan, Rome and the wider EU), the Middle East (Dubai, Abu Dhabi and Riyadh) and India. Talk to us wherever you operate.