AI/LLM on-device capability & information privacy

Essay by Will Dzierson · June 10, 2025

Working with on-device AI/ML models to preserve privacy (especially in the space of healthcare/therapy) is only getting more important... Now that we've moved very solidly into the era of AI-based therapists (or the very least, "sympathetic ears "-- https://lnkd.in/euTqBi82), it's more important than ever to be sure this data remains as private as possible.

To bring it back to AI Governance: the ability to share that "AI on-phone functionality does not transmit patient information to outside (i.e. OpenAI) foundation models" is an important policy to stand behind.

With the news that Google is planning an on-device Gemma Nano implementation for iOS (already available for Android), folks (especially within healthcare/behavioral health) are thinking more than ever about the fact that, up until now, having LLM-like functionality on-device has been somewhat prohibitive without leveraging a foundation model like OpenAI, Anthropic, or Llama (each of which requires a network call to an API).

Good news, though! I took the weekend to experiment with some of the already available on-device LLM, OCR, image recognition, and embedding creation models -- and came away encouraged to say the least.

With the help of Cursor and XCode's terminal tools, as well as the excellent https://lnkd.in/ehGu9eha, I was able to get something up and running relatively quickly with the attached results.

Very proof of concept-y but, again -- the future of advanced, on-device capabilities is bright.

On device models I used: Core ML LLM (Mistral 7B Int4), MobileBERT embeddings, sqlite-vec, Vision OCR, Local RAG libs (Swiftrag / LumoKit), Accelerate cosine sim blog.

From here, agentic implementations, visualizations, alerts based on smart devices (Oura ring), patient empowerment through data access... are all looking possible.