クラウドを超えて：プライバシー重視のリサーチアシスタントをGemini NanoとオンデバイスRAGで構築する

The landscape of mobile development is currently undergoing its most significant transformation since the introduction of Jetpack Compose. We are moving away from the "Cloud-First" era of Artificial Intelligence toward a "Device-Centric" paradigm. For years, developers have relied on massive LLMs hosted in the cloud, accepting the trade-offs of high latency, recurring API costs, and—most importantly—the sacrifice of user privacy. But what if you could build a research assistant that lives entirely on the user's hardware? An assistant that can parse sensitive legal documents, medical records, or private research papers without a single byte of data ever leaving the device. This isn't a futuristic concept; it is the reality of modern Android development using Gemini Nano , AICore , and On-Device RAG (Retrieval-Augmented Generation) . In this deep dive, we will explore the architectural philosophy of on-device GenAI, the mechanics of local RAG pipelines, and how to orchestrate these complex systems using Kotlin 2.x and Jetpack Compose. (This article is based on the ebook On-Device GenAI with Android Kotlin ) The Architectural Philosophy of On-Device GenAI The transition to on-device intelligence represents a fundamental shift in how we think about resource management. In the cloud, we have virtually infinite compute power but limited by the speed of the network. On-device, the network is irrelevant, but we are governed by the strict laws of thermodynamics and hardware constraints: RAM, battery life, and thermal throttling. To manage this, Google introduced Gemini Nano , a model specifically distilled for mobile efficiency, and AICore , a system-level abstraction layer that changes how we interact with AI hardware. AICore: The System-Level AI Provider One of the biggest mistakes a developer can make in the new AI era is bundling a 2GB+ LLM binary directly into their APK. Doing so would lead to catastrophic storage bloat and memory fragmentation. Instead, Android provides AICore , a system service that manages the underlying Neural Processing Unit (NPU) and GPU acceleration. Think of AICore as the CameraX of the AI world. Before CameraX, developers had to wrestle with device-specific hardware quirks for every different phone manufacturer. CameraX abstracted that complexity. AICore does the same for AI by providing: Centralized Model Management: Gemini Nano is managed via Google Play Services. It is updated and optimized independently of your app, ensuring the user always has the most efficient version of the model. Resource Arbitration: If three different apps try to run LLM inference simultaneously, the system would crash. AICore acts as a traffic controller, queuing requests and managing memory pressure to prevent the Android OS from killing background processes. Hardware Optimization: AICore knows if the device is running a Tensor G3 or a Snapdragon 8 Gen 3. It optimizes the model weights specifically for the Silicon on that specific device. The Local RAG (Retrieval-Augmented Generation) Framework A research assistant is only as good as the data it can access. While Gemini Nano is incredibly smart, it doesn't know what is inside your user’s private PDF files. Furthermore, LLMs have a "context window"—a limit on how much text they can process at once. You cannot simply feed a 500-page book into a mobile LLM and ask for a summary. The solution is Retrieval-Augmented Generation (RAG) . The RAG Pipeline: Giving the LLM a Library Think of RAG as a Room database migration for an LLM’s memory . Just as Room allows an app to persist data that exceeds the device's RAM, RAG allows the LLM to "query" a massive external dataset and pull only the most relevant snippets into its immediate "thought process." The pipeline follows five critical steps: Ingestion (The Embedding Phase): We take the research documents and break them into small "chunks." Each chunk is passed through an embedding model (a specialized, tiny TFLite model) that converts text into a high-dimensional vector—essentially a list of numbers that represent the meaning of the text. Storage (The Vector Store): These vectors are stored in a local index. Unlike a SQL database that looks for exact word matches, a vector store allows for semantic search . If a user asks about "quantum entanglement," the system can find chunks about "spooky action at a distance" because they are mathematically similar in vector space. Retrieval: When the user asks a question, that question is also turned into a vector. We perform a "Cosine Similarity" search to find the top 3 or 5 most relevant chunks from our local store. Augmentation: We "stuff" the prompt. We take the user's question and wrap it with the retrieved chunks. Generation: Gemini Nano receives the augmented prompt (e.g., "Using these three snippets from the document, answer this question...") and generates a grounded, factual response. Connecting Modern Kotlin to AI Orchestration Building a RAG-based assistant requires handling highly asynchronous data. LLMs generate text one "token" (roughly a word or part of a word) at a time. If we waited for the entire response to finish before showing it to the user, the app would feel sluggish. 1. Asynchronous Token Streaming with Flow In Kotlin, we use Flow<String> to stream tokens from AICore directly to the Compose UI. This allows the user to start reading the answer the moment the first token is generated, significantly reducing "perceived latency." 2. Context Receivers for AI Scope In a complex app, many different components need access to the ModelInstance or the VectorStore . Passing these as parameters to every single function leads to "parameter pollution." Kotlin’s Context Receivers (introduced in recent versions) allow us to define a required context for a function without explicitly passing it. 3. Type-Safe Configuration with Serialization AI prompts are no longer just strings; they are structured templates. We use kotlinx.serialization to manage these schemas, ensuring that our metadata (like document source names and page numbers) remains consistent throughout the pipeline. Technical Implementation: The Foundation Let’s look at how we translate this theory into production-ready Kotlin code. First, we need to set up our dependencies to include the MediaPipe GenAI SDK, which provides the interface for Gemini Nano. Gradle Dependencies dependencies { // MediaPipe LLM Inference API for Gemini Nano implementation ( "com.google.mediapipe:tasks-genai:0.10.14" ) // Jetpack Compose & Lifecycle implementation ( "androidx.lifecycle:lifecycle-viewmodel-compose:2.7.0" ) implementation ( "androidx.lifecycle:lifecycle-runtime-compose:2.7.0" ) // Hilt for Dependency Injection implementation ( "com.google.dagger:hilt-android:2.51" ) kapt ( "com.google.dagger:hilt-compiler:2.51" ) // Kotlin Serialization implementation ( "org.jetbrains.kotlinx:kotlinx-serialization-json:1.6.3" ) } The AI Orchestrator The Orchestrator is the "brain" of our operation. It connects the vector search to the LLM generation. @Singleton class ResearchAssistantOrchestrator @Inject constructor ( private val repository : LocalResearchRepository , private val vectorStore : LocalVectorStore ) { /** * Executes the RAG pipeline: Retrieves context, builds the prompt, and streams the response. */ fun askResearchQuestion ( query : String ): Flow < String > = flow { // Step 1: Semantic Retrieval // We fetch the most relevant 'knowledge chunks' from our local vector store val relevantDocs = vectorStore . searchSimilar ( query , limit = 3 ) // Step 2: Prompt Augmentation // We combine the user query with the retrieved context val augmentedPrompt = buildPrompt ( query , relevantDocs ) // Step 3: Generation via Gemini Nano // We use flow to stream tokens to the UI as they are generated repository . generateStreamingResponse ( augmentedPrompt ) . collect { token -> emit ( token ) } } private fun buildPrompt ( query : String , docs : List < ResearchSnippet >): String { val context = docs . joinToString ( "\n\n" ) { it . content } return """ You are a Private Research Assistant. Answer the query using ONLY the provided context. Context: $context Query: $query Answer: """ . trimIndent () } } The Repository: Managing the LLM Lifecycle The Repository handles the heavy lifting of initializing the model. Loading a 1.5GB+ model into RAM is an expensive operation, so we must treat the inference engine as a singleton and ensure it is offloaded from the Main thread. @Singleton class LocalResearchRepository @Inject constructor ( @ApplicationContext private val context : Context ) { private var llmInference : LlmInference ? = null // Path to the Gemini Nano model file on device private val modelPath = "/data/local/tmp/gemini_nano.bin" private suspend fun ensureModelInitialized () = withContext ( Dispatchers . IO ) { if ( llmInference == null ) { val options = LlmInference . LlmInferenceOptions . builder () . setModelPath ( modelPath ) . setMaxTokens ( 1024 ) . setTemperature ( 0.7f ) . build () llmInference = LlmInference . createFromOptions ( context , options ) } } fun generateStreamingResponse ( prompt : String ): Flow < String > = callbackFlow { ensureModelInitialized () // MediaPipe provides a streaming listener llmInference ?. generateResponseAsync ( prompt ) { result , done -> trySend ( result ) if ( done ) close () } awaitClose { /* Handle cleanup if necessary */ } } } Real-World Performance: The "Pitfalls" of Local AI While the code above looks straightforward, building for mobile AI requires a deep understanding of hardware limitations. If you ignore these, your app will be uninstalled faster than it can generate a token. 1. The ANR (Application Not Responding) Trap LLM inference is a synchronous, CPU/GPU-intensive operation. If you call generateResponse() on the Main thread, your UI will freeze for 5 to 10 seconds. Always wrap your repository calls in withContext(Dispatchers.Default) . Use Dispatchers.Default rather than Dispat