Local AI in 2026: what to run without the cloud

What became more practical this autumn

Google's EmbeddingGemma 2 has 740 million parameters and is designed for local multimodal retrieval. It maps text, code, images, video and audio into a shared space, while applications can use only the encoders they need. Google says vector truncation can reduce storage and memory requirements for a local vector database by up to six times.

Microsoft Foundry Local runs compatible language models on a CPU, GPU or NPU and can operate without a cloud dependency after the model is downloaded. Microsoft also states that its native SDKs remain pre-release, so production implementations should pin versions and plan for API changes.

Local does not automatically mean secure

Keeping prompts and outputs on a device or organisational network removes one class of exposure. File permissions, disk encryption, logs, backups, model updates and physical access still matter. A local model with excessive privileges can disclose or modify data just as easily.

Document what downloads on first use, which telemetry requests may occur, where the model cache lives and who can read the local database. Evaluate the exact configuration rather than relying on the word ‘local’ in a product name.

Four workloads worth testing first

Good first use cases have a bounded data set, clear validation and a tangible benefit from low-latency or offline execution.

• Search across internal documents, images, audio and video without transferring the source files externally
• Classify requests, documents or photographs at a workplace or in the field
• Draft, summarise and extract fields from local documents under employee review
• Provide a fallback mode for essential functions when connectivity is unstable or unavailable

When to retain a cloud component

A local environment has limits in memory, throughput, energy use, concurrent users and operational maintenance. Cloud execution may be more efficient for demanding reasoning, very large contexts, workload peaks or immediate access to new models.

A hybrid architecture is often practical. Retrieval, filtering, de-identification and simple repetitive work happen locally; only an approved context reaches the cloud for a harder task. The router needs explicit policies rather than model discretion about which data may leave the environment.

How to test before buying hardware

Use 50–100 representative examples and measure quality, response time, memory consumption, energy use and the share of cases that require cloud or human continuation. Test on the target hardware because the same model can perform very differently across processors and accelerators.

Calculate total cost: devices, storage, configuration, updates, monitoring and team effort. No per-token charge does not mean zero cost. The pilot should produce a table of quality, performance, risk and the conditions for moving between local and cloud execution.

Practical takeaway

Do not move every workload into a local environment at once. Start with one process where data placement, latency or resilience has measurable value. Define permitted sources, test the model on your own examples and design a controlled cloud path only for tasks the local system cannot complete at the required quality.