- Evaluate each task and its data flows.
- Include hardware, energy, operations and replacement in local TCO.
- Keep a hybrid option when tasks have different constraints.
Decide task by task
Field extraction, document search, translation, vision, image generation and general assistants have different needs. Build a representative test set, define quality and latency thresholds, then compare actual models.
Model size alone is not enough. Context, GPU memory, concurrency, quantisation and serving software affect performance. Hardware should follow this evaluation.
| Criterion | Questions |
|---|---|
| Data | What enters, where does it flow and what must be retained? |
| Quality | What error rate is acceptable and who reviews? |
| Load | How many users, requests and peaks? |
| Continuity | What latency and downtime are tolerable? |
| Change | How often must the model be replaced? |
Local does not automatically mean private or compliant
An on-premise server can reduce some transfers, but data can still leave through logs, telemetry, backups, updates, connected tools or support access. Cloud services may offer useful contractual and technical controls too. Map the real flow.
Ollama offers a local-only mode and listens only on the local interface by default, but you must manage those settings. GDPR still requires suitable technical and organisational measures. Owning hardware guarantees neither privacy nor compliance.
- Network, authentication and role-based access
- Logs, telemetry and retention periods
- Backups, encryption and tested recovery
- Maintenance access and external dependencies
Local cost goes beyond the GPU price
Local TCO includes the server, the rest of the system, energy, cooling, warranty, installation, operations, patching, monitoring, backups, outages and replacement. Cloud TCO includes usage, storage, network, managed services and pricing risk.
The NVIDIA RTX PRO 6000 Blackwell Workstation Edition, with 96 GB of GDDR7 memory, is a high-end option for some models and professional workloads. It is not a minimum or a default. Tests and budget drive sizing.
Hybrid is often a reasonable architecture
Sensitive documents and stable tasks can stay on controlled infrastructure, while the cloud handles peaks or missing capabilities. Routing must be explicit: data category, allowed model, logging, fallback and human review.
Servers such as vLLM expose APIs compatible with common clients, making backend changes easier. Compatible does not mean interchangeable: test parameters, outputs, tools and limits for every model.
Check the model licence too
Open-weight does not automatically mean open source or unrestricted commercial use. Check the model card, licence, use restrictions and redistribution duties for the exact version you deploy.