Private LLM for Enterprise: How to Host One
Private LLM for Enterprise: How to Host One
TL;DR
- Hosting a private enterprise LLM eliminates data egress, secures corporate IP, and satisfies strict data residency requirements under HIPAA and GDPR.
- Serving an unquantized 70B parameter model requires 140 GB of VRAM for weights alone, scaling to 160 GB to 320 GB once KV cache memory is allocated for production context windows.
- Serving engines like vLLM and TensorRT-LLM deliver up to 2,780 tokens per second on NVIDIA H100 hardware through PagedAttention and fused CUDA kernels.
- Self-hosting breaks even against proprietary commercial APIs once enterprise query volume crosses 20 million to 50 million tokens per day under steady load.
- Client-side local execution provides a zero-infrastructure alternative for sensitive desktop workflows without incurring cloud cluster management overhead.
Data privacy regulations and proprietary IP exposure make multi-tenant cloud AI APIs a non-starter for regulated enterprise workloads. At Drevon, we designed our architecture around isolated client runtimes, and we built a free Mac app for local prospect research because enterprise teams cannot afford to leak context to third-party endpoints. When your organization decides to run open-weights foundation models internally, you need a deterministic engineering roadmap covering hardware provisioning, inference serving runtimes, network boundaries, and access control.
Architectural Models for Private Enterprise LLMs
Private enterprise LLMs operate across three primary deployment topologies: isolated Virtual Private Clouds (VPCs), dedicated on-premises bare-metal GPU clusters, and local client execution runtimes. Choosing between these topologies depends on data residency boundaries, existing hardware investments, and whether workloads run centralized server tasks or distributed user-facing desktop jobs.
An isolated VPC deployment places model serving instances within private subnets that have no internet gateways or public IP assignments. All ingestion traffic from internal microservices, CRMs, and ETL pipelines routes across private endpoints using AWS PrivateLink or GCP Private Service Connect. Network egress is blocked at the routing table level. This architecture prevents weights, intermediate prompts, and completion payloads from traversing public transit networks, making it ideal for teams evaluating where your prospect data goes during automated processing.
On-premises bare-metal hosting eliminates third-party cloud infrastructure entirely. Enterprises install dedicated server racks populated with NVIDIA HGX systems inside their own colocation data centers. While this structure demands significant upfront capital expenditure, it provides absolute physical control over model memory buffers and eliminates cloud multi-tenancy risks. When architecting high-throughput workloads, evaluating bare-metal options connects directly to analyzing the integration tax of third-party SaaS vendors.
Local client execution distributes inference directly to end-user workstations equipped with unified memory silicon. Rather than centralizing every inference request through an overloaded enterprise server cluster, individual worker tasks run locally on Apple Silicon (M-series) or dedicated workstation GPUs. We explored this pattern extensively in our review of why Drevon runs on your desktop not in the cloud, demonstrating that running tasks directly in the client sandbox eliminates transmission latency and server infrastructure costs for desktop-driven workflows.
While proprietary foundation model vendors market Zero Data Retention (ZDR) contractual agreements, contractual guarantees remain legally distinct from architectural isolation. In multi-tenant environments, prompts still travel over external networks and reside temporarily in vendor memory. Private hosting guarantees zero egress at the physical network layer, satisfying regulatory frameworks that mandate absolute physical data segregation.
Hardware Sizing and GPU Provisioning
Sizing GPU infrastructure for enterprise LLMs requires calculating the raw precision footprint of the model weights alongside dynamic key-value (KV) cache memory overhead. Miscalculating KV cache growth under multi-tenant concurrency leads directly to Out-Of-Memory (OOM) runtime crashes and degraded inference throughput.
To calculate baseline VRAM requirements for model weights, multiply the total parameter count by the bytes per parameter based on the chosen numeric precision. In standard 16-bit half-precision (FP16 or BF16), each parameter occupies 2 bytes. An 8-billion parameter model requires approximately 16 GB of VRAM solely for its weights. A dense 70-billion parameter model requires 140 GB of VRAM in FP16. Quantizing model weights using 8-bit formats (INT8/FP8) halves this baseline footprint to 1 byte per parameter (70 GB for 70B models), while 4-bit quantization methods (INT4/AWQ/GPTQ) reduce it to 0.5 bytes per parameter (35 GB for 70B models).
In addition to model weights, production serving engines must allocate memory for the KV cache to store attention states during long context generation. The formula for calculating total KV cache memory across active requests is:
$$\text{Memory}_{\text{KV}} = 2 \times \text{Layers} \times \text{Attention Heads} \times \text{Head Dim} \times \text{Precision Bytes} \times \text{Context Tokens} \times \text{Batch Size}$$
For a 70B model running a 32,768-token context window with a concurrency batch size of 16 users, the KV cache alone demands more than 60 GB of dedicated VRAM. When context windows extend to 128,000 tokens, serving clusters require multi-GPU tensor parallelism configurations across 4× or 8× 80 GB GPUs simply to prevent cache exhaustion under production load.
| Model Architecture | Precision Format | Weight VRAM | Recommended Production VRAM (Weights + KV) | Optimal Server Hardware Profile |
|---|---|---|---|---|
| 8B Dense (e.g., Llama 3.1 8B) | FP16 / BF16 | 16 GB | 24 GB – 32 GB | 1× NVIDIA L4 (24 GB) or 1× A10G (24 GB) |
| 8B Dense (e.g., Llama 3.1 8B) | INT4 / AWQ | 5 GB | 8 GB – 16 GB | 1× NVIDIA RTX 4090 or Apple Silicon (16 GB+) |
| 70B Dense (e.g., Llama 3.3 70B) | FP8 Precision | 70 GB | 160 GB (Tensor Parallelism = 2) | 2× NVIDIA H100 SXM5 (80 GB) |
| 70B Dense (e.g., Llama 3.3 70B) | FP16 / BF16 | 140 GB | 320 GB (Tensor Parallelism = 4) | 4× to 8× NVIDIA A100 (80 GB) / H100 (80 GB) |
| 8x22B MoE (e.g., Mixtral 8x22B) | FP8 / INT8 | 142 GB | 320 GB – 640 GB | 4× to 8× NVIDIA H100 SXM5 (80 GB) |
Cloud GPU instance pricing varies substantially across providers and commitment tiers. In standard regions such as AWS us-east-1, an 8× A100 node (p4d.24xlarge with 320 GB total VRAM) bills at $21.96 per hour on-demand, dropping to approximately $13.92 per hour on a 1-year reserved commitment. For frontier-scale throughput, an 8× H100 node (p5.48xlarge with 640 GB VRAM) costs $55.04 per hour on-demand and approximately $37.40 per hour on a 1-year commitment. On Google Cloud Platform (GCP us-central1), a single L4 instance (g2-standard-4) costs $0.7068 per hour on-demand ($0.4453 on a 1-year Committed Use Discount), serving as a cost-efficient baseline for 8B parameter deployments.
Inference Engines and Serving Runtimes
Selecting the right inference serving engine determines sustained token throughput, memory utilization efficiency, and p95 latency under high concurrency. Standard PyTorch execution engines cannot handle multi-user enterprise traffic due to linear memory fragmentation and non-batched sequential scheduling.
Modern production deployments rely on three primary serving runtimes: vLLM, NVIDIA TensorRT-LLM, and Hugging Face Text Generation Inference (TGI). Detailed benchmarks from Spheron Network's serving engine comparisons show how throughput scales across concurrency loads on NVIDIA H100 SXM5 hardware serving dense 70B parameter models in FP8 precision.
NVIDIA TensorRT-LLM delivers the highest raw token throughput under high-concurrency saturation. By utilizing ahead-of-time (AOT) engine compilation, specialized FP8 memory layouts, and fused CUDA kernels (combining Linear, LayerNorm, and Attention operations), TensorRT-LLM achieves up to 2,780 tokens per second when handling 100 concurrent requests. However, it imposes a high operational burden, requiring lengthy compilation cycles whenever model weights or tensor parallel dimensions change.
vLLM represents the open-source production standard for enterprise infrastructure. By introducing PagedAttention, vLLM manages KV cache allocations in non-contiguous virtual memory blocks, driving GPU memory utilization to 85–95% without pre-compilation overhead. Under identical 100-user saturation tests on H100 hardware, vLLM sustains 2,400 tokens per second with cold-start deployment times under 60 seconds. Additional open research on serving optimizations is documented in technical papers on inference serving architectures.
Hugging Face TGI provides a production-grade Rust router with native integration for gated model registries, but peaks between 1,400 and 1,600 tokens per second under heavy saturation. Comparative evaluations on serving systems demonstrate that TGI trails vLLM by 1.3× to 2× once batch concurrency crosses 32 simultaneous requests, as analyzed in recent research on scalable inference infrastructure.
| Serving Engine | Throughput @ 1 Concurrency | Throughput @ 10 Concurrency | Throughput @ 50 Concurrency | Throughput @ 100 Concurrency | Operational Overhead |
|---|---|---|---|---|---|
| NVIDIA TensorRT-LLM | ~130 tok/s | ~710 tok/s | ~2,100 tok/s | ~2,780 tok/s | High (AOT compilation & Triton server setup) |
| vLLM (PagedAttention) | ~120 tok/s | ~650 tok/s | ~1,850 tok/s | ~2,400 tok/s | Low (Dynamic loading, OpenAI-compatible API) |
| Hugging Face TGI | ~105 tok/s | ~500 tok/s | ~1,200 tok/s | ~1,500 tok/s | Moderate (Rust container, HF ecosystem locks) |
To maximize hardware utilization, configure continuous batching (dynamic iteration-level scheduling) rather than static request batching. Continuous batching evicts completed sequences immediately at each token generation step, allowing new incoming requests to join the active forward pass without waiting for an entire batch to finish. For enterprise developers connecting internal applications, both vLLM and TGI expose drop-in OpenAI-compatible REST and gRPC endpoints (/v1/chat/completions), enabling straightforward integration with internal agent orchestrations such as those found in 7 GTM workflows now run by AI agents.
Security, Identity, and Governance Controls
Deploying private enterprise inference shifts full regulatory and data governance obligations directly onto internal engineering teams. Compliance frameworks such as SOC 2 Type II and the HIPAA Security Rule require rigid technical controls across authentication, memory sanitation, and audit logging.
At the identity and perimeter layer, unauthenticated LLM endpoints must be strictly blocked. Enforce Mutual TLS (mTLS) or OpenID Connect (OIDC) JSON Web Tokens at the API gateway layer. Under 45 CFR § 164.312 of the HIPAA Security Rule and the HHS HIPAA Security Standards, enterprise access systems must guarantee unique user identification, role-based access control (RBAC), and transmission encryption via FIPS-compliant TLS 1.3. Additional deployment blueprints are detailed in the HAIIS security documentation.
When connecting private models to enterprise knowledge retrieval systems, enforce tenant-level access isolation. Vector databases and search indexes must filter context chunks against user identity metadata before populating the prompt payload. This design pattern mitigates sensitive information disclosure risks and aligns with privacy engineering practices outlined in our guide on GDPR-compliant lead research.
Memory sanitation represents a unique vulnerability in self-hosted model serving. Multi-tenant serving runtimes must actively isolate and zero out GPU VRAM buffers and volatile KV cache segments between user sessions. Failing to clear memory pages creates cross-session data remanence risks, where fragments of prior confidential prompts can inadvertently bleed into subsequent inference contexts. Engineering teams managing private systems must pair memory isolation with cryptographic model weight verification (verifying SHA-256 signatures before loading artifacts into memory) to prevent supply-chain tampering.
For audit logging, establish immutable telemetry pipelines that capture caller identity, timestamps, model version IDs, and token usage. Raw prompt texts and generation completions must pass through inline PII redaction filters before being stored in persistent log sinks like Datadog, Splunk, or Amazon CloudWatch to prevent secondary compliance exposure.
Total Cost of Ownership: Self-Hosted vs. Cloud APIs
Determining whether to self-host an enterprise LLM requires modeling infrastructure spend, utilization rates, and engineering maintenance against commercial API token costs. Self-hosting fixed costs do not change with token volume, whereas commercial APIs scale linearly with usage.
Peer-reviewed economic research from Carnegie Mellon University and IEEE BigData (Pan et al., 2025, also verified in the complete CMU cost-benefit analysis) demonstrates that self-hosting becomes financially viable once an enterprise reaches steady query volume thresholds. For 24B–32B parameter models, payback periods against proprietary APIs occur within 0.3 to 3 months. For dense 70B parameter models, payback spans 3.8 to 34 months depending on whether the alternative baseline is a mid-tier model or a frontier-tier API.
| Model Scale | Dedicated Infrastructure Setup | Effective Self-Hosted Cost / 1M Tokens (At >60% Saturation) | Commercial API Baseline / 1M Tokens | Break-Even Volume (Raw Hardware) | Break-Even Volume (Fully Loaded Enterprise TCO) |
|---|---|---|---|---|---|
| 7B – 8B Specialized | 1× NVIDIA L4 / A10G ($0.44 – $0.75/hr) | $0.05 – $0.15 | $0.20 – $0.60 (Mini API tiers) | 1M – 3M tokens/day | 5M – 10M tokens/day |
| 24B – 32B Medium | 1× A100 / H100 ($2.00 – $2.50/hr) | $0.15 – $0.35 | $1.00 – $2.50 (Mid API tiers) | 3M – 5M tokens/day | 15M – 25M tokens/day |
| 70B Dense Frontier-Class | 2× to 4× H100 80GB ($5.00 – $10.00/hr) | $0.47 – $0.90 | $2.50 – $15.00 (Frontier API tiers) | 5M – 10M tokens/day | 30M – 50M+ tokens/day |
Compute hardware accounts for only 20% to 30% of total enterprise deployment costs. Fully loaded Total Cost of Ownership (TCO) includes network egress, load balancing, continuous monitoring, and the engineering personnel required to manage CUDA updates, Triton orchestration, and on-call rotations. A dedicated infrastructure engineer adds $150,000 to $250,000 annually to operational overhead.
GPU utilization patterns heavily impact effective token costs. When an enterprise cluster runs at full batch saturation (over 70% continuous load), the effective compute cost drops to $0.47–$0.60 per million tokens. Under bursty enterprise traffic with idle overnight periods and 5–10% average utilization, the effective cost surges above $15.00 per million tokens. Commercial APIs remain the economically superior choice for bursty, experimental, or low-volume workloads under 10 million tokens per day. Self-managed private clusters become the preferred path once volume sustains above 30 to 50 million tokens per day and data sovereignty requirements prohibit external transit.
For specialized revenue operations and research workflows, teams can bypass centralized server maintenance by using client-side tools. Rather than paying cloud GPU markups or navigating credit-based pricing models that penalize discovery, engineers can run local agents directly within their workstations. For technical teams establishing automated pipelines, this model aligns with the core principles of what is a GTM engineer and demonstrates how to build evidence-based prospecting workflows without adding complex server infrastructure.
Frequently Asked Questions About Enterprise Private LLMs
How much VRAM is required to host an open-weights 70B parameter model in an enterprise VPC?
Hosting a dense 70B parameter model in FP16 precision requires 140 GB of VRAM for model weights alone. When factoring in KV cache overhead for multi-user concurrency and a 32k context window, production deployments require at least 160 GB to 320 GB of total VRAM. This is typically provisioned using two to four 80 GB GPUs (such as NVIDIA A100 or H100) configured with tensor parallelism.
Can a private enterprise LLM run completely air-gapped without internet access?
Yes. Once model weights, container base images, and dependencies are downloaded and cryptographically verified, an inference engine like vLLM or TensorRT-LLM operates entirely within an isolated private network. Disabling external routing tables and internet gateways guarantees zero external telemetry, preventing prompt and completion leakage outside enterprise boundaries.
What is the throughput difference between vLLM and TensorRT-LLM on enterprise hardware?
On an NVIDIA H100 SXM5 serving a 70B FP8 model under peak concurrency (100 concurrent requests), TensorRT-LLM achieves approximately 2,780 tokens per second compared to vLLM's 2,400 tokens per second, a roughly 16% throughput advantage. However, vLLM eliminates the complex ahead-of-time compilation pipeline required by TensorRT-LLM, making it the preferred choice for standard enterprise setups.
How does local-first client execution compare to centralized private server hosting for data security?
Local-first client execution runs inference directly within the user's workstation hardware or authenticated local browser session, completely eliminating central server infrastructure and data transmission hops. Centralized private hosting runs models inside a corporate VPC, which centralizes compute management but requires managing network isolation, API gateways, and multi-tenant memory buffers.
If you want to run research workflows locally without managing centralized GPU infrastructure or sending data to cloud vendors, download Drevon for Mac to run evidence-backed workflows on your existing AI subscriptions today.