Allocation is not containment
A scheduler answers where a workload may run and whether a requested resource is available. It does not, by itself, determine what that workload can observe or disrupt after launch. A namespace quota can stop one tenant from claiming every advertised GPU, while the same tenant might still reach another namespace over the default pod network, mount an unsafe host path, or share a physical device fault domain. Treat allocation fairness and security isolation as separate control objectives.
This distinction is especially important with GPU time slicing. NVIDIA documents time slicing as multiplexing workloads on one underlying GPU and states that replicas do not have memory or fault isolation. A request for multiple time-sliced GPU replicas also does not promise a proportional compute share. Time slicing can improve utilization for trusted, cooperative workloads, but it must not appear in a threat model as a boundary between mutually distrustful tenants.
Record five boundaries for every tenant class
Start with a boundary register rather than a scheduler setting. For each tenant class, record quota, process, device memory, network, and data boundaries. Quota covers aggregate consumption and admission. Process isolation covers host namespaces, Linux capabilities, privilege escalation, and access to device files. Device memory and fault isolation cover whether another workload shares the same physical GPU context or failure domain. Network isolation covers east-west reachability and bandwidth contention. Data isolation covers volumes, object stores, caches, logs, and secrets.
The useful question is not whether the cluster is multi-tenant. It is which controls are enforced for a particular pair of tenants. Two teams under one administrator may accept shared nodes and time slicing. External customers processing confidential inputs may require separate nodes, dedicated devices or supported hardware partitioning, default-deny networking, distinct credentials, and storage encryption domains. Write that decision per service tier so an operator cannot silently weaken it while chasing utilization.
| Layer | Control to verify | What it does not prove |
|---|---|---|
| Quota | Namespace GPU, CPU, RAM and object limits | Runtime containment |
| Process | Restricted pod policy, no host namespaces or host paths | GPU memory separation |
| Device | Dedicated GPU or supported MIG profile | Network or storage isolation |
| Network | Default deny plus explicit ingress and egress | Bandwidth fairness unless separately enforced |
| Data | Tenant-scoped identity, volumes, caches and logs | Process isolation |
Choose sharing mode from the trust boundary
Use whole-device assignment when a tenant needs exclusive device access but can share the host under a reviewed container boundary. Use MIG only on supported GPUs and profiles when its hardware-partitioned memory and fault isolation match the workload, while remembering that the host operating system, control plane, storage, and network remain shared. Use time slicing for utilization among workloads that are inside the same trust domain and can tolerate interference and shared failures.
A measurable admission rule makes this enforceable: if two workloads are controlled by different security principals and either handles non-public inputs, disallow time-sliced co-residency. If the agreed blast radius excludes cross-tenant GPU faults, require a dedicated GPU, a validated MIG arrangement, or separate nodes according to the platform threat model. ClusterBid can help match those measured isolation and capacity requirements to available infrastructure, but the service owner still must verify the provider's actual tenancy controls.
Diagnose interference without calling it a breach
When a tenant reports latency variance, first establish placement and physical sharing: pod, node, GPU UUID or MIG device, and concurrent processes. Next compare queue wait, GPU active time, memory use, copy-engine activity, host CPU throttling, storage latency, and network counters over the same interval. Then reproduce with the neighbor absent. A performance change is evidence of contention, not evidence that memory or data crossed a boundary. Keep the incident classifications separate.
Conversely, clean latency does not prove isolation. Validate negative permissions directly: rejected API actions, blocked network probes, denied volume mounts, inaccessible secrets, and absent host namespace access. NVIDIA notes that time-sliced metrics may not be attributable to containers through DCGM Exporter, an observability limitation that should affect the sharing decision. If accounting cannot identify the consumer, set conservative concurrency or move chargeable workloads to attributable device assignments.
Make isolation a release gate
Turn the boundary register into tests that run after changes to the device plugin, container runtime, network plugin, node image, or admission policy. The minimum gate should include quota exhaustion, an attempted privileged workload, cross-namespace ingress and egress probes, unauthorized storage reads, device enumeration from inside the container, and a controlled GPU process failure. Record expected denials and the exact policy version that produced them.
No single mechanism completes this model. Containers share a host kernel, MIG supports only defined hardware and profiles, dedicated GPUs may still share host and fabric resources, and separate clusters can still share identity or storage systems. Recheck controls when a tenant changes trust class or data sensitivity. The honest objective is bounded, documented risk. Scheduling is successful when it places work fairly. Tenant isolation is successful only when every required boundary has an owner, an enforcement point, and a repeatable negative test.