Draw the stack boundary first
A GPU container does not supply an independent NVIDIA kernel driver. The host owns the kernel driver, and the NVIDIA container stack injects selected devices and driver-side libraries or mounts into the container. The image normally carries the CUDA runtime libraries and application dependencies it was built to use. Above those sit the framework, serving engine, extensions, and model code. Compatibility must hold across every boundary.
This explains why installing a full CUDA toolkit on the host rarely repairs a container that already includes its runtime. NVIDIA's container documentation states that the host needs the driver and that the toolkit need not be installed there. It also explains why a working nvidia-smi is not an application acceptance test: device enumeration can succeed while the framework fails to load a library, rejects the GPU architecture, or invokes a feature that needs a newer driver.
Inventory five identities before changing anything
Capture the physical GPU model and compute capability, host driver version, container image digest, CUDA runtime reported inside the container, and framework build string. Then inventory cuDNN, NCCL, TensorRT or other linked libraries when the application uses them. Record the exact failing command and error. Do not infer the runtime version solely from the CUDA version displayed by nvidia-smi, which reflects driver capability rather than proving which runtime the process loaded.
For a framework binary, consult its official installation matrix. PyTorch publishes builds only for selected CUDA runtime versions, so a package label is part of the compatibility record. For the driver-runtime boundary, use the CUDA release notes or compatibility guide for the specific release. Do not guess from version ordering alone. CUDA supports defined backward, minor-version, and forward-compatibility paths, each with constraints.
Read compatibility in the right direction
A newer driver generally supports applications built with older CUDA toolkits through backward compatibility. From CUDA 11 onward, minor-version compatibility can allow an application built within the same major toolkit family to run on a sufficiently new minimum driver, but NVIDIA lists limitations. Applications using PTX can encounter runtime issues on older drivers, and features spanning toolkit and driver may return an error indicating that a newer driver is required.
Forward compatibility across major families is a different mechanism and can require a specific CUDA compatibility package on supported configurations. It is not a blanket instruction to copy driver libraries into an image. Check the official table for the deployed toolkit, driver branch, operating system, and GPU class. If a framework vendor's support matrix is narrower than CUDA's general guarantee, follow the narrower, tested combination for that framework.
Worked example: enumeration works, framework does not
Assume an illustrative host runs driver branch X, while a container uses a framework wheel built for CUDA runtime Y. nvidia-smi works inside the container, but framework initialization reports that the driver is insufficient. First verify the image digest and framework build. Next read the official minimum-driver table for runtime Y. If branch X is below that minimum, the valid choices are to qualify a newer host driver or select an officially supplied framework build for an older compatible runtime.
If the driver meets the table, run a minimal framework device allocation, then inspect the failing extension. A custom extension may contain native cubins only for other GPU architectures, or it may depend on PTX JIT behavior that the installed driver does not support under the selected compatibility path, even though the base framework is valid. Rebuild it with supported architecture targets or use a matching published binary. This sequence avoids replacing the driver, runtime, and framework simultaneously, which destroys the evidence needed to identify the incompatible edge.
Use a layered diagnostic sequence
Start on the host: confirm the driver loads, the expected GPUs enumerate, and the node is healthy. Enter the exact image and confirm only intended devices are visible. NVIDIA documents NVIDIA_VISIBLE_DEVICES for device enumeration and NVIDIA_DRIVER_CAPABILITIES for selecting driver features such as compute or utility that are exposed in the container. Omitting compute from NVIDIA_DRIVER_CAPABILITIES can prevent CUDA use even though utility tools are present; this setting is unrelated to the GPU's numeric compute capability. Then inspect the loaded CUDA runtime and run the smallest device query or allocation available.
Move upward one layer at a time: framework import, framework device allocation, one basic kernel, required library operation, custom extension, then the real model. Stop at the first failure and compare that component's official compatibility table. A release gate should require all steps on every node image and GPU class. If identical images behave differently, compare host driver, device model, runtime-injected mounts, and node configuration before blaming application code.
Pin a tested matrix, not a version myth
Maintain a matrix keyed by node image, driver branch, GPU model, container digest, framework build, and application test receipt. Admit a workload only to a row that passed. A practical decision rule is zero unqualified combinations in production: a new driver, GPU, base image, or framework creates a new row and requires smoke, correctness, and representative workload tests before general scheduling.
Official tables define supported possibilities, not performance or freedom from every defect. Minor compatibility may exclude a feature the application uses; a framework can bundle libraries with additional requirements; custom kernels can target a limited architecture set. Container immutability also cannot freeze the host driver that is injected at runtime. Retain one canary node for each qualified row, and roll driver changes through it before expanding the node pool. Capture kernel logs and framework diagnostics with the receipt so later failures can be compared to a known-good launch. When sourcing capacity through ClusterBid, specify the qualified driver branch, GPU model, and acceptance command rather than requesting only a CUDA version. Treat compatibility as a verified relationship between pinned layers. That approach is slower than reading one version string, but much faster than debugging a fleet built on an assumed match.