Why preprocessing placement matters
Image preprocessing looks cheap until it blocks the accelerator. A request pipeline that decodes JPEG bytes, resizes to model input, converts color layout, and normalizes values can consume tens of milliseconds before the model sees a tensor. When GPUs wait for those bytes, serving capacity drops even though GPU utilization charts look healthy.
The natural response is to move everything to the GPU, but that is not always faster. Small batches, small images, and simple pipelines may finish sooner on CPU workers, while large batches of high resolution JPEGs may benefit strongly from hardware assisted decode. The right split depends on image sizes, batch sizes, concurrency, and the serving hardware in use.
This article gives a measurement procedure for three stages: decode, resize, and normalize. Time each stage on the CPU path and on the GPU path, then assign each stage to the side that shortens end to end latency. The NVIDIA DALI documentation describes GPU accelerated loading and preprocessing with CPU and GPU execution options, and its image decoder reference documents CPU and mixed backends with format dependent behavior.
Isolate decode, resize, and normalize
Measure decode separately from the rest because format support differs by device. GPU accelerated decoding covers only a subset of formats, JPEG and JPEG 2000, while other formats fall back to CPU code paths. The DALI image decoder reference states this split explicitly, so a pipeline that receives PNG, WebP, and JPEG traffic cannot assume uniform GPU decode coverage.
Measure with production image size distribution, not with a single sample image. Decode time grows with pixel count and with encoder settings, so a test set of thumbnails will understate production decode cost. Sample at least a few hundred recent production images, keep their sizes and formats, and replay them through both paths.
Record per stage latency at fixed batch sizes, plus host to device transfer time for the CPU path and device memory use for the GPU path. A GPU decode that saves 8 milliseconds but adds 6 milliseconds of transfer and doubles peak device memory may still lose on end to end latency under load.
Build comparable CPU and GPU paths
Build two minimal pipelines with identical outputs. The CPU baseline uses worker processes with a standard image library and tensor transforms, following the performance guidance in the torchvision transforms documentation, which recommends tensor based v2 transforms and documents CPU and CUDA tensor support. The GPU candidate moves decode and selected stages to an accelerated loader such as DALI, keeping color conversion and normalization numerically consistent.
Verify numerical equivalence before benchmarking. Compare output tensors within a stated tolerance across several hundred images, including orientation metadata, alpha channels, and grayscale edge cases. A faster pipeline that changes pixel values is a model accuracy change, not a performance win.
Pin the test hardware and software. Run both paths on the same GPU type, same driver, same container image, and same concurrency level. Preprocessing results move with thread counts, worker counts, and power limits, so an unpinned comparison measures the test setup rather than the pipeline.
Worked example with batch and size effects
Use an illustrative measurement at batch size 32 with mixed 1080p JPEG traffic. Suppose the CPU path takes 18 ms for decode, 9 ms for resize, and 4 ms for normalize plus transfer, for an end to end preprocessing time of 31 ms. Suppose the GPU path takes 7 ms for decode, 4 ms for resize, and 3 ms for normalize on device, for a total of 14 ms plus 3 ms of scheduling overhead, or 17 ms. All figures are illustrative example values for showing the arithmetic, not measured benchmark claims.
The end to end improvement is 14 ms on a 31 ms baseline, or about 45 percent. By the decision rule below, GPU offload is justified here because the gain far exceeds the threshold. Now change one condition: at batch size 4 with small thumbnails, suppose the CPU path takes 6 ms total while the GPU path takes 5 ms plus 3 ms overhead, or 8 ms. The GPU path is slower, so those stages stay on CPU.
Report results as a small matrix of batch size by image class rather than as one headline number. The placement answer for thumbnails at low concurrency is often different from the answer for 4K images at high concurrency. That matrix becomes the configuration reference for the serving deployment.
| Stage | Stays on CPU when | Moves to GPU when | Watch out for |
|---|---|---|---|
| Decode | Mixed PNG and WebP traffic, small batches | Large JPEG batches with hardware decode available | Format fallback silently returns to CPU timing |
| Resize | Small thumbnails, low concurrency | Large downscale factors at high throughput | Interpolation differences change pixels |
| Normalize | Already fused with model input conversion | On device math removes a transfer | Dtype and range mismatches shift accuracy |
Decision rule for each stage
Assign stages independently using end to end latency, not kernel microbenchmarks. The brief for this article suggests keeping a stage on CPU only when GPU offload shortens end to end latency by at least 10 percent. Read that 10 percent value as an illustrative team policy, not an established industry recommendation. It exists to prevent migrations that add operational complexity for negligible gain.
Apply the rule per stage in pipeline order. Measure the full pipeline with the candidate stage on CPU, then with it on GPU, holding everything else fixed. If the GPU placement reduces median end to end preprocessing latency by at least the team threshold, and does not regress 99th percentile latency or exceed device memory budget, adopt it. Otherwise keep the stage on CPU.
Prefer a workload derived alternative when traffic is heterogeneous. A practical version is to adopt GPU placement only when it improves both median and 90th percentile latency on the production image sample at the deployed concurrency level. Median only rules can approve placements that help the common case while harming tail latency during bursts.
Diagnose before moving stages
Start from the bottleneck the measurements identify. If decode dominates, test hardware decode availability, thread counts, and cache behavior for repeated images before touching resize. If resize dominates, test interpolation settings and target size alignment with model input before adding GPU stages. If transfer dominates, test pinned memory, batching, and keeping normalize on device.
Check CPU worker saturation alongside GPU timing. Data loader workers that run at full CPU with growing queue depth indicate a CPU side bottleneck that more GPU offload may relieve. Conversely, idle CPU workers with a busy GPU indicate the pipeline is already feeding the model fast enough and further offload adds little.
Limits and revalidation triggers
GPU preprocessing shares the device with inference. Under high model load, decode and resize kernels compete with attention and matmul kernels for the same streaming multiprocessors and memory bandwidth. A placement that wins in an isolated benchmark can lose when the model runs concurrently, so final validation must run preprocessing and inference together.
Image diversity also limits generality. A pipeline tuned on JPEG photographs may behave differently on screenshots, scanned documents, or transparency heavy assets. Revalidate placement after product changes that alter the image mix, and keep a CPU fallback path for formats the accelerated decoder does not handle.
Finally, numerical differences deserve ongoing checks. Library updates can change resize kernels, color conversion, or default interpolation. Pin versions, rerun the equivalence sample after upgrades, and treat unexpected accuracy movement as a preprocessing suspect until the tensor comparison clears it.