Theses

Peak-Memory-aware Partitioning and Scheduling for Multi-tenant DNN Model Inference [abstract] (PDF)
Jaeho Lee
Ph.D. Thesis, School of Electronical and Electronic Engineering, Yonsei University, August 2026.

Deep neural network (DNN) inference on GPU-based systems spends a large fraction of its time on memory management. Dynamic memory allocation through CUDA APIs consumes 33% to 50% of the total inference time, while the computation itself accounts for only 20% to 25%. The problem worsens in multi-tenant deployments, where the temporary workspace that backend libraries such as cuDNN and cuBLAS allocate internally stays invisible to graph-level analysis and triggers unexpected out-of-memory (OOM) errors.

This study presents a DNN inference compiler and scheduler that plans GPU memory before execution. Because an inference model has a fixed computation graph and deterministic tensor shapes, its memory demand can be analyzed before the model runs, which allows per-tensor dynamic allocation to be replaced with a small number of pool-level allocations. A tensor liveness analysis first unifies visible tensors and the hidden backend workspace into a single memory demand profile, and operation-level optimization such as layer fusion and tensor coalescing removes redundant memory operations. A constraint programming formulation then solves the underlying Dynamic Storage Allocation problem with the CP-SAT solver, producing a provably optimal layout for each memory pool. For concurrent execution, the Tailor-and-Stitch algorithm partitions a model into memory-balanced schedulable units called Tasks according to its compile-time peak memory profile. A dual memory pool design keeps tensors shared across Task boundaries resident in GPU memory and avoids the backup and restore transfers they would otherwise require. A compiler-generated scheduler interface drives a yield-based scheduler that admits each new Task only after a worst-case memory check, so that concurrent tenants run free of OOM failures.

On NVIDIA RTX 3090 and Jetson AGX Xavier, single-tenant inference uses 34.6% less memory than PyTorch on average and runs 1.25 times faster in geometric mean. On NVIDIA RTX 4090 and Jetson Orin Nano, the system runs multiple models concurrently without any OOM failure at an average performance overhead of 3.20% on the desktop GPU, and it serves requests with up to 10.8% lower latency than the NVIDIA Triton Inference Server and 43.81% lower latency than SwapNet on embedded platforms.