ECE PhD Prospectus Defense: Xiting Yao

ECE PhD Prospectus Defense: Xiting Yao

Title: Distributed LLM Systems: From Network Offload to Rack-Scale Scheduling

Presenter: Xiteng Yao

Advisor: Professor Martin Herbordt

Chair: Professor Yigong Hu Committee: Professor Martin Herbordt, Professor Yigong Hu, Professor Osama Alshaykh, Professor Tali Moreshet

Google Scholar Link: https://scholar.google.com/citations?user=X9AzwrMAAAAJ

Abstract: Large language model (LLM) systems distribute computation and state across accelerators, memory, storage, and networks. This distribution creates recurring boundaries as training tensors cross collective paths, inference activations cross worker pools, and state moves among memory tiers. The cost depends on bandwidth and the placement and scheduling decisions at each crossing. This dissertation asks when placing computation or state along a shared path, or scheduling work at a handoff between resources, reduces end-to-end overhead.

For training, the research reconsiders the role of SmartNICs, programmable switches, and network-attached memory in LLM systems. The completed studies examine whether network resources can support limited computation and shared state along paths that already carry LLM data. They cover placement across CPUs, GPUs, and SmartNICs, operations integrated with collectives, and shared state near the network. Hardware measurements, co-simulation, and system models estimate avoided endpoint work and added path overhead. A planned study will build and evaluate an FPGA-based switch-attached prototype with LLM workloads.

For inference, the work focuses on attention--FFN disaggregation (AFD). AFD places attention and feed-forward-network computation on separate worker groups that exchange activations at each layer. Different key--value (KV) cache lengths and expert demands can create uneven work across attention and FFN resources. Preliminary work addresses admission within one AFD system. The planned extension studies how rack-scale scheduling can reduce this imbalance. At rack scale, the scheduler can share attention capacity and FFN GPUs across worker pools.

Together, these studies develop methods for boundary-aware placement and scheduling. Component measurements and end-to-end comparisons identify when these choices improve LLM performance under similar resource budgets and when worker-side execution remains preferable.