“Heterogeneity-Enriched Communication for Parallel and Distributed AI Systems”
Tuesday, July 1 at 12:45pm
MALA 5050
Add to Calendar
Abstract
Modern parallel and distributed computing systems are growing increasingly complex as applications in high-performance computing (HPC) and artificial intelligence (AI) demand higher computation and communication capabilities. To meet these needs, many HPC and AI systems now integrate multiple heterogeneous devices such as CPU, GPU, and DPU (or SmartNICs) within the same compute node—a configuration trend I refer to as multi-rail heterogeneity. While this trend offers opportunities for improved performance, it also brings significant communication challenges that limit scalability. Existing studies either over-rely on a single device type or lack effective coordination between computation and communication across heterogeneous components, resulting in inefficiencies. To address these limitations, this talk introduces a new data movement paradigm, heterogeneity-enriched communication, which embraces hardware diversity by modeling, composing, and optimizing communication across multi-rail systems. I will present recent research and case studies demonstrating its benefits in real-world applications, along with ongoing projects such as OpenDOTA, an open-source effort advancing Data Offloading and Transfer Architecture for modern DPU platforms.
Biography
Dr. Xiaoyi Lu is an associate professor in the Department of Computer Science and Engineering at the University of California, Merced, where he leads the Parallel and Distributed Systems Laboratory (PADSYS Lab). His research interests include parallel and distributed computing; scalable and efficient systems for HPC, big data, AI, cloud, and edge computing; high-performance communication and I/O technologies (e.g., RDMA, NVMe, GPU, DPU); scalable algorithms and applications; and interdisciplinary research for social good. Dr. Lu has published over 170 papers in prestigious conferences and journals, receiving ten Best (Student) Paper Awards or Nominations, including at SC 2019 and IPDPS 2024. He has delivered more than 100 invited talks, tutorials, and presentations worldwide and is actively involved in the academic community. His research outputs, including OpenDOTA, SR-APPFL, PMIdioBench, HiBD, MVAPICH2-Virt, and DataMPI, have made a broad impact across both industry and academia. Dr. Lu has received notable honors, including the NSF CAREER Award and research awards from Amazon, Google, and Meta/Facebook.