Keynote

Democratizing FPGA-Accelerated Infrastructure for the AI Era: The MangoBoost Perspective

Eriko Nurvitadhi

Co-Founder and the Chief Product Officer of MangoBoost, Inc

Abstract: The explosion of Large Language Models (LLMs) has scaled AI systems to unprecedented levels, requiring seamless orchestration across GPUs, high-speed NICs, and storage. As GPUs offer a growing abundance of raw compute, the primary bottleneck has shifted to data movement. This shift demands sophisticated “super smart” NICs, often called Data Processing Units (DPUs), capable of accelerating data-centric infrastructure functions (e.g., moving/processing data across GPUs, network, storage) while complying with evolving standards such as RoCEv2, NVMe-oF, and Ultra Ethernet.

Historically, FPGAs struggled to gain mainstream AI adoption when positioned as “AI compute accelerator” that directly competes with GPUs. However, FPGAs are uniquely suited as infrastructure accelerators, complementing GPUs by managing the data-centric infrastructure functions that benefit from customizable dataflow-style hardware-accelerations that GPUs are not designed for. While hyperscalers like Microsoft have successfully deployed FPGA-accelerated infrastructure at scale (e.g., Project Catapult), these solutions have remained exclusive within the hyperscaler, and not available in mainstream servers that are for sale to the general public.

MangoBoost aims to bridge this gap. Born from a decade of academic research, our mission is to unlock the full potential of FPGAs as DPUs to dramatically boost GPU-based AI systems. We achieve this through a library of composable FPGA soft IPs for infrastructure accelerations, and an agile flow to productize these IPs onto server-qualified FPGA cards, readily integrated into mainstream GPU and storage servers for AI systems. Furthermore, by co-designing the FPGA DPU solutions with AI system software, we enable end-users to leverage FPGA acceleration seamlessly, without requiring any FPGA expertise.

This talk first discusses trends in AI infrastructure and opportunities for FPGA-based DPUs. Second, it offers a survey of existing DPUs and SmartNICs. Next, it presents MangoBoost’s technical approach for FPGA-based DPUs for AI infrastructure. Then, it highlights technical results from official industry-standard AI benchmarks (MLPerf) demonstrating system-level deployments of the FPGA-accelerated AI infrastructure for inference, training, and storage. Finally, the talk concludes by sharing lessons learned from our journey so far in productizing FPGA research, as well as offering insights on promising research areas in the AI infrastructure domain for the FPGA community to consider.

Speaker: Dr. Nurvitadhi is a Co-Founder and the Chief Product Officer of MangoBoost, Inc. that offers FPGA-based customizable data processing unit (DPU) solutions to boost AI server systems performance and efficiency. MangoBoost, Inc. is a rapidly growing $65M-funded start-up, based on a decade of academic research with full-system prototypes published in computer architecture, systems, and FPGA conferences. Previously, Dr. Nurvitadhi was a Principal Engineer at Intel, focused on FPGAs, accelerators, and AI technologies. He has 70+ peer-reviewed publications, 120+ patents granted/pending, with an
 H-index of 42. In 2020, he was recognized as a top 30 inventor by Intel Patent Group and received a Mahboob Khan Outstanding Liaison Award from SRC. He has served on program committees of IEEE/ACM conferences, and as the Technical Program Chair for FCCM 2022. He received a PhD in ECE from Carnegie Mellon University, and an MBA from Oregon State University.