AI Infrastructure

POD-Attention: Unlocking Full Prefill-Decode Overlap for Faster LLM Inference

Aditya K Kamath, Ramya Prabhu, Jayashree Mohan, Simon Peter, Ramachandran Ramjee, Ashish Panwar

Architectural Support for Programming Languages and Operating Systems (ASPLOS) 2025 | March 2025

Publication

vAttention: Dynamic Memory Management for Serving LLMs without PagedAttention

Ramya Prabhu, Ajay Nayak, Jayashree Mohan, Ramachandran Ramjee, Ashish Panwar

Architectural Support for Programming Languages and Operating Systems (ASPLOS) 2025 | March 2025

PDF Preprint Github

Towards Efficient Large Multimodal Model Serving

Haoran Qiu, Anish Biswas, Zihan Zhao, Jayashree Mohan, Alind Khare, Esha Choukse, Íñigo Goiri, Zeyu Zhang, Haiying Shen, Chetan Bansal, Ramachandran Ramjee, Rodrigo Fonseca

February 2025

Preprint Project

Etalon: Holistic Performance Evaluation Framework for LLM Inference Systems

Amey Agrawal, Anmol Agarwal, Nitin Kedia, Jayashree Mohan, Souvik Kundu, Nipun Kwatra, Ramachandran Ramjee, Alexey Tumanov

July 2024

Preprint

VIDUR: A Large-Scale Simulation Framework for LLM Inference

Amey Agrawal, Nitin Kedia, Jayashree Mohan, Ashish Panwar, Nipun Kwatra, Bhargav S. Gulavani, Ramachandran Ramjee, Alexey Tumanov

MLSys | May 2024

PDF Github

Just-In-Time Checkpointing: Low Cost Error Recovery from Deep Learning Training Failures

Tanmaey Gupta, Sanjeev Krishnan, Rituraj Kumar, Abhishek Vijeev, Bhargav Gulavani, Nipun Kwatra, Ramachandran Ramjee, Muthian Sivathanu

2024 European Conference on Computer Systems | April 2024

DOI Publication

Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve

Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav Gulavani, Alexey Tumanov, Ramachandran Ramjee

OSDI ’24 | March 2024

DOI Publication Publication Github

SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills

Amey Agrawal, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav S. Gulavani, Ramachandran Ramjee

September 2023

Preprint

Renee: End-to-end training of extreme classification models

Vidit Jain, Jatin Prakash, Deepak Saini, Jian Jiao, Ramachandran Ramjee, Manik Varma

In Proceedings of the Conference on Machine Learning and Systems, Miami, Florida | June 2023

Publication

Varuna: Scalable, Low-cost Training of Massive Deep Learning Models

Sanjith Athlur, Nitika Saran, Muthian Sivathanu, Ramachandran Ramjee, Nipun Kwatra

EuroSys 2022 | April 2022

Best Paper Award

Publication PDF

Singularity: Planet-Scale, Preemptive and Elastic Scheduling of AI Workloads

Dharma Shukla, Muthian Sivathanu, Srinidhi Viswanatha, Bhargav Gulavani, Rimma Nehme, Amey Agrawal, Chen Chen, Nipun Kwatra, Ramachandran Ramjee, Pankaj Sharma, Atul Katiyar, Vipul Modi, Vaibhav Sharma, Abhishek Singh, Shreshth Singhal, Kaustubh Welankar, Lu Xun, Ravi Anupindi, Karthik Elangovan, Hasibur Rehman, Zhou Lin, Rahul Seetharaman, Cheng Xu, Eddie Ailijiang, Suresh Krishnappa, Mark Russinovich

February 2022

Publication

Balancing efficiency and fairness in heterogeneous GPU clusters for deep learning

Shubham Chaudhary, Ramachandran Ramjee, Muthian Sivathanu, Nipun Kwatra, Srinidhi Viswanatha

Fifteenth European Conference on Computer Systems (EuroSys’20) | April 2020

Publication

Quiver: An Informed Storage Cache for Deep Learning

Abhishek Vijaya Kumar, Muthian Sivathanu

18th USENIX Conference on File and Storage Technologies (FAST 2020) | February 2020

Publication Project

Astra: Exploiting Predictability to Optimize Deep Learning

Muthian Sivathanu, Tapan Chugh, Sanjay S. Singapuram, Lidong Zhou

ACM Symposium on Architectural Support for Programming Languages and Operating Systems (ASPLOS 2019) | April 2019

Publication Project Project

Gandiva: Introspective Cluster Scheduling for Deep Learning

Wencong Xiao, Romil Bhardwaj, Ramachandran Ramjee, Muthian Sivathanu, Nipun Kwatra, Zhenhua Han, Pratyush Patel, Xuan Peng, Hanyu Zhao, Quanlu Zhang, Fan Yang, Lidong Zhou

13th USENIX Symposium on Operating Systems Design and Implementation (OSDI 2018) | October 2018

Publication Project Project

POD-Attention: Unlocking Full Prefill-Decode Overlap for Faster LLM Inference

Aditya K Kamath, Ramya Prabhu, Jayashree Mohan, Simon Peter, Ramachandran Ramjee, Ashish Panwar

Architectural Support for Programming Languages and Operating Systems (ASPLOS) 2025 | March 2025

Publication

vAttention: Dynamic Memory Management for Serving LLMs without PagedAttention

Ramya Prabhu, Ajay Nayak, Jayashree Mohan, Ramachandran Ramjee, Ashish Panwar

Architectural Support for Programming Languages and Operating Systems (ASPLOS) 2025 | March 2025

PDF Preprint Github

Towards Efficient Large Multimodal Model Serving

Haoran Qiu, Anish Biswas, Zihan Zhao, Jayashree Mohan, Alind Khare, Esha Choukse, Íñigo Goiri, Zeyu Zhang, Haiying Shen, Chetan Bansal, Ramachandran Ramjee, Rodrigo Fonseca

February 2025

Preprint Project

Etalon: Holistic Performance Evaluation Framework for LLM Inference Systems

Amey Agrawal, Anmol Agarwal, Nitin Kedia, Jayashree Mohan, Souvik Kundu, Nipun Kwatra, Ramachandran Ramjee, Alexey Tumanov

July 2024

Preprint

VIDUR: A Large-Scale Simulation Framework for LLM Inference

Amey Agrawal, Nitin Kedia, Jayashree Mohan, Ashish Panwar, Nipun Kwatra, Bhargav S. Gulavani, Ramachandran Ramjee, Alexey Tumanov

MLSys | May 2024

PDF Github

Just-In-Time Checkpointing: Low Cost Error Recovery from Deep Learning Training Failures

Tanmaey Gupta, Sanjeev Krishnan, Rituraj Kumar, Abhishek Vijeev, Bhargav Gulavani, Nipun Kwatra, Ramachandran Ramjee, Muthian Sivathanu

2024 European Conference on Computer Systems | April 2024

DOI Publication

Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve

Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav Gulavani, Alexey Tumanov, Ramachandran Ramjee

OSDI ’24 | March 2024

DOI Publication Publication Github

SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills

Amey Agrawal, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav S. Gulavani, Ramachandran Ramjee

September 2023

Preprint

Varuna: Scalable, Low-cost Training of Massive Deep Learning Models

Sanjith Athlur, Nitika Saran, Muthian Sivathanu, Ramachandran Ramjee, Nipun Kwatra

EuroSys 2022 | April 2022

Best Paper Award

Publication PDF

Singularity: Planet-Scale, Preemptive and Elastic Scheduling of AI Workloads

Dharma Shukla, Muthian Sivathanu, Srinidhi Viswanatha, Bhargav Gulavani, Rimma Nehme, Amey Agrawal, Chen Chen, Nipun Kwatra, Ramachandran Ramjee, Pankaj Sharma, Atul Katiyar, Vipul Modi, Vaibhav Sharma, Abhishek Singh, Shreshth Singhal, Kaustubh Welankar, Lu Xun, Ravi Anupindi, Karthik Elangovan, Hasibur Rehman, Zhou Lin, Rahul Seetharaman, Cheng Xu, Eddie Ailijiang, Suresh Krishnappa, Mark Russinovich

February 2022

Publication

Balancing efficiency and fairness in heterogeneous GPU clusters for deep learning

Shubham Chaudhary, Ramachandran Ramjee, Muthian Sivathanu, Nipun Kwatra, Srinidhi Viswanatha

Fifteenth European Conference on Computer Systems (EuroSys’20) | April 2020

Publication

Quiver: An Informed Storage Cache for Deep Learning

Abhishek Vijaya Kumar, Muthian Sivathanu

18th USENIX Conference on File and Storage Technologies (FAST 2020) | February 2020

Publication Project

Astra: Exploiting Predictability to Optimize Deep Learning

Muthian Sivathanu, Tapan Chugh, Sanjay S. Singapuram, Lidong Zhou

ACM Symposium on Architectural Support for Programming Languages and Operating Systems (ASPLOS 2019) | April 2019

Publication Project Project

Gandiva: Introspective Cluster Scheduling for Deep Learning

Wencong Xiao, Romil Bhardwaj, Ramachandran Ramjee, Muthian Sivathanu, Nipun Kwatra, Zhenhua Han, Pratyush Patel, Xuan Peng, Hanyu Zhao, Quanlu Zhang, Fan Yang, Lidong Zhou

13th USENIX Symposium on Operating Systems Design and Implementation (OSDI 2018) | October 2018

Publication Project Project

Towards Efficient Large Multimodal Model Serving

Haoran Qiu, Anish Biswas, Zihan Zhao, Jayashree Mohan, Alind Khare, Esha Choukse, Íñigo Goiri, Zeyu Zhang, Haiying Shen, Chetan Bansal, Ramachandran Ramjee, Rodrigo Fonseca

February 2025

Preprint Project

Renee: End-to-end training of extreme classification models

Vidit Jain, Jatin Prakash, Deepak Saini, Jian Jiao, Ramachandran Ramjee, Manik Varma

In Proceedings of the Conference on Machine Learning and Systems, Miami, Florida | June 2023

Publication

Quiver: An Informed Storage Cache for Deep Learning

Abhishek Vijaya Kumar, Muthian Sivathanu

18th USENIX Conference on File and Storage Technologies (FAST 2020) | February 2020

Publication Project

Astra: Exploiting Predictability to Optimize Deep Learning

Muthian Sivathanu, Tapan Chugh, Sanjay S. Singapuram, Lidong Zhou

ACM Symposium on Architectural Support for Programming Languages and Operating Systems (ASPLOS 2019) | April 2019

Publication Project Project

Renee: End-to-end training of extreme classification models

Vidit Jain, Jatin Prakash, Deepak Saini, Jian Jiao, Ramachandran Ramjee, Manik Varma

In Proceedings of the Conference on Machine Learning and Systems, Miami, Florida | June 2023

Publication

POD-Attention: Unlocking Full Prefill-Decode Overlap for Faster LLM Inference

Aditya K Kamath, Ramya Prabhu, Jayashree Mohan, Simon Peter, Ramachandran Ramjee, Ashish Panwar

Architectural Support for Programming Languages and Operating Systems (ASPLOS) 2025 | March 2025

Publication

vAttention: Dynamic Memory Management for Serving LLMs without PagedAttention

Ramya Prabhu, Ajay Nayak, Jayashree Mohan, Ramachandran Ramjee, Ashish Panwar

Architectural Support for Programming Languages and Operating Systems (ASPLOS) 2025 | March 2025

PDF Preprint Github

Etalon: Holistic Performance Evaluation Framework for LLM Inference Systems

Amey Agrawal, Anmol Agarwal, Nitin Kedia, Jayashree Mohan, Souvik Kundu, Nipun Kwatra, Ramachandran Ramjee, Alexey Tumanov

July 2024

Preprint

VIDUR: A Large-Scale Simulation Framework for LLM Inference

Amey Agrawal, Nitin Kedia, Jayashree Mohan, Ashish Panwar, Nipun Kwatra, Bhargav S. Gulavani, Ramachandran Ramjee, Alexey Tumanov

MLSys | May 2024

PDF Github

Just-In-Time Checkpointing: Low Cost Error Recovery from Deep Learning Training Failures

Tanmaey Gupta, Sanjeev Krishnan, Rituraj Kumar, Abhishek Vijeev, Bhargav Gulavani, Nipun Kwatra, Ramachandran Ramjee, Muthian Sivathanu

2024 European Conference on Computer Systems | April 2024

DOI Publication

Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve

Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav Gulavani, Alexey Tumanov, Ramachandran Ramjee

OSDI ’24 | March 2024

DOI Publication Publication Github

Renee: End-to-end training of extreme classification models

Vidit Jain, Jatin Prakash, Deepak Saini, Jian Jiao, Ramachandran Ramjee, Manik Varma

In Proceedings of the Conference on Machine Learning and Systems, Miami, Florida | June 2023

Publication

Varuna: Scalable, Low-cost Training of Massive Deep Learning Models

Sanjith Athlur, Nitika Saran, Muthian Sivathanu, Ramachandran Ramjee, Nipun Kwatra

EuroSys 2022 | April 2022

Best Paper Award

Publication PDF

Balancing efficiency and fairness in heterogeneous GPU clusters for deep learning

Shubham Chaudhary, Ramachandran Ramjee, Muthian Sivathanu, Nipun Kwatra, Srinidhi Viswanatha

Fifteenth European Conference on Computer Systems (EuroSys’20) | April 2020

Publication

Quiver: An Informed Storage Cache for Deep Learning

Abhishek Vijaya Kumar, Muthian Sivathanu

18th USENIX Conference on File and Storage Technologies (FAST 2020) | February 2020

Publication Project

Astra: Exploiting Predictability to Optimize Deep Learning

Muthian Sivathanu, Tapan Chugh, Sanjay S. Singapuram, Lidong Zhou

ACM Symposium on Architectural Support for Programming Languages and Operating Systems (ASPLOS 2019) | April 2019

Publication Project Project

Gandiva: Introspective Cluster Scheduling for Deep Learning

Wencong Xiao, Romil Bhardwaj, Ramachandran Ramjee, Muthian Sivathanu, Nipun Kwatra, Zhenhua Han, Pratyush Patel, Xuan Peng, Hanyu Zhao, Quanlu Zhang, Fan Yang, Lidong Zhou

13th USENIX Symposium on Operating Systems Design and Implementation (OSDI 2018) | October 2018

Publication Project Project

Towards Efficient Large Multimodal Model Serving

Haoran Qiu, Anish Biswas, Zihan Zhao, Jayashree Mohan, Alind Khare, Esha Choukse, Íñigo Goiri, Zeyu Zhang, Haiying Shen, Chetan Bansal, Ramachandran Ramjee, Rodrigo Fonseca

February 2025

Preprint Project

SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills

Amey Agrawal, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav S. Gulavani, Ramachandran Ramjee

September 2023

Preprint

Singularity: Planet-Scale, Preemptive and Elastic Scheduling of AI Workloads

Dharma Shukla, Muthian Sivathanu, Srinidhi Viswanatha, Bhargav Gulavani, Rimma Nehme, Amey Agrawal, Chen Chen, Nipun Kwatra, Ramachandran Ramjee, Pankaj Sharma, Atul Katiyar, Vipul Modi, Vaibhav Sharma, Abhishek Singh, Shreshth Singhal, Kaustubh Welankar, Lu Xun, Ravi Anupindi, Karthik Elangovan, Hasibur Rehman, Zhou Lin, Rahul Seetharaman, Cheng Xu, Eddie Ailijiang, Suresh Krishnappa, Mark Russinovich

February 2022

Publication