Metron: Holistic Performance Evaluation Framework for LLM Inference Systems
Amey Agrawal, Anmol Agarwal, Nitin Kedia, Jayashree Mohan, Souvik Kundu, Nipun Kwatra, Ramachandran Ramjee, Alexey Tumanov
July 2024
Publications
Publications by Year
-
vAttention: Dynamic Memory Management for Serving LLMs without PagedAttention
Ramya Prabhu, Ajay Nayak, Jayashree Mohan, Ramachandran Ramjee, Ashish Panwar
May 2024
VIDUR: A Large-Scale Simulation Framework for LLM Inference
Amey Agrawal, Nitin Kedia, Jayashree Mohan, Ashish Panwar, Nipun Kwatra, Bhargav S. Gulavani, Ramachandran Ramjee, Alexey Tumanov
The Seventh Annual Conference on Machine Learning and Systems | May 2024
Just-In-Time Checkpointing: Low Cost Error Recovery from Deep Learning Training Failures
Tanmaey Gupta, Sanjeev Krishnan, Rituraj Kumar, Abhishek Vijeev, Bhargav Gulavani, Nipun Kwatra, Ramachandran Ramjee, Muthian Sivathanu
2024 European Conference on Computer Systems | April 2024
Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve
Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav Gulavani, Alexey Tumanov, Ramachandran Ramjee
OSDI ’24 | March 2024
-
SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills
Amey Agrawal, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav S. Gulavani, Ramachandran Ramjee
September 2023
Renee: End-to-end training of extreme classification models
Vidit Jain, Jatin Prakash, Deepak Saini, Jian Jiao, Ramachandran Ramjee, Manik Varma
In Proceedings of the Conference on Machine Learning and Systems, Miami, Florida | June 2023
-
Varuna: Scalable, Low-cost Training of Massive Deep Learning Models
Sanjith Athlur, Nitika Saran, Muthian Sivathanu, Ramachandran Ramjee, Nipun Kwatra
EuroSys 2022 | April 2022
Best Paper Award
Singularity: Planet-Scale, Preemptive and Elastic Scheduling of AI Workloads
Dharma Shukla, Muthian Sivathanu, Srinidhi Viswanatha, Bhargav Gulavani, Rimma Nehme, Amey Agrawal, Chen Chen, Nipun Kwatra, Ramachandran Ramjee, Pankaj Sharma, Atul Katiyar, Vipul Modi, Vaibhav Sharma, Abhishek Singh, Shreshth Singhal, Kaustubh Welankar, Lu Xun, Ravi Anupindi, Karthik Elangovan, Hasibur Rehman, Zhou Lin, Rahul Seetharaman, Cheng Xu, Eddie Ailijiang, Suresh Krishnappa, Mark Russinovich
February 2022
-
Balancing efficiency and fairness in heterogeneous GPU clusters for deep learning
Shubham Chaudhary, Ramachandran Ramjee, Muthian Sivathanu, Nipun Kwatra, Srinidhi Viswanatha
Fifteenth European Conference on Computer Systems (EuroSys’20) | April 2020
Quiver: An Informed Storage Cache for Deep Learning
Abhishek Vijaya Kumar, Muthian Sivathanu
18th USENIX Conference on File and Storage Technologies (FAST 2020) | February 2020
-
Astra: Exploiting Predictability to Optimize Deep Learning
Muthian Sivathanu, Tapan Chugh, Sanjay S. Singapuram, Lidong Zhou
ACM Symposium on Architectural Support for Programming Languages and Operating Systems (ASPLOS 2019) | April 2019
-
Gandiva: Introspective Cluster Scheduling for Deep Learning
Wencong Xiao, Romil Bhardwaj, Ramachandran Ramjee, Muthian Sivathanu, Nipun Kwatra, Zhenhua Han, Pratyush Patel, Xuan Peng, Hanyu Zhao, Quanlu Zhang, Fan Yang, Lidong Zhou
13th USENIX Symposium on Operating Systems Design and Implementation (OSDI 2018) | October 2018
Publications by Research Area
-
Metron: Holistic Performance Evaluation Framework for LLM Inference Systems
Amey Agrawal, Anmol Agarwal, Nitin Kedia, Jayashree Mohan, Souvik Kundu, Nipun Kwatra, Ramachandran Ramjee, Alexey Tumanov
July 2024
vAttention: Dynamic Memory Management for Serving LLMs without PagedAttention
Ramya Prabhu, Ajay Nayak, Jayashree Mohan, Ramachandran Ramjee, Ashish Panwar
May 2024
VIDUR: A Large-Scale Simulation Framework for LLM Inference
Amey Agrawal, Nitin Kedia, Jayashree Mohan, Ashish Panwar, Nipun Kwatra, Bhargav S. Gulavani, Ramachandran Ramjee, Alexey Tumanov
The Seventh Annual Conference on Machine Learning and Systems | May 2024
Just-In-Time Checkpointing: Low Cost Error Recovery from Deep Learning Training Failures
Tanmaey Gupta, Sanjeev Krishnan, Rituraj Kumar, Abhishek Vijeev, Bhargav Gulavani, Nipun Kwatra, Ramachandran Ramjee, Muthian Sivathanu
2024 European Conference on Computer Systems | April 2024
Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve
Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav Gulavani, Alexey Tumanov, Ramachandran Ramjee
OSDI ’24 | March 2024
SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills
Amey Agrawal, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav S. Gulavani, Ramachandran Ramjee
September 2023
Varuna: Scalable, Low-cost Training of Massive Deep Learning Models
Sanjith Athlur, Nitika Saran, Muthian Sivathanu, Ramachandran Ramjee, Nipun Kwatra
EuroSys 2022 | April 2022
Best Paper Award
Singularity: Planet-Scale, Preemptive and Elastic Scheduling of AI Workloads
Dharma Shukla, Muthian Sivathanu, Srinidhi Viswanatha, Bhargav Gulavani, Rimma Nehme, Amey Agrawal, Chen Chen, Nipun Kwatra, Ramachandran Ramjee, Pankaj Sharma, Atul Katiyar, Vipul Modi, Vaibhav Sharma, Abhishek Singh, Shreshth Singhal, Kaustubh Welankar, Lu Xun, Ravi Anupindi, Karthik Elangovan, Hasibur Rehman, Zhou Lin, Rahul Seetharaman, Cheng Xu, Eddie Ailijiang, Suresh Krishnappa, Mark Russinovich
February 2022
Balancing efficiency and fairness in heterogeneous GPU clusters for deep learning
Shubham Chaudhary, Ramachandran Ramjee, Muthian Sivathanu, Nipun Kwatra, Srinidhi Viswanatha
Fifteenth European Conference on Computer Systems (EuroSys’20) | April 2020
Quiver: An Informed Storage Cache for Deep Learning
Abhishek Vijaya Kumar, Muthian Sivathanu
18th USENIX Conference on File and Storage Technologies (FAST 2020) | February 2020
Astra: Exploiting Predictability to Optimize Deep Learning
Muthian Sivathanu, Tapan Chugh, Sanjay S. Singapuram, Lidong Zhou
ACM Symposium on Architectural Support for Programming Languages and Operating Systems (ASPLOS 2019) | April 2019
Gandiva: Introspective Cluster Scheduling for Deep Learning
Wencong Xiao, Romil Bhardwaj, Ramachandran Ramjee, Muthian Sivathanu, Nipun Kwatra, Zhenhua Han, Pratyush Patel, Xuan Peng, Hanyu Zhao, Quanlu Zhang, Fan Yang, Lidong Zhou
13th USENIX Symposium on Operating Systems Design and Implementation (OSDI 2018) | October 2018
-
Renee: End-to-end training of extreme classification models
Vidit Jain, Jatin Prakash, Deepak Saini, Jian Jiao, Ramachandran Ramjee, Manik Varma
In Proceedings of the Conference on Machine Learning and Systems, Miami, Florida | June 2023
-
Renee: End-to-end training of extreme classification models
Vidit Jain, Jatin Prakash, Deepak Saini, Jian Jiao, Ramachandran Ramjee, Manik Varma
In Proceedings of the Conference on Machine Learning and Systems, Miami, Florida | June 2023
Quiver: An Informed Storage Cache for Deep Learning
Abhishek Vijaya Kumar, Muthian Sivathanu
18th USENIX Conference on File and Storage Technologies (FAST 2020) | February 2020
Astra: Exploiting Predictability to Optimize Deep Learning
Muthian Sivathanu, Tapan Chugh, Sanjay S. Singapuram, Lidong Zhou
ACM Symposium on Architectural Support for Programming Languages and Operating Systems (ASPLOS 2019) | April 2019
Publications by Type
-
Metron: Holistic Performance Evaluation Framework for LLM Inference Systems
Amey Agrawal, Anmol Agarwal, Nitin Kedia, Jayashree Mohan, Souvik Kundu, Nipun Kwatra, Ramachandran Ramjee, Alexey Tumanov
July 2024
VIDUR: A Large-Scale Simulation Framework for LLM Inference
Amey Agrawal, Nitin Kedia, Jayashree Mohan, Ashish Panwar, Nipun Kwatra, Bhargav S. Gulavani, Ramachandran Ramjee, Alexey Tumanov
The Seventh Annual Conference on Machine Learning and Systems | May 2024
Just-In-Time Checkpointing: Low Cost Error Recovery from Deep Learning Training Failures
Tanmaey Gupta, Sanjeev Krishnan, Rituraj Kumar, Abhishek Vijeev, Bhargav Gulavani, Nipun Kwatra, Ramachandran Ramjee, Muthian Sivathanu
2024 European Conference on Computer Systems | April 2024
Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve
Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav Gulavani, Alexey Tumanov, Ramachandran Ramjee
OSDI ’24 | March 2024
Renee: End-to-end training of extreme classification models
Vidit Jain, Jatin Prakash, Deepak Saini, Jian Jiao, Ramachandran Ramjee, Manik Varma
In Proceedings of the Conference on Machine Learning and Systems, Miami, Florida | June 2023
Varuna: Scalable, Low-cost Training of Massive Deep Learning Models
Sanjith Athlur, Nitika Saran, Muthian Sivathanu, Ramachandran Ramjee, Nipun Kwatra
EuroSys 2022 | April 2022
Best Paper Award
Balancing efficiency and fairness in heterogeneous GPU clusters for deep learning
Shubham Chaudhary, Ramachandran Ramjee, Muthian Sivathanu, Nipun Kwatra, Srinidhi Viswanatha
Fifteenth European Conference on Computer Systems (EuroSys’20) | April 2020
Quiver: An Informed Storage Cache for Deep Learning
Abhishek Vijaya Kumar, Muthian Sivathanu
18th USENIX Conference on File and Storage Technologies (FAST 2020) | February 2020
Astra: Exploiting Predictability to Optimize Deep Learning
Muthian Sivathanu, Tapan Chugh, Sanjay S. Singapuram, Lidong Zhou
ACM Symposium on Architectural Support for Programming Languages and Operating Systems (ASPLOS 2019) | April 2019
Gandiva: Introspective Cluster Scheduling for Deep Learning
Wencong Xiao, Romil Bhardwaj, Ramachandran Ramjee, Muthian Sivathanu, Nipun Kwatra, Zhenhua Han, Pratyush Patel, Xuan Peng, Hanyu Zhao, Quanlu Zhang, Fan Yang, Lidong Zhou
13th USENIX Symposium on Operating Systems Design and Implementation (OSDI 2018) | October 2018
-
vAttention: Dynamic Memory Management for Serving LLMs without PagedAttention
Ramya Prabhu, Ajay Nayak, Jayashree Mohan, Ramachandran Ramjee, Ashish Panwar
May 2024
SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills
Amey Agrawal, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav S. Gulavani, Ramachandran Ramjee
September 2023
-
Singularity: Planet-Scale, Preemptive and Elastic Scheduling of AI Workloads
Dharma Shukla, Muthian Sivathanu, Srinidhi Viswanatha, Bhargav Gulavani, Rimma Nehme, Amey Agrawal, Chen Chen, Nipun Kwatra, Ramachandran Ramjee, Pankaj Sharma, Atul Katiyar, Vipul Modi, Vaibhav Sharma, Abhishek Singh, Shreshth Singhal, Kaustubh Welankar, Lu Xun, Ravi Anupindi, Karthik Elangovan, Hasibur Rehman, Zhou Lin, Rahul Seetharaman, Cheng Xu, Eddie Ailijiang, Suresh Krishnappa, Mark Russinovich
February 2022