I am a Professor at the Institute of Science and Technology Austria (ISTA), and ML Research Lead at Neural Magic, Inc.

My research focuses on efficient algorithms and systems for machine learning, and spans from algorithms and lower bounds, to practical implementations. Before ISTA, I was a researcher at ETH Zurich and Microsoft Research, Cambridge, UK. Prior to that, I was a Postdoctoral Associate at MIT CSAIL, working with Prof. Nir Shavit. I received my PhD from the EPFL, under the guidance of Prof. Rachid Guerraoui.

During Fall 2023, I was a Visiting Professor at MIT.

My research is supported by a the Austrian FWF Center of Excellence BILAI, ERC Proof-of-Concept and Starting Grants, and generous grants from NVIDIA, Google, and Amazon.

Our lab’s code can be found at https://github.com/IST-DASLab

Open Positions

We often have open intern, PhD and postdoc positions. The ideal candidate would have a strong background in CS or Math, with a PhD in CS or a related field. Applicants with experience in  optimization theory or distributed systems implementation are particularly encouraged, although all strong applicants will be given proper consideration.
For questions about the positions and the application process, please contact me via email at dan.alistarh@ist.ac.at. The application should contain a CV, publication list, a statement describing motivation and research interests, and, if applicable, links to your publications. 
Due to time constraints, I may not be able to answer all queries.

News:

  • Our group has 4 research papers accepted at NeurIPS 2024, of which one for Oral Presentation: PV-Tuning (fine-tuning for quantized models), microAdam (space-efficient optimization), Iterative OBS (accurate sparsification), and QuaRot (quantized LLMs).
  • Our work on Marlin (quantized kernels for mixed-precision LLM inference) has been accepted to PPoPP 2025!
  • Our work on benchmarking LLM mathematical skills (MathadorLM) and on quantizing weights and activations for LLMs (QUIK) has been accepted to EMNLP 2024!
  • Elias Frantar graduated in early September! His work on LLM compression including GPTQ, Marlin, SparseGPT and QMoE was quite influential in the area of Machine Learning efficiency: for instance, GPTQ models have been downloaded millions of times from repositories such as HuggingFace! He has decided to join OpenAI.
  • We have had 4 papers accepted to ICML 2024, on sparsity for interpretability (SPADE), extreme LLM compression (AQLM), efficient sparse-quantized LLM fine-tuning (RoSA), and compressed preconditioners in optimization.
  • Two papers at MLSys 2024: QMoE (quantized execution of massive MoEs) and L-GreCo (layer-wise adaptive optimization of gradient compression).
  • Our lab presented two papers at ICLR 2024: Scaling laws for sparse models (Spotlight), and SpQR, a sparse-quantized format for accurate compressed LLM inference.
  • We presented three papers at NeurIPS 2023: a state of the art unstructured pruning algorithm (CAP), a Variance-Reduction interpretation of Knowledge Distillation, and a state-of-the-art algorithm for structured pruning (ZipLM)
  • Our lab had three papers accepted to ICML 2023 (SparseGPT, SparseProp and QSDP), of which the former two for oral presentation!
  • Our work on analyzing the dependency depth of Union-Find was accepted to SPAA23, where it won a distinguished paper award. Congratulations to Alex Fedorov, who led the work!
  • Our long-term project on fast, scalable and expressive channels in Kotlin led to papers in PLDI23 and PPoPP23, whereas Nikita Koval’s Lincheck tool was accepted to CAV23.
  • Our paper on Lower Bounds for Leader Election under Bounded Contention won the best paper award at DISC 2021. 
  • Together with Torsten Hoefler, I gave a tutorial at ICML 2021 on Sparsity in Deep Learning. The recordings are available here, and the JMLR survey on which the tutorial is based is available here. 

Recent/Future service: 

  • Editor for Journal of Machine Learning Research (JMLR) and Distributed Computing Journal
  • PC Chair of the 38th International Symposium on Distributed Computing (DISC 2024)
  • Local Chair for ICML 2024
  • Selected PCs: PODC 2024, MLSys 2024, NeurIPS 2023 (AC), NeurIPS 2022, NeurIPS 2021, DCC 2021 (Program Chair), DISC 2021, ICDCS 2021 (Track Chair), PPoPP 2021, MLSys 2021
  • FOCS 2020, PODC 2020, PPoPP 2020, AAAI 2020, MLSys 2020, NeurIPS 2019, DISC 2019, ICML 2019, ISCA 2019 (external), ICML 2019, SysML 2019, PPoPP 2019, NIPS 2018, DISC 2018, PODC 2018, ICDCS 2018, DISC 2017, ALGOCLOUD 2017 (PC chair)

Members & Alumni

I am extremely lucky to be able to work with the following students and postdocs:

  • Mher Safaryan (Postdoc @IST)
  • Roberto L. Castro (Postdoc @IST)
  • Eldar Kurtic (Researcher @IST)
  • Jen Iofinova (PhD Student @IST)
  • Mahdi Nikdan (PhD Student @IST)
  • Ionut Modoranu (PhD Student @IST)
  • Jiale Chen (PhD Student @IST)
  • Alexander Fedorov (PhD Student @IST)

Visitors / Collaborators / Friends of the Lab: 

  • Prof. Nahmoon Lee (visiting early 2025)
  • Prof. Faith Ellen (visiting February-March 2020, June 2024)
  • Prof. Nir Shavit (visiting November 2019, September 2024)
  • Prof. Thomas Sauerwald (visiting Oct 2019)
  • Prof. Gregory Valiant (visiting Nov 2018)
  • Prof. Robert Tarjan (visiting May 2018)
  • Dr. Frank McSherry (visiting May 2018)
  • Dr. Aleksandar Prokopec (visiting April 2018)
  • Prof. Ce Zhang (ETH)
  • Prof. Markus Pueschel (ETH)
  • Prof. Torsten Hoefler (ETH)

Alumni:

  • Elias Frantar (PhD @IST -> OpenAI)
  • Ilia Markov (PhD @IST -> Neural Magic)
  • Alex Shevchenko (PhD @IST -> Postdoc @ETH)
  • Giorgi Nadiradze (PhD Student @ IST -> Aptos)
  • Alexandra Peste (PhD @ IST, co-advised with Christoph Lampert -> startup)
  • Sasha Voitovych (Intern @IST -> PhD Student at MIT)
  • Kimia Noorbakhsh (Intern @IST -> PhD Student at MIT)
  • Kaveh Alimohammadi (Intern @IST -> PhD Student at MIT)
  • Shayan Talaei (Intern @IST -> PhD Student at Stanford)
  • Joel Rybicki (PostDoc @IST -> Assistant Professor at HU Berlin)
  • Peter Davies-Peck (PostDoc @ IST -> Lecturer at University of Surrey)
  • Janne Korhonen (PostDoc @ IST)
  • Bapi Chatterjee (PostDoc @ IST -> Faculty at IIIT Delhi)
  • Vitaly Aksenov (PostDoc @ IST -> Lecturer @ ITMO)
  • Trevor Brown (PostDoc @ IST -> Assistant Professor @ Waterloo)
  • Amirmojtaba Sabour (Intern @ IST -> PhD Student @ UofT)
  • Vijaykrishna Gurunanthan (Intern @ IST -> PhD Student @Stanford)
  • Saleh Ashkboos (Intern @ IST -> PhD @ ETH Zurich)
  • Antonio Polino (MSc @ ETH -> Google Search)
  • Rati Gelashvili (PhD@MIT, now at Aptos)
  • Justin Kopinsky (PhD@MIT, now at JaneStreet)
  • Cédric Renggli (MSc Thesis (ETH Medal), PhD@ETH)
  • Nandini Singhal (Intern @ IST -> Microsoft Research)
  • Amirkeivan Mohtashami (Intern @ IST -> PhD at EPFL -> Anthropic)
  • Aditya Sharma (Intern @ IST -> Samsung)
  • Ekaterina Goltsova (ISTern @ IST -> Master’s @ EPFL)
  • Demjan Grubić (MSc at ETH -> Google)
  • Jerry Z. Li (Intern, PhD@MIT -> Microsoft Research -> Prof at UW)
  • Jenny Iglesias (Intern, PhD@CMU)
  • Syed Kamran Haider (Intern @ MSR, now Researcher @ Qualcomm)
  • Hyunjik Kim (Intern @ MSR -> PhD @ Oxford -> Researcher at DeepMind)