Marc Marone

I’m a final year PhD student at Johns Hopkins University advised by Benjamin Van Durme. My research centers around understanding the datasets that enable foundation models – from knowledge acquisition in pretraining to systematic data pipelines for reasoning, code, and multilingual capabilities. I recently interned at Meta, working on the team that builds datasets for large scale models (like Llama). I also interned at Databricks Mosaic Research, working on LLM datasets. My PhD work at JHU explores knowledge from pretraining and tools for understanding datasets - our academic work was honored at CoLM 2024 and I recently released a state-of-the-art multilingual encoder that is better, faster, and more multilingual than similar models. See News for more!

Some research problems I’m interested in:

• How can we efficiently query large datasets for LLMs (Data Portraits)? How does large data influence models as knowledge sources (Dated Data, outstanding paper award at CoLM 2024)?
• What do we need to build a highly multilingual encoder (MMBert)? How do encoders and decoders differ when trained on the same data (Ettin)?
• How can we find high quality web data?
• How can we efficiently balance compute used to create datasets with compute spent on training models? Are there favorable FLOP tradeoffs?

Before coming to Johns Hopkins, I worked with the research group at Microsoft Translate (under Hany Hassan Awadalla). My undergraduate degree is from Georgia Tech where I worked in Jacob Eisenstein’s lab. I also spent lots of time organizing educational events and campus outreach groups. I previously interned at Microsoft Semantic Machines and my other work experience includes undergrad SWE internships at Microsoft and a quantitative trading firm.