Research Scientist

ABOUT THE ROLE

Join a research and engineering team developing benchmarks and evaluation methods for advanced language models. Your work will help AI labs and enterprise teams assess model capabilities and make informed decisions before deploying them in production.

WHAT YOU'LL DO

• Evaluate newly released AI models across a range of use cases and assess their capabilities.
• Build benchmarks from the ground up, including dataset development and coordination with labelers.
• Develop and improve automated methods for evaluating generated text.
• Work with engineering teams to implement and scale evaluation approaches in production systems.
• Collaborate with AI labs and enterprise teams to understand evaluation needs and shape research initiatives.
• Share research findings, including through publication where appropriate.

WHAT WE'RE LOOKING FOR

• At least 3 years of full-time AI or machine learning research experience, particularly in benchmarking, evaluation, NLP, or language models.
• A track record of published research, ideally in NLP or benchmarking, alongside an ability to turn research insights into practical systems.
• Strong Python skills and experience building maintainable software; experience with Django, Flask, or similar Python HTTP frameworks.
• Hands-on experience with PyTorch or TensorFlow, language modeling, and LLM infrastructure.
• Experience designing benchmarks and evaluation methodologies; familiarity with diffusion models and other relevant AI/ML techniques is useful.
• A background in a research lab, AI startup, or early-stage company, with experience collaborating through Git workflows, code review, and development sprints.
• A degree in Computer Science, Machine Learning, or a related field; a master's or PhD is preferred. Candidates with a bachelor's degree and 1 to 3 years of applied experience may also be considered.
• Strong communication skills and a collaborative approach to giving and incorporating feedback.

COMPENSATION & BENEFITS

Base salary is $150,000 to $200,000 annually. Visa sponsorship is available.

LOCATION

On-site in San Francisco, California.