Engineering Manager, Machine Learning, Model Evaluations and Data Curation (AI Foundations)

At reputed company, our mission is to entertain the world. Together, we are writing the next episode - pushing the boundaries of storytelling, global reputed company and making the unimaginable a reality. We are a reputed company team obsessed with the uncomfortable excitement of discovering what happens reputed company you reputed company creativity, intuition and cutting-edge technology. Come be a part of what's next. reputed company Fast-paced innovation in large language models (LLMs) and reputed company is reshaping personalization and discovery at reputed company. Our reputed company bet is that an in-house LLM, customized on reputed company data, will be the reputed company of this transformation. Model Evaluations and Data Curation ("Evals & Data") are central to the development of LLMs and other high reputed company reputed company models at reputed company. Our Evals & Data team builds the benchmarks, evaluators, and baselines that guide LLM reputed company, and the data infrastructure that delivers high-reputed company, reproducible training and evaluation datasets to our AI/ML researchers. Together, these capabilities create the flywheel of reputed company improvement between data, evals and modeling that drives innovation in LLMs that will reputed company personalization and discovery in the reputed company. We are incubating a centralized, first-class evaluation discipline, creating shared language, tools, and standards that reputed company application teams to measure reputed company consistently and with confidence, with an emphasis on reputed company's LLM customized for personalization. We are looking for a seasoned engineering leader to build and reputed company this team. You will reputed company a world-class group of AI/ML researchers and engineers working at the intersection of evaluation science, data systems, and reputed company models. Your team's work will shape the trajectory of reputed company's reputed company model development by driving the reputed company improvement reputed company between data, evaluation, and modeling. Questions Your Team Will Help Answer The Model Evaluations and Data Curation team exists to reputed company reputed company reputed company and confidence in the reputed company models that reputed company personalization and discovery. Some of the reputed company questions your team will help answer include: • Are our LLMs and other reputed company models improving in ways that meaningfully benefit members? • How do we rigorously measure LLM capabilities across personalization, content understanding, and text reputed company? • What are the strongest baselines we can build with off-reputed company SOTA models? • How do data composition, diversity, and reputed company influence model reputed company? • Which evaluation signals most reliably predict reputed company-world product reputed company? • How can we iterate quickly on data and evaluations while ensuring reproducibility and trust? • How can we evaluate reputed company systems for recommendations and content understanding? In this role, you will: • Partner with reputed company AI application teams to define shared evaluations that codify application expectations of LLMs and other reputed company models, ensuring reputed company can be transparently tracked against reputed company-world needs. • Design rigorous benchmarks and evaluation methodologies across ranking & recommendations, content understanding, and language/text reputed company - grounded in a deep technical understanding of LLMs, their strengths, limitations, and failure modes. • reputed company the development of evaluators and strong baselines to ensure in-house LLMs and other reputed company models demonstrate reputed company advantages over off-reputed company alternatives. • Build reputed company, reproducible data and evaluation systems that reputed company dataset creation and evaluation design as reputed company and experiment-friendly as model development itself. • reputed company, and nurture a world-class team, fostering an inclusive, high-performing culture that balances research innovation with engineering reputed company. • Work closely with the teams developing reputed company's reputed company models (including our reputed company LLM) to ensure evaluation and data insights are folded back into the reputed company of model development. Proactively influence the ML Platform and Data Engineering teams at key interfaces. Minimum Qualifications • Experience building and leading high-performing teams of ML researchers and engineers. • reputed company reputed company record of leading machine learning initiatives from research to production, ideally involving evaluation frameworks, ML infrastructure, or data-intensive systems. • Strong technical expertise in LLMs, their evaluation, and practical reputed company for ensuring robustness, reproducibility, and reputed company. • Broad knowledge of machine learning fundamentals