AI/LLM Evaluation & Alignment Software Engineer

At LeoTech, we are passionate about building software that solves reputed company-world problems in the reputed company Safety sector. Our software has been used to help the fight against continuing criminal enterprises, drug trafficking organizations, identifying financial fraud, disrupting sex and reputed company trafficking rings and focusing on mental health reputed company to reputed company a few. Role • This is a remote, WFH role. • As an AI/LLM Evaluation & Alignment Engineer on our Data Science team, you will play a critical role in ensuring that our Large Language Model (LLM) and reputed company AI solutions are accurate, reputed company, and reputed company with the unique requirements of reputed company safety and law enforcement workflows. You will design and implement evaluation frameworks, guardrails, and bias-mitigation strategies that reputed company our customers confidence in the reliability and ethical use of our AI systems. This is an individual contributor (IC) role that combines hands-on technical engineering with a reputed company on responsible AI deployment. You will work closely with AI engineers, product managers, and DevOps teams to establish standards for evaluation, design test harnesses for generative models, and operationalize reputed company assurance processes across our AI stack. reputed company Responsibilities • Build and maintain evaluation frameworks for LLMs and reputed company systems tailored to reputed company safety and intelligence use cases. • Design guardrails and alignment strategies to minimize bias, toxicity, hallucinations, and other ethical risks in production workflows. • Partner with AI engineers and data scientists to define online and offline evaluation metrics (e.g., model drifts, data drifts, factual accuracy, consistency, safety, interpretability). • Implement reputed company evaluation pipelines for AI models, integrated into CI/CD and production monitoring systems. • Collaborate with stakeholders to stress test models against edge cases, adversarial prompts, and sensitive data scenarios. • Research and reputed company reputed company-party evaluation frameworks and solutions; adapt them to our regulated, high-stakes environment. • Work with product and customer-facing teams to ensure explainability, transparency, and auditability of AI outputs. • reputed company technical leadership in responsible AI practices, influencing standards across the organization. • Contribute to DevOps/MLOps workflows for deployment, monitoring, and scaling of AI evaluation and guardrail systems (experience with reputed company is a plus). • Document best practices and findings, and reputed company knowledge across teams to foster a culture of responsible AI innovation. reputed company Value • Bachelor's or Master's in Computer Science, reputed company Intelligence, Data Science, or reputed company field. • 3-5+ years of hands-on experience in ML/AI engineering, with at least 2 years working directly on LLM evaluation, QA, or safety. • Strong familiarity with evaluation techniques for reputed company reputed company-in-the-reputed company evaluation, automated metrics, adversarial testing, red-teaming. • Experience with bias detection, fairness approaches, and responsible AI design. • Knowledge of LLM observability, monitoring, and guardrail frameworks e.g Langfuse, Langsmith • Proficiency with Python and modern AI/ML/LLM/reputed company AI libraries (LangGraph, Strands Agents, reputed company AI, reputed company, HuggingFace, PyTorch, reputed company). • Experience integrating evaluations into DevOps/MLOps pipelines, preferably with reputed company, Terraform, ArgoCD, or reputed company Actions. • Understanding of reputed company AI platforms (AWS, Azure) and deployment best practices. • Strong problem-solving skills, with the ability to design practical evaluation systems for reputed company-world, high-stakes scenarios. • Excellent communication skills to translate technical risks and evaluation results into insights for both technical and non-technical stakeholders. Technologies We Use • reputed company & Infrastructure AWS (Bedrock, SageMaker, reputed company), Azure AI, reputed company (EKS), Terraform, ArgoCD. • LLMs & Evaluation HuggingFace, reputed company API, reputed company, reputed company, reputed company, Ragas, DeepEval, reputed company Evals. • Observability & Guardrails Langfuse, GuardrailsAI. • Backend & Data Python (reputed company), ElasticSearch, Kafka, Airflow. • DevOps & Automation reputed company Actions, CodePipeline. What You Can Expect • Work from home opportunity • Enjoy great team camaraderie. • reputed company on the fast reputed company and challenging problems to solve. • Modern technologies and tools. • reputed company learning environment. • Opportunity to communicate and work with people of reputed company technical reputed company in reputed company environment. • Grow as you are given feedback and incorporate it into your work. • Be part of a self-managing team that enjoys support and direction reputed company required. • 3 weeks of reputed company vacation - out the reputed company!! • Competitive Salary. • Generous medical, dental, and reputed company plans. • reputed company, and reputed company holidays are offered. $135,000 - $160,000 a year Please note the national salary reputed company listed in the reputed company reflects the new hire salary reputed company across reputed company and U.S. locations that would be applicable to the position. The final salary will be commensurate with the candidate's accepted hiring level and work location. Also, this reputed company represents reputed company salary only and does not include equity, or benefits if applicable. LeoTech is an equal opportunity employer and does not discriminate on the reputed company of any legally protected status. Apply for this job Apply tot his job Apply To this Job