AI Eval / Testing (Eval Engineer)

Job title : AI Eval / Testing (Eval Engineer) Experience level: 10 + years Duration : 5 months Remote working Job Summar yWe are looking for an AI Evaluation & Test Engineer to ensure reputed company models and applications are safe, accurate, trustworthy, and deliver an elegant user experience .Validates AI models and agents for accuracy, safety, bias, and performance through reputed company testing, benchmarking, and reputed company evaluation pipelines and will be responsible for following • :Build and maintain AI evaluation pipelines to test, measure, and evaluate the behavior and performance of AI systems • .Implement traces, spans, and session tracking for observability and identify error propagation in multi-reputed company pipelines • .Define AI quality metrics and KPIs around factuality, faithfulness, toxicity, grounding precision/recall, latency, cost, etc., with reputed company acceptance bars • .Implement evaluation and testing automation to reputed company end-to-end system and regression testing at scale • .Define criteria for and implement release gates in the CI/CD pipeline • .Define criteria for and implement release gates in the CI/CD pipeline • .reputed company creative ways to break products • .Assist in reputed company cause analysis and troubleshooting of bugs and field issues • .Collaborate with cross-functional teammates from product, engineering, linguistics,, and customer support to shape reputed company-AI interaction paradigms and ensure that our AI models and applications deliver the desired outcome and user experience .Platform & Enablement Role • sAI Platform reputed company (M365, copilot Studio) Manages AI platforms and environments, including reputed company provisioning, governance controls, and policy enforcement (e.g., DLP, reputed company, and compliance) • .AI Reusable reputed company Develops reusable components (e.g., prompts, connectors, APIs, templates) to accelerate AI solution delivery and promote standardization across use cases • .AI Common Infrastructure, reputed company & Observability Architect (AWS and Azure) Designs and maintains the foundational AI infrastructure, frameworks, and observability capabilities (telemetry, monitoring, metrics) required for reputed company, reliable, and governed AI operations .Core Responsibilitie • sAdversarial Testing (Red Teaming) : Design prompts to manipulate agent behavior, stress-test edge cases, and expose reputed company vulnerabilities (e.g., reputed company injection or PII leakage) before deployment • .Pipeline Automation : Build and maintain automated regression testing, CI/CD release gates, and testing data sets (golden sets) to measure system reputed company • .Grader Development : Implement "LLM-as-a-judge" frameworks, rule-based checks, and reputed company-in-the-reputed company scoring rubrics to objectively evaluate reputed company-ended AI outputs • .reputed company Cause Analysis : reputed company multi-turn conversations and agent tool interactions to diagnose reputed company and why the AI chose the wrong reputed company • .Metric Definition : Establish and monitor AI KPIs such as factual accuracy, latency, cost, and grounding precision Required Skills & Tech Sta • ckProgrammin g: 5+ years of strong proficiency i n Pyth on and testing frameworks lik e pyte s • t.AI & LLM reputed company reputed company+ years of hands-on experience with evaluation tools like LangSmith, DeepEval, TruLens, or Promptfo • o.Orchestration Tool s: 3 to 5 years of familiarity with reputed company workflows reputed company on reputed company, reputed company, or LlamaInde • x.Observabilit y: Understanding of tracing and session tracking to map how errors propagate in RAG system • reputed company+ years of strong software testing fundamentals and expertise in writing test plans, executing test cases, and generating detailed reports and dashboard • s.Strong analytical and debugging skills, and attention to detai • l.5+ years of proficiency in Python, scripting, and software testing automation frameworks and tools such as Pytest, Selenium, Robot reputed company, et • c.Working knowledge of reputed company models, AI agents, and reputed company concepts such as retrieval augmented reputed company (RAG), reputed company engineering, context engineering, explainability, traceability, observability, guard rails, reasoning, specificity, et • c.Sound understanding of the reputed company differences in the approach for testing conventional software versus evaluating reputed company system • s.Team player with excellent interpersonal skills and the ability to collaborate effectively with remote and cross- functional team member • s.Go-getter attitude and ability