Part-Time AI Operations Analyst

Definable Solutions is looking for a part-time AI Operations Analyst to support our AI development team. This role will play a pivotal part in building School Data Connect AI – an AI-powered analytics layer that helps K-12 districts get answers from their data in plain language. The product is live with real users, and the AI Operations Analyst will work closely with the development team to continue to grow the customer base and keep the product performing well. This role is operationally focused, you will own the monitoring, analysis, and tuning workflows that keep our AI agent accurate, efficient, and trustworthy. Though this position about 20 hour per week to start, we expect for this to grow into a full-time role over time. A successful candidate will be curious, have analytic rigor, attention to detail, and the ability to pick up technical concepts quickly. If you have ever wanted to work at the intersection of data, language models, and real-world product impact in education, we’d love to hear from you. Responsibilities: Agent Performance Monitoring • Pull and review conversation logs on a regular cadence to • identify failure patterns, unexpected behaviors, and edge cases. • Tag and categorize query outcomes to build a structured picture of where the AI Agent succeeds and where it struggles. • Flag performance regressions and anomalies before they reach end users, with documented findings escalated to the AI Engineer.internal review.Cost Optimization Analysis • Maintain performance dashboards and reporting summaries for • Track cost-per-exchange metrics across model configurations • and usage patterns. • Analyze query logs to surface optimization opportunities. • Support prompt caching analysis and document efficiency findings for engineer review.output quality, flagging trade-offs.Prompt Tuning Support • Monitor the relationship between configuration choices and • Draft prompt revisions based on observed failure patterns, • for review and approval by the AI Engineer. • Run structured tests on prompt candidates and document results in a consistent format. • Support guardrail tuning by analyzing edge cases and documentingchanged, why, and what outcome was observed.Documentation & Consultant Support • Maintain a prompt iteration log that tracks what was • Update schema documentation as the underlying data model • evolves, in coordination with engineering. • Maintain internal runbooks and operational documentation for AI monitoring workflows. • Prepare materials and document findings summaries for advisory sessions with our retained consultant.Requirements: • Comfort working with data (querying, organizing, interpreting, and presenting structured information) • Strong written communication • High attention to detail • Ability to spot patterns in large volumes of text or log data • Intellectual curiosity and a demonstrated ability to learn new technical concepts independently. • Proficiency with productivity tools (spreadsheets, documents, task tracking systems). Preferred: • Bachelor's degree in a technical or analytical field, or equivalent demonstrated capability. • Familiarity with SQL or data querying. • Exposure to Python or scripting environments. • Prior exposure to AI or LLM tooling • Experience in education technology, data analytics, or a SaaS product environment. Traits We Value • You read documentation carefully and ask good clarifying questions. • You are comfortable with ambiguity and can make structured progress without complete direction. • You take ownership of the workflows you are responsible for and follow through. • You communicate proactively when you encounter blockers or unexpected findings. What You’ll Learn: This role comes with a structured development and growth path. In your first year, you will develop working knowledge in: • Production LLM operations — how large language models behave in real applications, where they fail, and how to address it. • Prompt engineering — writing, testing, and iterating prompts for agentic AI systems. • AI cost management — how configuration, caching, and query patterns affect the economics of AI-powered products. • Evaluation methodology — how to design and run structured tests on AI behavior. • K-12 education data — the data model that powers SchoolData Connect AI. As we expand the Connect AI customer base, we expect this position to move from 20 hours per week to a full-time role, and increased compensation range. High performers are well-positioned to grow into a dedicated AI Engineer or ML Operations role as the product scales. Pay: $20.00 - $24.00 per hour Benefits: • Flexible schedule Application Question(s): • Are you able to work 20 hours per week consistently, overlapping with US Business hours? Work Location: Remote