GovWorx is helping public safety rise to today's greatest challenge: the loss of experience. Our AI-powered platform, CommsCoach, supports 9-1-1 and emergency communications centers across the country by automating quality assurance, training, and real-time call evaluation—allowing agencies to strengthen their teams and better serve their communities.
We're looking for an experienced AI Evaluation Engineer to help build and improve the next generation of AI systems used by public safety agencies across the country. This role sits at the intersection of AI engineering, prompt engineering, and data science.
You'll own the evaluation and continuous improvement of production AI systems, developing automated evaluation pipelines, designing prompt experiments, analyzing model performance, and building tooling that enables rapid iteration. You'll work closely with data scientists, data engineers, and product managers to ensure our AI systems remain accurate, reliable, and trustworthy in real-world public safety environments.
Design, build, and maintain automated AI evaluation pipelines for production LLM applications
Develop prompt engineering strategies and iterate on prompts and compare LLMs using quantitative evaluation methods
Build offline evaluation datasets and regression testing frameworks to measure AI performance over time
Analyze production AI behavior using Python, SQL, and statistical techniques to identify opportunities for improvement
Design experiments, A/B tests, and benchmarking methodologies for evaluating prompt and model changes
Develop dashboards and reporting that communicate AI quality, reliability, and performance metrics
Partner with engineering and product teams to safely deploy and monitor improvements to production AI systems
Investigate model failures through detailed error analysis and recommend improvements to prompts, evaluation datasets, and workflows
Help establish best practices for Responsible AI, evaluation methodologies, and continuous model improvement
Must have US citizenship and pass FBI fingerprint and background check in multiple states
3+ years of experience in software engineering, machine learning, data science, or a related technical field
Experience designing evaluation metrics and interpreting AI model performance
Understanding of statistical methods including hypothesis testing and experiment design
Strong Python development experience
Strong SQL skills with experience analyzing large datasets
Experience building or supporting production LLM or Generative AI applications
Experience with prompt engineering and systematic prompt evaluation
Experience using AI evaluation or observability platforms such as Langfuse, LangSmith, MLflow, or Label Studio
Experience with AWS services such as Bedrock, Lambda, S3, Glue, or SageMaker
Experience building dashboards using Tableau, Sisense, Power BI, or similar tools
Knowledge of Responsible AI principles and evaluation methodologies
...someone to notice? Is your energy nonstop, your drive unshakable, and your sights set on leadership? Bast Hatfield Construction is hiring a project manager with a proven construction background and someone ready to own the process to lead multiple projects ranging from...
Kitchen Team Member Our kitchen is the heart of the restaurant. We take pride in the quality of our ingredients and the presentation of our food. Our fun team environment makes the kitchen a great place to work. We are looking for dependable, hard-working people to fill...
...require candidates to complete a language evaluation, an open-book qualification exam, and... ...online tools. We need you to assess search results for relevance and overall quality... ...improve the user experience for search engine users. We expect you to contribute to...
...Role: Search Platform Engineer Location: New York City, NY Employment type: C2C/W2 JOB DESCRIPTION Role Summary We're seeking a Search Platform Engineer with strong Apache Solr experience (primary) and ELK stack. The role focuses on Solr platform support...
...Job Description Description SUMMARY The role of the Project Manager is to plan, execute, and finalize projects according to strict... ...job costs. Educate colleagues to better understand construction documents. Willingness to become a LEED Accredited Professional...