Senior Software Engineer (Applied AI and Evaluations)
• Smartsheet is building the next generation of AI-powered work management through SmartAssist, our intelligent agent platform. As we scale from early demos to production-grade agents, quality is the critical frontier and we are looking for a Agent Quality Engineer to own it
- This is not a QA role. It’s a deeply technical, high-autonomy position at the intersection of LLM evaluation, prompt and context engineering, and retrieval-augmented generation. You will diagnose why our agents fail, design the systems that catch regressions, and drive measurable improvements across our orchestrator and subagent fleet
- You will work closely with our Agent Engineering and AI Platform teams, embedded in a team that has already shipped evaluation infrastructure on Databricks/MLflow and is building toward a mature Agent Development Lifecycle (ADLC)
- Own agent quality end-to-end: diagnosis, improvement, and validation across SmartAssist’s orchestrator and subagents
- Identify failure modes across quality dimensions factual accuracy, completeness, tone, actionability, and latency and prioritize what to fix
- Drive quality improvements through prompt engineering, context engineering, and RAG retrieval tuning
- Extend and mature our evaluation framework: scorers, golden datasets, regression gates, and online evaluation for production traffic
- Close the feedback loop ensure that every change has a measurable, attributable quality signal
- Collaborate with our Agent Architecture lead to distinguish quality problems that require prompt/context solutions from those that require structural fixes
- Establish repeatable methodology that scales beyond any single agent or subagent
- In your first 6 months, you will have:
- Delivered measurable, validated quality improvement on at least one SmartAssist agent across our defined quality dimensions
- Expanded evaluation coverage to close the most significant blind spots in our current framework
- Established a repeatable quality improvement methodology the broader team can apply
- Why This Role:
- You will have real ownership, not a supporting role on someone else’s roadmap
- SmartAssist is shipping to real users; this work has immediate, visible impact
- You will be part of a fast-moving team that deeply values engineering rigor and actively seeks diverse perspectives
Benefits
- Lucrative Employee Stock Purchase Program (15% discount)
- 401k Match to help you save for your future (50% of your contribution up to the first 6% of your eligible pay)
- Monthly stipend to support your work and productivity
- Flexible Time Away Program, plus Incidental Sick Leave
- Up to 24 weeks of Parental Leave
- Personal paid Volunteer Day to support our community
- Opportunities for professional growth and development including access to LinkedIn Learning online courses
- Company Funded Perks, including a counseling membership, primary care membership, local retail discounts, and your own personal Smartsheet account
- US employees receive 12 paid holidays per year
- US employees are automatically covered under Smartsheet-sponsored life insurance, short-term, and long-term disability plans
- Teleworking options from any registered location (role specific)
- HSA, 100% employer-paid premiums, or Buy-up medical/vision and dental coverage options for full-time employees
- Equity - Restricted Stock Units (RSUs) with all offers
- Monthly contributions toward your pension
- Mployer-paid Private Medical and Dental, additional cost for family members