Skip to content

feat(check): rename the status to Vapi Simulations; one target, one status - #82

Merged
scott-lowe-vapi merged 1 commit into
mainfrom
feat/vapi-simulations-status
Oct 14, 2026
Merged

scott-lowe-vapi merged 1 commit into
mainfrom
feat/vapi-simulations-status

Conversation

@scott-lowe-vapi

@scott-lowe-vapi scott-lowe-vapi commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

Value

V.A.L.U.E. tier: small. It changes the name of a commit status that customers may require in branch protection. Not micro: it changes behavior, not just intended behavior restored.

  • Problem: the PR check posts a status called Vapi Evals, but every run is a simulation, and the public docs now separate Simulations (whole conversations) from Evals (single decisions). Next to "4 of 4 simulations passed", the name blurs exactly that line. Dogfooding also showed a single-target repo gets an overall status plus a per-target status that say the same thing and link the same run: three Vapi rows on a PR where two would do.
  • Who it affects: every repo that turns on PR checks; most have one target.
  • What changes:
    • The status is now Vapi Simulations, and per-target statuses are Vapi Simulations / <check> / <target>. The job summary heading matches.
    • With one target, an --all run posts only the overall status. It's pending with the run link while the run is going, then the verdict, still linking the run. With several targets, each still gets its own status linking its run, and the overall status links the workflow run. Running one named check by hand is unchanged.
    • Docs: the README and PR checks guide use the new name, with a migration note: a repo that already requires Vapi Evals must require Vapi Simulations after merging the update, or PRs will wait on a status that no longer posts.

Alignment needed: Agent Loop (PAL-608 specifies Vapi Evals for the planned shared GitHub Action). Merge after they agree, or update PAL-608 to match, so the two don't diverge. The public docs page (docs.vapi.ai/config-as-code) also names Vapi Evals: VapiAI/docs#1330 renames it there, and should merge after this PR.

Evidence of value

Repo with Vapi rows on a PR, before After
1 target 3 (job, overall, 1 per-target) 2 (job, overall linking the run)
N targets N + 2 N + 2, unchanged
  • One target: check-cmd.test.ts runs a live --all against a stub. It posts only Vapi Simulations (pending, then success), both linking the run.
  • Several targets: a second check makes two targets. Each gets a pending and a success status, and the overall status links the workflow run.
  • Manual named run: the existing test still sees per-target statuses only.

Testing plan

  • npm test (547) and npx tsc --noEmit pass.
  • After merge: pull the update into the dogfood repo and check the PR shows Vapi Simulations, with two rows for the goat and lebron checks (two targets).
  • Known flaky test (not introduced here): a check-cmd status test failed once in about 28 full runs and didn't reproduce. A similar one-off happened on an earlier branch. Worth a dedicated look; not blocking.

Refs TEST-141

🤖 Generated with Claude Code

…tatus

The PR check posted `Vapi Evals`, though every run is a simulation and
the docs now separate Simulations (whole conversations) from Evals (single
decisions). It is now `Vapi Simulations`, matching the product the
Details link opens.

With one target, an --all run no longer posts a per-target status that
repeats the aggregate: the aggregate carries the pending state and links
straight to the run. With several targets, each still gets its own status
linking its run. Running one named check by hand still posts per-target
statuses only.

- src/check-status.ts, check-cmd.ts, check-report.ts and the workflow use
  the new name.
- Tests: one target posts only the aggregate (pending, then the verdict,
  both linking the run); several targets each get a status and the
  aggregate links the workflow run.
- Docs: README and the PR checks guide use the new name, with a note for
  repos that already require `Vapi Evals` in branch protection.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Copy link
Copy Markdown
Contributor Author

This stack of pull requests is managed by Graphite. Learn more about stacking.

@scott-lowe-vapi
scott-lowe-vapi marked this pull request as ready for review October 14, 2026 04:15
stephenvapiai pushed a commit to VapiAI/docs that referenced this pull request Oct 14, 2026
Follows VapiAI/gitops#82, which renames the status the gitops PR check
posts from Vapi Evals to Vapi Simulations.
@scott-lowe-vapi
scott-lowe-vapi merged commit 898a75d into main Oct 14, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants