Gate CI on evals

Run an eval in CI and fail the job when its gate fails.

Start Rundial and create an API key with the Quickstart. Keep the stack running and set RUNDIAL_API_KEY.

1. Create the CI script

Follow Run an eval and copy the Suite ID it prints. Save this as eval_ci.py. Replace the example scores with the results from your eval. Each job creates a new run in the existing suite.

import json
import os
import sys
from urllib.request import Request, urlopen

import rundial as rd

endpoint = os.environ.get("RUNDIAL_ENDPOINT", "http://127.0.0.1:8787")
key = os.environ["RUNDIAL_API_KEY"]
client = rd.RundialClient(endpoint=endpoint, api_key=key)
target = client.resolve_run_target(workspace="default-workspace", project="default-project")
suite = rd.RundialEvalSuite(
    client=client, org_id=target["orgId"], project_id=target["projectId"],
    suite={"suiteId": os.environ["RUNDIAL_SUITE_ID"]}, dataset={},
)
with suite.run() as evaluation:
    with evaluation.run.trace("example:refund") as span:
        if not evaluation.log_result(
            "refund", scores={"accuracy": 1.0, "groundedness": 0.8},
            passed=True, trace=span,
        ):
            raise RuntimeError("Could not record eval result")
    if not evaluation.flush(timeout_seconds=30):
        raise RuntimeError("Could not flush eval results")
    eval_run_id = evaluation.eval_run_id
client.close()

body = json.dumps({"projectId": target["projectId"], "evalRunId": eval_run_id}).encode()
request = Request(endpoint + "/api/v1/query/evals/scorecard", data=body, headers={
    "Content-Type": "application/json", "Authorization": "Bearer " + key,
})
with urlopen(request) as response:
    gate = json.load(response)["gateSummary"]
print("Gate:", gate)
if not gate or gate["status"] in ("failed", "not_evaluated"):
    sys.exit(1)

A failed or unevaluated gate exits with status 1. A warning does not fail the job.

To try it locally, run RUNDIAL_SUITE_ID=<suite-id> python eval_ci.py. It prints Gate: and the gate summary, and exits with status 0 when the gate passes.

2. Run in GitHub Actions

Set repository secrets RUNDIAL_ENDPOINT and RUNDIAL_API_KEY to the API URL reachable from your runner and a project key. Replace <suite-id> with the ID printed by the guide, and commit eval_ci.py to your repository.

name: Eval
on: [pull_request]
jobs:
  eval:
    runs-on: ubuntu-latest
    env:
      RUNDIAL_ENDPOINT: ${{ secrets.RUNDIAL_ENDPOINT }}
      RUNDIAL_API_KEY: ${{ secrets.RUNDIAL_API_KEY }}
      RUNDIAL_SUITE_ID: <suite-id>
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: '3.11'
      - run: pip install rundial
      - run: python eval_ci.py

For baseline comparisons, rundial evals diff --fail-on-regression can fail a job on returned regressions; see CLI.

Next steps

• Run an eval
• CLI