RundialDocs

Appearance

Run an eval ​

Record scores with trace evidence and inspect a scorecard and gate.

Start Rundial and create an API key with the Quickstart. Keep the stack running and set RUNDIAL_API_KEY.

1. Define and run the eval ​

Save this as eval.py and run python eval.py. It creates a suite and a version with thresholds, then records an eval result. Replace the dataset reference and example scores with your own. Rundial records the scores you supply; it does not run the scorer for you.

python
import json
import os
from urllib.request import Request, urlopen
from uuid import uuid4

import rundial as rd

name = "support-" + uuid4().hex[:8]
dataset = {"name": name, "version": "v1", "artifactRef": name + ":v1"}
suite = rd.eval.suite(
    name, workspace="default-workspace", project="default-project",
    dataset=dataset, scorers=["accuracy", "groundedness"],
)
print("Suite:", name)
print("Suite ID:", suite.suite_id)
scorers = [
    {"name": scorer, "config": {"aggregation": "mean", "objective": "maximize", "unit": "ratio"}}
    for scorer in ("accuracy", "groundedness")
]
policy = {
    "version": "v2", "objective": "Evaluate answer quality",
    "datasetRefs": [{"datasetId": suite.dataset["datasetId"], "version": "v1"}],
    "scorerRefs": scorers,
    "gatePolicy": {"thresholds": [
        {"scorer": "accuracy", "op": ">=", "value": 0.9, "unit": "ratio", "blocking": True},
        {"scorer": "groundedness", "op": ">=", "value": 0.7, "unit": "ratio", "blocking": False},
    ]},
}
url = ("http://127.0.0.1:8787/api/v1/workspaces/default-workspace/"
       f"projects/default-project/evals/suites/{suite.suite_id}/versions")
request = Request(url, data=json.dumps(policy).encode(), headers={
    "Content-Type": "application/json",
    "Authorization": "Bearer " + os.environ["RUNDIAL_API_KEY"],
})
with urlopen(request) as response:
    print("Suite version:", json.load(response)["suiteVersion"]["suiteVersionId"])

with suite.run(config={"model": "candidate"}) as evaluation:
    print("Project ID:", suite.project_id)
    print("Run ID:", evaluation.run.run_id)
    print("Eval run ID:", evaluation.eval_run_id)
    with evaluation.run.trace("example:refund") as span:
        evaluation.log_result("refund", scores={"accuracy": 1.0, "groundedness": 0.8},
                              passed=True, trace=span)

Each result needs trace or artifact evidence. For numeric scores, supply passed for each scorer or one value for all scorers. A failed blocking threshold fails the gate; a failed non-blocking threshold produces a warning.

2. Read the scorecard ​

Open Default Project, open Evals, and select the latest run on the suite card. Open Scorecard to review the scores and gate status.

Next steps ​