elelem-evals

Alpha. It works, with known gaps. Check the known issues before you rely on it.

elelem-evals answers one question: given this provider and system prompt, does the agent finish real tasks? Each case copies a small project into a temporary directory, sends the agent one or more prompts with the elelem-builtins tools, then checks what changed. The repository ships 30 cases in 15 groups. It is not on RubyGems; run it from a clone.

Install #

git clone https://git.mokhan.ca/elelem/evals.git
cd evals
bin/setup

bin/setup runs bundle install. The Gemfile includes elelem-llama and elelem-ollama. To evaluate another provider, add its gem to the Gemfile first.

Use #

bundle exec exe/elelem-evals -p ollama

PASS build/diagnose-failing-test
FAIL fix/fix-failing-test
  verify failed (exit 1): ruby test.rb
  lib/calc.rb missing "a + b"
ERROR git/commit-50-72: Elelem::Evals::BoundedProvider::TurnLimitExceeded: turn limit 30 exceeded

1/3 passed

Each case prints PASS, FAIL with one line per failed check, or ERROR if it raised. The last line is the pass count. The exit status is 0 only when every case passed, so it works in CI.

Usage: elelem-evals [options]
    -p, --provider NAME              Provider to run against (default: llama)
    -c, --cases GLOB                 Filter cases by group/id glob, e.g. build/*
        --cases-dir DIR              Directory of case yaml files
        --fixtures-dir DIR           Directory of fixture directories
    -P, --prompt FILE                System prompt template file
    -l, --list                       List matching cases without running them

Run one group, list cases, or try a different system prompt:

bundle exec exe/elelem-evals -p ollama -c 'search/*'
bundle exec exe/elelem-evals -l
bundle exec exe/elelem-evals -p ollama -P my-prompt.erb

-P replaces elelem’s system prompt template with the file you give, which is how you compare 2 prompts on the same cases. The -c glob is matched against group/id.

Write a case #

Cases live in cases/*.yml. The file name is the group. Each file is a list of cases:

- id: fix-failing-test
  fixture: broken
  setup:
    - git init -q
  turns:
    - run test.rb and fix whatever is broken
  expect:
    verify: ruby test.rb
    files:
      lib/calc.rb:
        contains: ["a + b"]
        not_contains: ["a - b"]
    response_contains: ["add"]
    response_not_contains: ["sorry"]
    response_matches: ['\A[\s\S]{0,400}\z']
    max_turns: 30

Only id, fixture, and turns are required. A fixture is a plain directory; add one under fixtures/ for a new case. Use --cases-dir and --fixtures-dir to keep your own suite outside the repository.

How it works #

For each case, the runner creates a directory with Dir.mktmpdir, copies the fixture into it, and runs the setup commands with bash -c. It then changes into that directory, builds a fresh session with the provider, and applies every setup block, which adds the elelem-builtins tools and anything from your plugin files. Each turn goes through the normal agent loop.

The provider is wrapped so that each request to the model counts as one turn. Going over max_turns raises, and the case is reported as an ERROR. Checks run after the last turn. The workspace is deleted when the case ends, pass or fail.

Limitations #

• Cases run one at a time, with no time limit beyond max_turns.
• The agent runs with the full builtin tools, including shell commands, and is not sandboxed. The temporary directory is the working directory, not a boundary.
• Plugin files in ~/.agents/plugins/ are loaded too, so your local plugins can change results.
• Every check is a string, regex, or exit status. There is no scoring by a model and no partial credit.
• Results are printed only. Nothing is saved for comparing runs.
• The repository has no specs yet.

Development #

git clone https://git.mokhan.ca/elelem/evals.git
cd evals
bin/setup
bundle exec exe/elelem-evals -l

The gem depends on elelem 0.13 or later and elelem-builtins.

Environment #