Compare models with elelem-evals

At the end of this guide you will have run a suite of real coding tasks against two models and compared the scores, and written a case of your own. elelem-evals runs each task in a scratch copy of a small project, lets the agent work, then checks the result with shell commands and text matches. It reports pass or fail per case and nothing else: no timings, no token counts.

Experimental: elelem-evals 0.1.0 is not published on RubyGems. You run it from source.

Before you start #

• Ruby 4.0 or newer, Bundler, and git.
• A C/C++ toolchain and CMake. The repository’s Gemfile includes elelem-llama, which compiles llama.cpp during bundle install.
• At least one working provider: an Ollama server with two models pulled, or GGUF files for elelem-llama. See Run elelem fully offline.

1. Get the source #

git clone https://git.mokhan.ca/elelem/evals.git
cd evals
bin/setup

bin/setup runs bundle install. The bundle includes elelem, elelem-builtins, elelem-llama, and elelem-ollama.

2. Fix the provider bug in 0.1.0 #

In 0.1.0, every case fails with undefined method 'fetch' for an instance of Proc in the log, before the model is called. The runner never builds the provider. Open lib/elelem/evals/runner.rb and change one line in the private provider method:

      def provider
        @built ||= @provider.call
      end

The original line is @provider ||= @provider.call, which returns the unbuilt factory. Skip this step if your checkout already has the fix.

3. See what is in the suite #

bundle exec exe/elelem-evals --list

build/diagnose-failing-test
create/fizzbuzz-from-scratch
edit/read-before-write
edit/restructure-with-write
edit/multi-line-edit
...

There are 30 cases in 15 groups, one group per file in cases/: editing, fixing a failing test, refactoring, git, SQL, shell, search, and others. The holdout group holds paraphrased twins of other cases, kept apart so a prompt tuned on the rest can be checked for overfitting.

Each case names a fixture, a small project in fixtures/. The runner copies it to a temporary directory, runs the case’s setup commands there, sends the prompts, and checks the result. The directory is deleted afterwards.

4. Run one group #

bundle exec exe/elelem-evals -p ollama -c 'git/*'

-p picks the provider. Without it, elelem-evals uses ELELEM_PROVIDER or, if that is unset, llama. -c filters cases with a shell glob on group/id.

The output looks like this:

PASS git/commit-50-72
FAIL git/last-commit-subject
  response missing "Add greeting script"

1/2 passed

Each failing case lists the checks that failed. The model’s thinking text is printed to stdout as well, so a PASS or FAIL may follow it on the same line. The last line is always the summary. The exit status is 0 only when every case passes.

5. Compare two models #

elelem-evals takes no --model flag. Each provider reads its model from its own environment variable, so set that per run:

for model in gpt-oss:latest qwen3:latest; do
  echo "== $model"
  OLLAMA_MODEL=$model bundle exec exe/elelem-evals -p ollama | tail -1
done

== gpt-oss:latest
...
== qwen3:latest
...

For elelem-llama, use ELELEM_GGUF_MODEL:

ELELEM_GGUF_MODEL=~/.agents/models/a.gguf bundle exec exe/elelem-evals -p llama | tee a.log
ELELEM_GGUF_MODEL=~/.agents/models/b.gguf bundle exec exe/elelem-evals -p llama | tee b.log
grep -oE '(PASS|FAIL) [a-z]+/[a-z0-9-]+' a.log > a.txt
grep -oE '(PASS|FAIL) [a-z]+/[a-z0-9-]+' b.log > b.txt
diff a.txt b.txt

The diff shows the cases where the two models disagree, which tells you more than the totals do. The provider is built once per run, so a GGUF model loads once, not once per case.

Small suites are noisy. Run each model more than once before trusting a difference of one or two cases.

6. Compare providers #

To run against a hosted model, add its provider gem to the bundle:

bundle add elelem-anthropic
ANTHROPIC_API_KEY=... bundle exec exe/elelem-evals -p anthropic | tail -1

elelem-openai works the same way with OPENAI_API_KEY, and ANTHROPIC_MODEL or OPENAI_MODEL pick the model.

7. Compare system prompts #

-P replaces elelem’s system prompt template for the run. It is an ERB template, like elelem’s default:

cat > terse.erb <<'EOF'
You are a terminal coding agent. Answer in as few words as the task allows.

<%= agents_md %>
EOF
bundle exec exe/elelem-evals -p ollama -P terse.erb | tail -1

<%= agents_md %> inserts ~/.agents/AGENTS.md and the nearest AGENTS.md, as in a normal session.

8. Write your own case #

Keep your cases in a directory of your own:

mkdir -p my-evals/cases my-evals/fixtures/sums
printf '3\n4\n5\n' > my-evals/fixtures/sums/numbers.txt

Create my-evals/cases/math.yml:

# Answer from a file without editing anything.
- id: sum-numbers
  fixture: sums
  turns:
    - what is the sum of the numbers in numbers.txt? reply with the number only.
  expect:
    response_contains: ["12"]
    response_matches: ['\A\D*12\D*\z']

# Write a file with the result, in at most 10 model calls.
- id: write-total
  fixture: sums
  turns:
    - write the sum of the numbers in numbers.txt to total.txt, nothing else
  expect:
    verify: test "$(cat total.txt)" = 12
    files:
      numbers.txt:
        contains: ["3", "4", "5"]
    max_turns: 10

The file name is the group (math). Each case has:

And expect can hold:

A case that goes over max_turns ends with the reply Error: turn limit N exceeded and is judged on what it left behind.

Run it:

bundle exec exe/elelem-evals -p ollama --cases-dir my-evals/cases --fixtures-dir my-evals/fixtures

--cases-dir replaces the built-in cases, and --fixtures-dir the built-in fixtures, so pass both when your cases use your own fixtures.

Things that affect results #

• The agent gets the tools from elelem-builtins plus every plugin in ~/.agents/plugins/ and .agents/plugins/ in the directory you run from. Your personal plugins change the scores. Move them aside for a clean comparison.
• Setup commands and the agent’s own commands run outside Bundler’s environment, so ruby and git in a fixture behave as they would in a fresh shell.
• Sampling is random. For elelem-llama, a fixed ELELEM_GGUF_SEED and a lower ELELEM_GGUF_TEMP reduce the variation between runs.

What you built #

A repeatable way to score providers, models, and system prompts on the same coding tasks, and a directory of cases for the work you care about.

Next steps #

• Write a provider plugin and score it against the published ones.
• Run elelem fully offline to try GGUF models.
• Read the elelem-evals page.