elelem-llama2

Lab. An experiment in how inference works. Built to learn from and read, not published to RubyGems.

Experimental: a story generator, not a coding assistant. No chat template, no tool calls, and replies show up as thinking text, not as the agent’s answer. Not published to RubyGems.

elelem-llama2 runs karpathy/llama2.c inside the elelem process. llama2.c is a single C file, run.c, that implements Llama 2 inference with no dependencies. This plugin compiles that file as-is and calls its functions from a small C++ shim. Use it to see the smallest complete inference loop elelem can host, on tiny models like the TinyStories checkpoints.

Install #

The gem is not on RubyGems. Build it from source:

git clone https://git.mokhan.ca/elelem/llama2.git
cd llama2
git submodule update --init --recursive
bundle install
bundle exec rake compile

llama2.c is vendored as a git submodule at vendor/llama2.c. There is no CMake. ext/elelem/llama2/extconf.rb runs cc on run.c and c++ on the shim, and writes ext/elelem/llama2/native/libelelem_llama2.so.

Configure #

You need a checkpoint in llama2.c’s own .bin format, made by vendor/llama2.c/export.py or downloaded. The upstream llama2.c README shows how to download stories15M.bin, a small TinyStories model. The vendored tokenizer.bin is used by default.

export ELELEM_LLAMA2_CHECKPOINT=/path/to/stories15M.bin

ELELEM_LLAMA2_STEPS counts every position, prompt tokens included. Keep it at or below the model’s seq_len: the shim does not clamp it.

Use #

bundle exec exe/elelem-llama2 ask "Once upon a time"

exe/elelem-llama2 is the elelem CLI with llama2 as the default provider. chat (the default command) gives you a REPL.

Tokens stream as they are sampled, but the plugin sends them as thinking events, not saying. In chat you watch the story appear as thinking text, and the recorded reply is empty. ask streams the text, then exits with elelem: no reply.

How it works #

run.c has no header and its own main(). The build compiles it to an object file and links it into the shim. The shim never calls main(); it calls the building blocks directly. Because there is no .h to include, ext/elelem/llama2/elelem.cpp copies the struct layouts and function signatures from run.c. If upstream changes them, the shim has to change too.

Open. el_open calls 3 functions from run.c:

• build_transformer loads the checkpoint and sets up the weights, activations, and KV cache.
• build_tokenizer loads tokenizer.bin.
• build_sampler stores temperature, top-p, and the RNG seed.

Prompt. llama2.c has no chat template. The Ruby provider joins the content of every message with newlines into one string. That includes elelem’s system prompt and any AGENTS.md.

Generate. el_generate is a short loop over run.c’s functions that hands each piece to a callback:

1. encode turns the prompt into token IDs, with a BOS token in front.
2. For each position, forward runs one token through the network and returns logits.
3. While still inside the prompt, the next token is the next prompt token (teacher forcing). After the prompt, sample picks one from the logits with temperature and top-p.
4. decode turns the token into a string, and the shim calls Ruby’s callback with it.
5. It stops at the step limit, or when the model emits BOS (ID 1), which llama2.c uses to separate sequences.

Ruby supplies the callback as a Fiddle::Closure::BlockCaller and calls el_generate with need_gvl: true, so the callback can run Ruby code from inside the C loop. Each piece is forced to UTF-8 and scrubbed before it reaches elelem.

Every call starts again at position 0, so the KV cache is not reused across turns.

Limitations #

• Only llama2.c’s .bin checkpoint format and the original Llama 2 architecture. No GGUF.
• No chat template and no tool calls. The tool list is ignored.
• Replies arrive as thinking events. The agent’s answer is always empty, so ask ends with elelem: no reply.
• The prompt is echoed back. The callback fires for every position, including the prompt tokens being fed in.
• The whole conversation, system prompt included, is one prompt. On a small model with a short seq_len this crowds out the reply.
• ELELEM_LLAMA2_STEPS is not clamped to seq_len.
• No quantized models. Upstream’s runq.c is not wired in.
• The gemspec does not package vendor/llama2.c/tokenizer.bin, so the default tokenizer path only works from a source checkout.

Development #

bin/setup
bin/compile
bin/lint

bin/setup runs bundle install only; run git submodule update --init --recursive first. There is no test suite.

Environment #