elelem-ggml

Lab. An experiment in how inference works. Built to learn from and read, not published to RubyGems.

Experimental: CPU only, greedy decoding only, no tool calls, and only Qwen2-family models decode correctly. Not published to RubyGems.

elelem-ggml runs a language model on ggml, the tensor library under llama.cpp, without llama.cpp. The transformer graph is written by hand in one C file. The tokenizer and chat template are plain Ruby. Use it to learn what happens between your prompt and the next token. For real work, use elelem-llama.

Install #

The gem is not on RubyGems. Build it from source:

git clone https://git.mokhan.ca/elelem/ggml.git
cd ggml
git submodule update --init --recursive
bundle install
bundle exec rake compile

ggml is vendored as a git submodule at vendor/ggml. rake compile runs ext/elelem/ggml/extconf.rb, which builds ggml with CMake, then compiles ext/elelem/ggml/elelem.c into ext/elelem/ggml/native/libelelem_ggml.so next to ggml’s shared libraries.

Configure #

Point it at a GGUF file:

export ELELEM_GGUF_MODEL=/path/to/qwen2-model.gguf

Or put the file in ~/.agents/models/. If that directory holds more than one .gguf, symlink the one you want to ~/.agents/models/default.gguf. These are the same variable and directory elelem-llama uses.

The context size is the model’s context_length, capped at 4096. The plugin always uses 4 CPU threads. Neither is configurable from the environment.

Use #

From the repository:

ruby -Ilib exe/elelem-ggml ask "hi"

exe/elelem-ggml is the elelem CLI with ggml as the default provider, so chat (the default command) and ask both work. To use it from the main elelem command, put lib on Ruby’s load path:

RUBYLIB=/path/to/ggml/lib elelem -p ggml

The reply appears all at once when generation finishes. There is no token streaming.

How it works #

A reply takes 5 steps. Each one is a short file you can read.

1. Read the GGUF metadata in Ruby. lib/elelem/ggml/gguf.rb parses the GGUF header and key-value section with IO#read and unpack. It skips the tensor data. The tokenizer needs 4 keys from it: tokenizer.ggml.tokens, tokenizer.ggml.merges, tokenizer.ggml.token_type, and tokenizer.ggml.eos_token_id.

2. Format the prompt as ChatML. lib/elelem/ggml/chat_ml.rb wraps each message in <|im_start|>role and <|im_end|>, then ends with <|im_start|>assistant so the model writes the next turn. If there is no system message, it adds Qwen’s default one. elelem always sends its own system prompt, so that default only matters when you call the provider directly.

3. Tokenize with byte-level BPE. ggml has no tokenizer, so lib/elelem/ggml/tokenizer.rb implements one. Strings like <|im_end|> that match a control token (token_type 3) map straight to their ID. Everything else is split with Qwen2’s pre-tokenizer regex, mapped byte-by-byte to printable characters (the GPT-2 byte encoder), and merged pair by pair, always taking the lowest-ranked merge, until no merge applies. The specs pin the exact token IDs against a real Qwen2 vocabulary.

4. Run the transformer in C. el_open in ext/elelem/ggml/elelem.c loads the GGUF with ggml’s gguf.h, reads the hyperparameters under the architecture’s prefix (qwen2.block_count, qwen2.embedding_length, and so on), and allocates the KV cache. Each forward pass builds a fresh ggml graph. For every layer:

• RMSNorm. Scale the hidden state by its root mean square, then by a learned weight. No mean subtraction, no bias.
• Attention projections. Multiply by the Q, K, and V weights. Add the bias if the file has one.
• RoPE, NEOX style. Rotate Q and K by an angle that depends on token position, so attention scores encode relative distance. NEOX pairs dimension i with i + d/2, not with its neighbour.
• Grouped-query attention (GQA). There are fewer K/V heads than query heads. The test model has 14 query heads and 2 K/V heads. The cache only stores the 2, and ggml’s matrix multiply broadcasts each one across 7 query heads.
• KV cache. The new K and V are copied into the cache at the current position. Attention then reads every cached position, with a causal mask of -INFINITY above the diagonal and softmax scaled by 1/sqrt(head_dim).
• Output projection and residual. Project back to the model width and add the layer input.
• SwiGLU feed-forward. RMSNorm, then down(silu(gate(x)) * up(x)), then another residual add.

After the last layer comes a final RMSNorm and the output projection. If the file has no output.weight, it reuses the token embedding (tied weights).

5. Pick tokens and decode. Generation is greedy: C scans the logits for the largest value. The first pass (prefill) runs the whole prompt at once but projects only the last position to logits. Each later pass feeds one token and reuses the cached K and V. It stops at the EOS token or ELELEM_GGUF_MAX_TOKENS. Ruby turns the IDs back into bytes with the inverse byte map and returns UTF-8 text.

Ruby talks to C through Fiddle with 4 functions: el_open, el_version, el_generate, and el_close. Token IDs cross the boundary as packed int32 arrays.

The KV cache is F32, head_dim × n_head_kv × n_ctx per layer, for K and V. It lives as long as the provider, but every reply starts again at position 0 and re-reads the whole conversation.

Limitations #

• Only Qwen2-family models decode correctly. The loader accepts other dense architectures (attention bias is optional, the output layer can be tied) and Llama-family files load without crashing, but RoPE ordering depends on the architecture. Qwen2 needs NEOX, Llama needs NORMAL, and no GGUF key says which. The code hardcodes NEOX, so Llama-family output is wrong. A spec with a 2-layer Llama-architecture model keeps the wrong output as a regression canary.
• Greedy decoding only. No temperature, top-p, or seed.
• No tool calls. The tool list elelem passes in is ignored, so the agent cannot read or edit files with this provider.
• CPU only. n_gpu_layers is accepted and ignored.
• No streaming. The reply arrives in one piece.
• If the prompt plus ELELEM_GGUF_MAX_TOKENS is more than the context size, the call fails with ggml: generation failed. elelem’s system prompt and any AGENTS.md count toward the prompt.
• Every turn re-processes the whole conversation. Long sessions get slower.

Development #

bin/setup
bin/compile
bin/test
bin/lint

bin/setup runs bundle install only; run git submodule update --init --recursive first. The tokenizer and ChatML specs use a vocabulary-only Qwen2 GGUF in spec/fixtures/. Specs that need real weights are skipped unless you set ELELEM_GGML_TEST_MODEL:

ELELEM_GGML_TEST_MODEL=/path/to/qwen2-model.gguf bin/test

Those specs expect a model with 24 layers, n_embd 896, 14 heads, 2 K/V heads, n_ff 4864, and a 151,936-token vocabulary, and a greedy reply to “hi” that starts with “Hello”.

Environment #