elelem-candle

Lab. An experiment in how inference works. Built to learn from and read, not published to RubyGems.

Experimental: this is a skeleton. There is no inference. Model#generate echoes the prompt back word by word. Not published to RubyGems.

elelem-candle is the start of a provider backed by huggingface/candle, a tensor and ML framework written in Rust. What works today is the boundary: Ruby calls a Rust method in the same process, Rust calls a Ruby block once per token, and Rust errors come back as Ruby exceptions. Read it to see how a Rust inference engine plugs into elelem without a subprocess.

Install #

The gem is not on RubyGems. Build it from source:

git clone https://git.mokhan.ca/elelem/candle.git
cd candle
bundle install
bundle exec rake compile

There are no submodules. Cargo fetches candle and the other crates. The build compiles candle-core, candle-nn, candle-transformers, tokenizers, and hf-hub even though the stub does not use them yet, so the first build takes a while.

Configure #

export ELELEM_CANDLE_MODEL=/path/to/model

The provider raises if ELELEM_CANDLE_MODEL is unset. Any non-empty value works, because nothing is read from it yet.

Use #

bundle exec exe/elelem-candle ask "hi"

The reply is the stub’s echo: [stub:MODEL on DEVICE], followed by every message in the conversation as role: content, elelem’s system prompt included. Each word arrives as a separate thinking event, then the whole text arrives once as the answer.

How it works #

There are 3 layers, each short.

Ruby provider (lib/elelem/candle/provider.rb, 33 lines). fetch flattens the messages into role: content lines, builds a Proc that forwards each piece to elelem as a thinking event and appends it to a buffer, and calls Elelem::Candle::Model#generate(prompt, token_cb). When it returns, the buffer goes to elelem as a saying event.

Rust extension (ext/elelem/candle/src/lib.rs, 75 lines). The #[magnus::init] function defines Elelem::Candle::Model with new and generate. #[magnus::wrap] makes a Rust struct a Ruby object, so Model.new(path, device) returns a Ruby object that owns Rust memory. generate takes the prompt as a String and the callback as a magnus Proc, and calls token_cb.call for each piece. If the model path is empty, it raises a RuntimeError in Ruby. The callback is an explicit argument, not a block, because magnus converts positional arguments, not an implicit block.

Build. rb_sys drives Cargo from Ruby’s extension machinery. extconf.rb is one call, create_rust_makefile("elelem/candle/candle"). The Rakefile’s RbSys::ExtensionTask puts the compiled library in lib/elelem/candle/, and lib/elelem/candle.rb loads it with require "elelem/candle/candle". This is the layout bundle gem --ext=rust generates.

Compare elelem-ratatui, which runs a separate Rust binary and talks JSON over stdio. Here Rust is linked into the Ruby process, with typed arguments and no serialization.

The source lists what comes next: load a tokenizer, load safetensors weights onto a candle device, build a model from candle-transformers, and replace the stub with a real forward pass and sampling loop.

Limitations #

• No inference. The output is the prompt echoed back.
• No chat template and no tool calls. The tool list is ignored.
• ELELEM_CANDLE_MODEL and ELELEM_CANDLE_DEVICE are only stored and printed.
• No GPU. The Cargo manifest notes the cuda and metal features are not enabled yet.
• No test suite.

Development #

bin/setup
bin/compile
bin/lint

bin/compile runs bundle exec rake compile. bundle exec rake build compiles first, then builds the gem.

Environment #