elelem-diffusion

Lab. An experiment in how inference works. Built to learn from and read, not published to RubyGems.

Experimental: no README, no published gem, and the specs mock the native library, so they do not exercise a real model.

elelem-diffusion gives the agent 3 tools: image (text to image, or image to image), video (text to PNG frames), and upscale. They run stable-diffusion.cpp inside the elelem process, on your machine. Any provider that supports tool calls can use them.

Install #

The gem is not on RubyGems. Build it from source:

git clone https://git.mokhan.ca/elelem/diffusion.git
cd diffusion
git submodule update --init --recursive
bundle install
bundle exec rake compile

stable-diffusion.cpp is vendored as a git submodule at vendor/stable-diffusion.cpp, with its own ggml submodule inside, hence --recursive. The build writes ext/elelem/diffusion/native/libelelem_diffusion.so (.dylib on macOS).

This is a tool plugin, not a provider, so elelem does not load it on its own. Add a plugin file that puts it on the load path:

# ~/.agents/plugins/diffusion.rb
$LOAD_PATH.unshift(File.expand_path("~/src/diffusion/lib"))
require "elelem/diffusion"

elelem loads every .rb file in ~/.agents/plugins/ and .agents/plugins/ at startup. Adjust the path to where you cloned the repository.

Configure #

Each tool reads its own variables when it is called. A missing variable does not stop elelem from starting. The tool returns an error to the model instead, for example {"ok":false,"error":"ELELEM_SD_IMAGE_MODEL is not set"}.

export ELELEM_SD_IMAGE_MODEL=/path/to/sd-model.gguf
export ELELEM_SD_ESRGAN_MODEL=/path/to/esrgan.pth

If the native library was not built, the plugin prints a warning and adds no tools.

Use #

Start elelem with any provider and ask for an image:

elelem chat

> Draw a lighthouse at dusk and save it to lighthouse.png, then upscale it to lighthouse-4x.png

The model decides when to call the tools. Their parameters:

image with input_path does img2img: it edits the input image, and strength sets how much changes. Width and height then come from the input image. video writes frame-0000.png, frame-0001.png, and so on into output_dir, which it creates. upscale scales by whatever factor the ESRGAN model was trained for. On success every tool returns {"ok":true,"path":...}.

How it works #

Build. ext/elelem/diffusion/extconf.rb configures stable-diffusion.cpp with CMake as a static library, with examples, WebP, and WebM turned off. It then links ext/elelem/diffusion/elelem.cpp together with libstable-diffusion.a and every ggml archive into one shared library. The shim is compiled with hidden visibility and, on Linux, linked with --exclude-libs,ALL, so only the 7 el_sd_* functions are exported. The copy of ggml inside stays private to this library.

Shim. elelem.cpp is 217 lines. Each function fills in one of stable-diffusion.cpp’s parameter structs and calls one library function:

• el_sd_open creates a context from a single model file (model_path).
• el_sd_open_video creates a context from 3 files: the diffusion model, the VAE, and the T5-XXL text encoder.
• el_sd_generate calls generate_image with prompt, size, steps, and seed, and writes the first image as PNG.
• el_sd_img2img reads the input with stb_image as 3-channel RGB, sets it as init_image with strength, and calls generate_image.
• el_sd_video calls generate_video and writes each frame as a PNG. Any audio it returns is freed and dropped.
• el_sd_upscale creates an ESRGAN upscaler, asks it for its scale factor, upscales once, and frees it.

stable-diffusion.cpp’s log and progress callbacks are replaced with no-ops, so it does not write to your terminal.

Ruby. lib/elelem/diffusion.rb binds the functions with Fiddle and registers the tools in a config.setup(:diffusion) block. Contexts are cached for the life of the process: one per image model path, one per set of video model files. The ESRGAN upscaler is not cached. Every native call runs under a single mutex, so only one generation runs at a time. The generate calls set need_gvl: false, so Ruby releases its global lock while C works and other Ruby threads keep running. Thread count is Etc.nprocessors.

Limitations #

• The seed is always 42 and the negative prompt is always empty. The same prompt and settings give the same image.
• video always renders at 512 by 512. Width and height are not exposed.
• Output is always PNG, whatever extension path has. Video comes out as frames, not a video file.
• Model contexts are never freed until elelem exits. An image model and a video model can both stay in memory.
• Thread count is always Etc.nprocessors. There is no setting for it.
• Calls are serialized. A second request waits for the first.
• The tool description calls video “slow: minutes per clip”.
• The specs replace the native library with test doubles. They check the Ruby wiring and defaults, not image output.

Development #

bin/setup
bin/compile
bin/test
bin/lint

bin/setup runs bundle install only; run git submodule update --init --recursive first. bin/test runs RSpec and does not need the native library or any model files.

Environment #