Guides

Run elelem fully offline

Run a GGUF model inside the elelem process with elelem-llama, with GPU offload and a context size you choose.

Intermediate 30 minutes, most of it compiling and downloading elelem-llamaelelem-ollamaelelem-builtins

At the end of this guide elelem will run a GGUF model inside its own process through llama.cpp. No model server, no network call. Once the gem is built and the model is on disk, you can unplug the network cable.

Before you start #

  • Ruby 4.0 or newer.
  • A C/C++ toolchain and CMake. elelem-llama compiles llama.cpp when you install it.
  • For GPU offload, the toolkit for your GPU: CUDA (nvcc), Vulkan (glslc and the Vulkan loader), ROCm (hipcc and rocminfo), or a Mac. Without one, you get a CPU build.
  • A GGUF model that fits in your RAM, or your VRAM if you want it all on the GPU.
  • elelem and elelem-builtins. If you have not installed them, see your first session.

1. Install elelem-llama #

gem install elelem-llama

The gem ships llama.cpp’s source. Installing it runs CMake and links a small C++ shim that elelem loads with Fiddle. This takes a few minutes.

The build picks a backend in this order: Metal on macOS, CUDA if nvcc is on your PATH, Vulkan if glslc and a Vulkan loader are present, HIP if hipcc and rocminfo find a GPU, otherwise CPU. If a GPU build fails, it falls back to CPU and says so. To force a backend:

ELELEM_LLAMA_BACKEND=vulkan gem install elelem-llama

Valid values are auto, cpu, vulkan, cuda, hip, and metal. For HIP, ELELEM_HIP_ARCH overrides the detected GPU target (for example gfx1100).

2. Put a model where elelem looks #

elelem looks for a model in this order:

  1. --model /path/to/model.gguf on the command line.
  2. The ELELEM_GGUF_MODEL environment variable.
  3. ~/.agents/models/default.gguf, a file or a symlink.
  4. The only .gguf file in ~/.agents/models/. If there are several and no default.gguf, elelem exits with an error telling you to set ELELEM_GGUF_MODEL or create the symlink.

Download a model into that directory. The elelem-llama repository’s own setup script uses this one:

mkdir -p ~/.agents/models
curl -fL -o ~/.agents/models/Qwen3.8-27B-UD-Q4_K_XL.gguf \
  https://huggingface.co/unsloth/Qwen3.8-27B-GGUF/resolve/main/Qwen3.8-27B-UD-Q4_K_XL.gguf

Any GGUF model with a chat template works the same way. If you keep more than one, point default.gguf at the one you use most:

ln -sf ~/.agents/models/Qwen3.8-27B-UD-Q4_K_XL.gguf ~/.agents/models/default.gguf

3. Start a session #

elelem chat -p llama

elelem-llama also installs an elelem-llama executable that defaults to this provider, so elelem-llama chat is the same thing.

Loading the model takes a moment. While the model generates, its raw output streams in dim italic text. When it finishes, elelem prints the parsed reply and runs any tool calls the model made.

To try another model for one run:

elelem chat -p llama --model ~/.agents/models/other-model.gguf

4. Tune GPU offload #

ELELEM_GGUF_N_GPU_LAYERS sets how many layers llama.cpp puts on the GPU. If you leave it unset, elelem decides:

  • On a CUDA, Vulkan, or Apple Silicon Metal build, it offloads every layer (999) when the model file is smaller than 80% of the GPU’s free memory.
  • Otherwise it uses 0: everything on the CPU.

Set it yourself when the model almost fits, or to split a model between GPU and CPU:

ELELEM_GGUF_N_GPU_LAYERS=30 elelem chat -p llama

Note: The automatic choice does not cover HIP builds. On an AMD GPU with ROCm, set ELELEM_GGUF_N_GPU_LAYERS=999 (or a lower number) or the model runs on the CPU.

5. Set the context size #

ELELEM_GGUF_N_CTX sets the context window in tokens. The default, 0, means the model’s trained context capped at 65,536 tokens. If llama.cpp cannot allocate that much, elelem halves it until it fits, down to 4,096.

ELELEM_GGUF_N_CTX=32768 elelem chat -p llama

A smaller context uses less memory and leaves more room for layers on the GPU. A prompt longer than the context window fails with an error that gives both numbers. Use /clear to start over in a long session.

Two more settings trade quality for memory and speed:

  • ELELEM_KV_CACHE_TYPE sets the KV cache type: q8_0 (default), q4_0, f16, or f32.
  • ELELEM_GGUF_MAX_TOKENS caps the length of each reply. The default, 0, allows the rest of the context.

Sampling uses ELELEM_GGUF_TEMP (default 0.7), ELELEM_GGUF_SEED (default -1, random), ELELEM_TOP_K (40), and ELELEM_TOP_P (0.95). The full list is on the elelem-llama page.

elelem-llama or elelem-ollama #

Both run the model on your machine. The difference is where.

elelem-llama loads the model into the elelem process. Nothing else to run, and nothing listening on a port. The cost is a native build at install time, and the model loads every time you start elelem.

elelem-ollama sends each request over HTTP to an Ollama server (OLLAMA_HOST, default localhost:11434). Ollama downloads models, manages the GPU, and keeps a model loaded between sessions. The cost is a second program to install and keep running.

Use elelem-llama when you want one process and no server. Use elelem-ollama when you already run Ollama or want models to stay warm. Switch between them in a session with /provider llama and /provider ollama.

What you built #

A coding agent with no network dependency. The model runs inside elelem through llama.cpp, on the GPU if you have one, with a context size you chose.

Next steps #