Jump-start a local model in the llama.cpp web UI

Tell this page about your computer and your model, and it writes the command. A web page can't start programs, so you paste it into a terminal yourself. Nothing leaves your browser.

How do I start a local llama.cpp server?

Four Linux commands. Replace the model path, then paste into a terminal. For macOS or Windows, run python3 src/jc.py from the GitHub repo.

All on GPU

Every layer on the GPU, with the largest context that fits. Usually the fastest.

llama-server \
  -m /path/to/your-model.gguf \
  --n-gpu-layers all \
  --fit on \
  --port 8080

Long memory

Asks for a 32K context. On a small GPU, some layers may move to the CPU.

llama-server \
  -m /path/to/your-model.gguf \
  --ctx-size 32768 \
  --port 8080

CPU only

Runs without the GPU. Slower, but works on any machine with enough RAM.

llama-server \
  -m /path/to/your-model.gguf \
  --n-gpu-layers 0 \
  --port 8080

Read-only tools

The model can search and read files your account can read, including private ones. It can't change anything. Experimental.

Tools are on. They're experimental and can read what your account can read.

llama-server \
  -m /path/to/your-model.gguf \
  --tools read_file,file_glob_search,grep_search,get_info \
  --port 8080

Your computer

System

Leave blank if it's already on your PATH.

Your model

Where it is

Downloads on the first run, then stays cached

Starting point

Fine-tune Context window, GPU, thinking, port and whether it fits

1,000 tokens is about 750 words. Bigger uses more memory.

GPU layers
Auto-sizing

Adjusts what you left unset so it fits. With it off, too large a context can stop the server at startup.

Thinking

How varied the wording gets. Lower is more predictable.

Who can connect

Cleared on reload. The key still appears in your command. Prefer to keep it out of shell history.

In the background
Will it fit?