llama.cpp argument

Here is the exact breakdown of what every single argument in your command is doing, from loading the weights to keeping your GTX 970 safe.

---

Core Model & Hardware Configuration

• **`Enable one of the gpu or both

$env:CUDA_VISIBLE_DEVICES="0"

$env:CUDA_VISIBLE_DEVICES="0,1"

• **`check gpu number

nvidia-smi

• ./llama-server.exe
  The main executable. It launches a local HTTP web server (compatible with OpenAI’s API format) so you can connect web UIs, apps, or scripts to your model.
• -m d:\llm_models\Qwen3.5-14B-A3B-Claude-Opus-Reasoning-Distilled-4.6-MXFP4_MOE.gguf
  Model Path: Tells the server exactly where to find your model file. The filename shows it is a 14-Billion parameter Qwen model, quantized (compressed) down to a highly efficient 4-bit format (MXFP4) utilizing a Mixture of Experts (MOE) architecture.
• -ngl 99
  Number of GPU Layers: Tells the engine how many transformer layers to offload to the GPU. Since the model only has 48 layers, setting this to 99 ensures 100% of the model is forced into VRAM, leaving 0 layers on your slower CPU.

---

Context Window & Slot Limits

• -c 32000
  Context Size: Sets the maximum memory buffer for the conversation history (input tokens + output tokens). A size of 32k tokens allows the model to remember incredibly long conversations or read large files.
• -np 1
  Number of Parallel Slots: Limits the server to handling exactly 1 user request at a time. By default, llama-server sets this to 4, which duplicates the context memory budget by 4x and crashes your limited VRAM. Keeping it at 1 saves a massive amount of memory.
• --reasoning off
  Reasoning Mode: Disables any internal thinking loops or extra token generation paths specific to reasoning models. This keeps response times faster and token counts lower.

---

Network & Performance Tuning

• --host 0.0.0.0
  Host Address: Binds the server to all network interfaces on your PC. Instead of only allowing connections from the host PC itself (127.0.0.1), 0.0.0.0 lets other devices on your home network access the AI server via your PC’s IP address.
• --port 8080
  Network Port: Sets the port number the web server listens on. You access it via http://localhost:8080.
• -t 4
  Threads: Dedicates 4 CPU threads to the backend processing. Even though your GPUs are doing the heavy lifting, a few CPU threads are still required to manage data flow and prompt processing.

---

Text Generation (Sampling) Settings

• --temp 0.7
  Temperature: Controls creativity. 0.0 is completely deterministic (always gives the exact same answer). 0.7 provides a balanced mix of creative phrasing while keeping the output logical and accurate.
• --repeat-penalty 1.1
  Repetition Penalty: Prevents the model from getting stuck in an infinite loop or repeating the same sentences. A value of 1.1 subtly discourages the model from reusing recently generated words.
• -n 4096
  Predict/Max Tokens: The absolute maximum number of tokens the model is allowed to generate in a single response. This acts as a hard stop so a prompt never drains your resources by writing forever.

---

Multi-GPU Balancing (The VRAM Shields)

• -sm row
  Split Mode (Row-tensor): Changes how calculations are split between multiple graphics cards. “Row” mode breaks individual computation matrices across the GPUs, which is required when using precise custom tensor split percentages.
• -ts 80,20
  Tensor Split Ratio: The most critical flag for your setup. It tells llama.cpp to distribute the workload disproportionately. It forces 80% of the layer weight onto your first GPU (CUDA0: GTX 1080) and restricts the second GPU (CUDA1: GTX 970) to just 20%, perfectly shielding the 970 from hitting its slow 3.5 GB memory cliff.

The good news is you don’t need to change anything!

llama-server.exe is already an API server by default. The moment it says server is listening on [http://0.0.0.0:8080](http://0.0.0.0:8080), it is fully operational as an OpenAI-compatible API.

Any application, script, or Web UI (like Open WebUI, AnythingLLM, or SillyTavern) can plug into it right now using the following details:

• API Base URL: http://localhost:8080/v1 (or your local IP like [http://192.168.1.](http://192.168.1.)X:8080/v1)
• API Key: Leave it blank (or type anything, like ollama or sk-1234, as it doesn’t require authentication by default).

---

Useful API Arguments (If you want to tweak it)

If you want to customize how the API behaves, you can append these optional flags to your existing command:

1. Add an API Key for Security (--api-key)

If you want to protect your server so other people on your home network can’t abuse your GPUs, add a password:

--api-key "your_secret_password_here"

2. Change the Display Name of the Model (--alias)

By default, your API clients will see the model name as the long, messy GGUF filename. You can clean this up so apps recognize it easily:

--alias "qwen14b-moe"

---

Your Final Command (With Clean API Naming)

If you want to implement the cleaner model name for your API apps, your updated command looks like this:

./llama-server.exe -m d:\llm_models\Qwen3.5-14B-A3B-Claude-Opus-Reasoning-Distilled-4.6-MXFP4_MOE.gguf -ngl 99 -c 32000 -np 1 --reasoning off --host 0.0.0.0 --port 8080 -t 4 --temp 0.7 --repeat-penalty 1.1 -n 4096 -sm row -ts 80,20 --alias "qwen14b-moe"

Quick Test: Verify it’s working

While the server is running, you can open a fresh PowerShell window and run this quick command to verify your API is responding locally:

Invoke-RestMethod -Uri "http://localhost:8080/v1/models" -Method Get

It will spit out a clean JSON response showing your active model, confirming your local API endpoint is up and ready for action.