Ollama on SwissGPU

Introduction

Ollama is one of the simplest ways to run local language models on a GPU server. It installs quickly, exposes a small HTTP API, and makes it easy to pull and test models directly from the terminal.

On a SwissGPU machine, the usual flow is to install Ollama, start the service, pull a model, confirm that it loads on the GPU, and access the API through an SSH tunnel.

What You Need

• An active SwissGPU Linux server with an NVIDIA GPU.
• Working SSH access.
• Enough VRAM for the model you plan to run.
• IPv6 connectivity from your local network; see Network Specs.

Step 1: Verify GPU Availability

After logging in by SSH, check that the GPU is visible:

nvidia-smi

Step 2: Install Ollama

curl -fsSL https://ollama.com/install.sh | sh

The official Linux installer sets up the Ollama binary and systemd service.

Step 3: Start the Service

sudo systemctl start ollama
sudo systemctl status ollama

Once the service is active, the local API is available on port 11434.

Step 4: Pull and Run a Model

Download a model first. This example uses gpt-oss:20b; choose a smaller model if needed for your available memory:

ollama pull gpt-oss:20b

Then launch it interactively:

ollama run gpt-oss:20b

For a quick first test, ask a short prompt and confirm that text is generated without errors.

Step 5: Confirm the Model Is Using the GPU

ollama ps

The processor column shows whether the active model is running on GPU, CPU, or split across both.

Step 6: Test the HTTP API

Ollama exposes a simple local API by default. Test it directly on the server with:

curl http://localhost:11434/api/generate -d '{
  "model": "gpt-oss:20b",
  "prompt": "Write a one-sentence hello from Ollama."
}'

Step 7: Access the API from Your Computer

Keep Ollama's default loopback binding. On your local computer, run:

ssh -N -L 11434:127.0.0.1:11434 ubuntu@<server-host>

Replace the SSH destination with the one from your dashboard. While the tunnel is open, usehttp://127.0.0.1:11434 on your local computer. For direct IPv6 access, configure an IPv6 listener and authenticated access; see the Ollama FAQ.

Common Commands

• ollama list shows downloaded models.
• ollama ps shows loaded models and where they run.
• ollama pull <model> downloads a new model.
• ollama run <model> starts an interactive session.

Troubleshooting

journalctl -e -u ollama

• Model loads on CPU only: confirm GPU drivers with nvidia-smi and choose a model that fits in available VRAM.
• API not reachable: verify the service is running, the SSH tunnel is open, and your local network has IPv6 connectivity.
• Pull fails: verify outbound Internet access from the server.