Work/Self-hosted models
Open models on one rented GPU
In 2026 I rented one NVIDIA A40 and turned it into a model server I controlled end to end: which weights loaded, how much context they got, when they started and stopped, and how to roll back a bad change.
- Role
- Design, build and operations
- Hardware
- One NVIDIA A40, 46 GB of GPU memory
- Stack
- llama.cpp (CUDA build), vLLM, Qwen models in GGUF and AWQ, Bash, Python, SSH tunnel, Tailscale, Nginx
The problem
A hosted API makes those decisions for you. Running the model yourself means owning the unglamorous parts.
Builds, memory limits, restarts and recovery are where a self-hosted model succeeds or fails, long before prompt quality matters.
What I built
A 27B model on llama.cpp.
Served in GGUF format by llama.cpp, built from source for CUDA. The model listened only on the machine itself, and my own server reached it through a private tunnel.
A 32B coding model kept ready.
Qwen2.5-Coder in AWQ format for vLLM, with scripts to start it, stop it and check its status.
A 7B vision model, trialled as a canary.
Qwen2.5-VL ran beside the coding model as a trial, with one script to run both together and another to roll back to the coding model alone.
The model reasons. Code decides.
On my side, one coordinator written in Python owned the assistant's state, queue, reminders and memory. Order, limits and recovery were never left to the model.
How I know it works
- The vision model was added as a canary, with its rollback script beside it.
- Every change kept a copy of the scripts it replaced.
- In August 2026 the whole machine was audited read-only: every process, port, model file and script, without loading a model or sending one request.
In numbers
- 46GBGPU memory on one NVIDIA A40
- 3models: 27B, 32B coding, 7B vision
My part
I chose the models and built the serving setup, its control scripts and the coordinator.
This page describes the setup as audited in August 2026.