Rafmer LLM Infrastructure

Rafmer · Domain: AI Infrastructure, MLOps, DevOps, System Architecture, Hardware, Observability · Skills: Kubernetes, k3s, Docker, CUDA, vLLM, llama.cpp, Python, Bash, Hardware Procurement, Hardware Architecture, Nvidia GPU, GPU Utilization, API Gateway, OpenAI API, HAProxy, Traefik, NFS, OpenVPN, Rancher, Prometheus, Grafana, Loki, DCGM, pynvml, High Availability, Load Balancing, Network Security, Cost Optimization

TLDR

What?

An in-house multi-node Kubernetes cluster (using the k3s engine) housing highly available LLM / embedding / service deployments, accessible through an authenticated OpenAI-conformant API gateway. The cluster housed 8 3090s across 6 GPU nodes and 2 compute / control-plane nodes. Control of the cluster was achieved through remote kubectl and Rancher. Observability was achieved through Loki, Prometheus and the NVIDIA DCGM exporter on Grafana. The cluster was built from the ground up, hardware through software, on consumer desktops running Ubuntu and accessible remotely with OpenVPN.

Why?

Validation of Rafmer thesis and Product PoC

The goal with Rafmer was to build a deployment-agnostic AI platform that, over time, could be spun up on site and run the same data / LLM offering any managed service could. The initial target was a software stack competitive with offerings like Astronomer, Zapier and of course ChatGPT. The hardware goal was to build something horizontally scalable (with multi-node capacity as an upsell) and compute intensive enough to run the software offering on one or more nodes. Translated into sysadmin terms:

Building the on-prem cluster from the ground up achieved several things: Learning the actual tech to bring about the Rafmer product goal, and building out demo infrastructure for later customer engagements.

Low Cost Infra

On top of being a plain product demo, it doubled as the hosted infrastructure for the SaaS version of the Agent Platform. This way, I could work on both the software and hardware angles while capitalizing on the on-prem capabilities. The initial build-out had ~100 MAUs in mind, but later optimizations and model releases made it capable of hosting closer to ~1000, through estimated improvements to peak concurrency.

How it was used later

As Rafmer’s products and business evolved, the customer infra requirements decreased drastically. The cluster was later repurposed for internal work. Among the workloads were:

How?

Building the Hardware

The initial step was to settle on what kind of hardware to use. After estimating cost, reviewing model requirements and prowling socials for builds and performance, I settled on the RTX3090. The 3090 had 24 GB of VRAM and was the best in class GPU at its release time and popular amongst gamers. This meant they were relatively easy to get ahold of, and sold at competitive prices on the second hand market. Pretty early on in the process, I found that buying entire gaming rigs with the cards was very cost efficient, costing nearly the same as a single card itself. Two of the rigs had sufficient space for two cards and larger power supplies, and they became the dual GPU nodes.

I had settled on the 20B - 40B model range, which fit snugly on a single card and meant I could free both RAM and CPU for regular compute workloads. A happy accident of the gaming rig approach was that most of them had 32 Gigs of RAM, more than enough to handle some load + model deployment (moving model from RAM to VRAM) at the same time. The dual GPU nodes needed new RAM to support larger deployments and new power supplies.

The cluster build-out happened gradually over several months and on each new device the computer was set up with the following:

The k3s agent was then connected to the control plane, a dedicated node that housed the VPN and served as the cluster entrypoint. The nodes ran on a private network, only remotely accessible through the VPN (managed with certs and appropriate profiles) and with specific auth for cluster management. The whole process of adding a new node to the cluster took roughly 1.5 hours, mostly spent waiting for Ubuntu to install.

Running k3s

k3s was chosen as the Kubernetes engine simply for cross compatibility with other engines, simplicity of install and an opinionated stack. I preferred the opinionated stack so I could have routing, container repos and distributed storage working out of the box. It’s no secret that kubectl is painful to use at times, and over time I mostly replaced it with Rancher, the native k3s cluster management interface.

The first infra to be deployed was the inference engine llm service, allocating each discrete GPU using the NVIDIA k8s-device-plugin. With the plain Kubernetes approach, the resulting deployment yaml, containers and code could be deployed on any other future infra, cloud or otherwise.

Building useful Inference

The first requirement to deal with was maximizing GPU utilization, with support for both plain LLMs and embeddings to support Chat and RAG within the Agent Platform. Generally, embeddings were exposed by the underlying inference engine, but swapping models periodically also meant swapping the embedding model. From a database POV, this was the equivalent of swapping out your index every time the server deployment changed. In other words, I needed to separate the embedding serving from the model serving. It was out of the question to commit a full 24 GB GPU to a 1 GB model, which would also make the embedding deployment non-redundant. Running the embed on CPU would be orders of magnitude slower than GPU and contend with data-workloads, and adding dedicated nodes with smaller cards would increase the bare-metal maintenance surface for a single software feature.

Normally the solution here would be to utilize GPU-virtualization to separate the deploys, but with this being a consumer card it didn’t come with that. The solution that added minimal complexity was to run 2 instances of the inference engine and route internally within the deployment (as opposed to just running the bare inference engine). This allowed me to bake both vLLM and llama.cpp inference engines into the single deploy, letting me pick and choose the right config and backend for whatever model was being run through deployment envs.

The engine approach turned out to be a stroke of genius in the following year: with an arms race between open-model providers, the vLLM and llama.cpp projects resorted to pure patches and hotfixes as opposed to long-term releases to support the plethora of models, architectures and configs. From the Rafmer POV this meant I could download a model the day it released, build the inference engines from source in a Docker image and A/B test the performance of vLLM vs. llama.cpp and deploy the day a new model was released.

High Availability and Concurrency

I exposed the k3s deployments using a service and ingress-rule. This ingress was initially managed by Traefik, the built-in proxy router of k3s. Traefik used round robin to distribute traffic across the underlying deployments, but for long running, compute intensive LLM-calls this quickly became an issue.

When a request was routed to an inference engine, it could usually handle the requests it was configured to process in parallel; anything above that and the request was either dropped or returned an error. When using round robin for long compute intensive requests, even with retries and at minimal GPU utilization, requests will drop by the time they have reached the inference engine. The deployment could be locked down at max concurrency using semaphores that returned errors, but Traefik would still round robin the retry.

My solution to this was to bake pynvml into the deployment and use the actual GPU utilization to inform about the busyness of the instance. At a certain utilization threshold, the deployment returned a 503 signaling the proxy to try somewhere else. The round robin problem and request queueing was still not solved by this, so another proxy was added between the client and llm-instance. This proxy was a simple HAProxy instance configured to match the parallelism of the underlying engine, re-route on 503 and queue everything else. With this approach, I was able to achieve maximum concurrency without dropping any requests.

Observability

As inference became more mature, problems became more ephemeral and hard to investigate across the entire stack. Moving from kubectl to Rancher helped debug the basic deployments and PVC issues, but still had some clear gaps.

The key recurrent issues were:

The solution to this was to plug NVIDIA DCGM into the cluster and hook it up to Rancher’s built-in Prometheus, with which I built dashboards to monitor temp, heat, memory and util. Based on this I was finally able to see the bigger picture and discovered the drops were caused by power surges. The solution here was to add power limiting to each node as a periodic system service, outside the cluster.

Combined with the Loki plugin for Grafana, I was finally able to preserve logs, whack recurrent issues and debug clearly. With this in place, each new model deploy was stable, and any issues were observable and dealt with fast. Using these standard cluster observability tools, I was able to create agent-skills dedicated specifically to this cluster debugging.

Bringing it all together

Every open model and inference engine has different parameters, prompt templates and thinking-specific parameters rarely supported by frontend clients without extensive config. A common thread amongst frontend clients is that they always support the OpenAI-API as specified by OpenAI. Modern inference engines also support the rough specification, but often have per-model or architecture specific parameters that have to be included per request. On top of this, many frontend clients will fail hard if API-keys or model names did not conform to the ones provided by OpenAI.

In order to bridge the gap between clients and engine, I built an API Gateway that translated the standard client requests into the engine-specific per-model version of said requests. On top of that it also handled context-based routing to the dual GPU nodes, and authentication with dedicated user-scoped API-keys (part of a separate system). This way, the infra could connect to any agent, framework or frontend supported by OpenAI.

Final Architecture:

The final llm-infra was three layers:

The API Gateway handled Auth, model name to parameter translation and deployment routing. The high availability proxy handled the request queue and forwarding to non-busy deploys.

The LLM Deployment handled GPU mem / util monitoring to report to the ha-proxy server. The deploy image had the necessary cuda libraries, vLLM and llama.cpp built from source and read engine parameters as injected deployment env variables. A second inference instance dedicated to a standardized embedding model ran alongside the llm engine routed within the deploy with plain python code. Each engine was managed with asyncio piping to stdout (to be captured by Loki) and a single engine crash was registered as a deployment failure for self healing.

The LLM model was read from a mounted PVC connected to a central NFS server on the control plane node. This meant the initial model deploy for new models took a very long time, waiting for the model to propagate. However, the simplicity of just downloading the model to a single dir on release made up for it.

Results

This was easily the most challenging technical undertaking of my career. The project spanned multiple years and had to adapt to several external factors, like model releases, consumer limitations and business requirements. This was a living project, maintained and improved over time. Across that timespan the following was achieved:

Integrations and Security

Deployment Flexibility

Utilization and Scheduling

Cost Achievements

As a mostly bootstrapped and cash-strapped start-up, the cluster build-out saved Rafmer a lot of capital. Running a similar stack in the cloud, with the same capability (distributed compute, storage and llm), would have cost close to 3000 dollars monthly (numbers from Sept 2025). The final infra cost approximately 10000 dollars in CapEx and roughly 300 dollars in recurring power cost (being in Norway helps, but this util and price dependent). The hardware cost justified itself at less than 4 months against the estimated cloud equivalent (Sept 2025 estimate), not revenue, and with late 2025 price hikes the impairment loss was minimal.

In terms of model capability, the consumer hardware combined with the internal deployment routing approach enabled Rafmer to squeeze out enterprise capability, throughput and concurrency from the whole cluster. An enterprise system running with similar capability would set you back between 30k - 54k dollars (comparing dual NVIDIA A100 and H100, and just the cards).

Model Performance and Inference Value

Rough performance numbers across Mixture of Experts and Dense llm deployments.

Model Node type Concurrent requests Context length Tokens/s per deployment Measurement month
Qwen3.5-35B-A3B q4 (MoE, llama.cpp) Not stated 4 300k 130 May 2026
Qwen3.8-27B q4 (Dense, vLLM) 2 dual GPU nodes 8 500k 80 Aug 2026
Qwen3.8-27B q4 (Dense, vLLM) 4 single GPU nodes 4 200k 40 Aug 2026

The dense 500k-context figure was measured at peak concurrency across the two dual-GPU nodes, and the MoE deployment and the 200k-context dense deployment each held a separate embedding model as well.

The MoE setup netted 2.7 Billion tokens across the cluster monthly. Basis: estimated at around-the-clock inference saturation of the measured throughput. OpenRouter value of the monthly output token throughput was roughly 2000 dollars. Calculated using llama.cpp priors, i.e. tps drops per client as concurrency increases.

The dense deployments netted 415 million + 415 million tokens per deployment mode, for a total of 830 million per month across the cluster. Basis: estimated at around-the-clock inference saturation of the measured throughput. OpenRouter value of the monthly throughput was 1800 dollars.

OpenRouter prices were taken from the equivalent models’ output tokens on Oct 22, 2026.