Rafmer Edge Node PoC

Rafmer · Domain: Hardware, AI Infrastructure, System Architecture, MLOps, Productization · Skills: CAD, CADQuery, 3D Printing, Python, Bash, Ubuntu, k3s, Kubernetes, Docker, GPU Passthrough, llama.cpp, LLM/AI, Model Benchmarking, Hardware Procurement, Hardware Architecture, Unified Memory, AMD ROCm, Nvidia Jetson, Supply Chain Management, Cost Optimization, Market Analysis, Pricing Strategy, Competitive Analysis, Claude

TLDR

What?

Built a small form-factor AI server with sufficient hardware to run agent platform v1, this includes procuring state-of-the-art hardware from business-oriented vendors, designing and printing a case and benchmarking compute and LLMs.

Why?

The core offerings of early Rafmer were the Agent Platform and the Edge Node. The node was an offering to allow businesses to run the agent platform on-prem. This allowed them full ownership of data, process (through the workflow) and AI-capabilities otherwise gatekept by cloud providers. The Edge Node was a key differentiator between Rafmer and other SaaS incumbents, and our hardware offering was a way to physically anchor our software product.

Due to the requirements of the Agent Platform system, we needed high-end hardware bundled with linux+k3s to match our Agent Platform deployments. The total price point needed to be well below competitive enterprise providers like Dell to target SMBs and Startups, and ideally competitive with local retail providers. For reference Dell charges approx. 200k NOK for an entry level AI Server, and Norwegian retailers charge ~70k for a 4090 gaming build. The ideal build would also be obviously cloud competitive, where we’ve set the average per month compute cost of 7.5k NOK (for g2-standard-12 with an L2 GPU).

The core hardware problem to solve was building a setup where there was enough resources for the Agent Platform services, as well as for frontier-level local models in the below 40B size class. The build also needed to have some level of future-proofing for new architectures, optimizations, size and memory bandwidth demands for possible future models and inference engines.

How?

Based on my experience building Agent Infra, with supporting rolling model-updates, compute-workloads on the same nodes and dealing with UX for slow models, I set the following requirements. The hardware needed to:

I also set the following product requirements for competitiveness:

Memory and Performance

Putting this all together, concretely I needed a hardware stack of 12 CPU-cores with at least 36 GB of memory, running on a high-end consumer GPU to meet the compute and LLM requirements. My first instinct was to straight up build a gaming computer, as it fits most requirements and the software was already validated on this sort of build.

There were 3 obvious issues with this approach:

power consumption:

A gaming build is power hungry, for sustained GPU-loads this will consume up to ~800W, going loud and warm.

formfactor:

Best case case-sizes (pun intended) are ITXs. They require specialized motherboards crammed with the card and PSU into a small case, exacerbating the heating problem. Due to the specialized boards, there are also only two RAM slots, minimizing memory extensions. It’s worth mentioning that ITX cases like this are not very small, and it would still be an eye-sore on someone’s desk.

procurement availability and cost:

Selling to end-users, you’re buying the parts new, which either means full retail cost domestically or discounted internationally with a 25% import tax. For the internal Rafmer server builds, each node is ~17k NOK, and that’s second-hand.

The same build new today costs ~50k NOK (with deflated GPU pricing, commonly available 5090s with 24 GB of memory are listing for 80k in Nov 2026). Alternate builds using AMD or Intel GPUs suffer similar issues, and any cost-difference usually comes at the expense of software support.

Embedded and unified memory

The biggest price driver of GPUs has become the cards’ VRAM. For cards above 16GB, you’re looking at 20k+ NOK (Nov 2026 Nvidia listings, and usually much higher). In order to get around this pricing problem I had to look into alternate memory approaches, and took a page out of the Apple M-series playbook; unified memory. Unified memory involves sharing RAM with both CPU and GPU, this means the memory pool is often of a slower kind than VRAM, but much cheaper than the dedicated cards.

At the time (Summer 2025), the most proven and easily available hardware was the Nvidia Jetson AGX Orin series, supporting up to 64 GB of memory for the entire system. The Jetsons are a series of SoCs (system on a chip), covering memory, CPU and GPU on a single board. These were originally made for industrial edge use-cases and as such are quite power-efficient. The individual chips were priced quite reasonably as well, coming in at ~30k for the full devkit (2025 listing) matching our COGS goal (note that a single devkit is more expensive than just the SoC bought in bulk). It fit the capability and pricing requirements quite well, so I purchased one to validate. In the context of unified memory and Apple, Mac Minis were off the table due to their strict reseller constraints.

An important caveat with the Jetson is that the GPU-architecture is the unusual Tegra, an older laptop-style CUDA GPU as well as the CPU being ARM x64. This meant that most services running on this device had to be built from a specialized nvidia-container for driver support as well as run on an Nvidia Ubuntu fork, all flashed directly onto the card.

For the underlying infra, this is usually fine. K3S is built with embedded (commonly running ARM) in mind, but this is more challenging for the underlying services. Due to the GPU-architecture, most inference engines do not support it well, if at all. Even with CUDA, an already resource-constrained GPU with poor software optimizations will suffer. To add insult to injury, all the underlying services had to be re-built with ARM support (docker images, Python libraries, etc…). This includes every component of Agent Platform and the software dependencies it relies on. The combination of arch-related performance issues and the service complexity of the agent platform v1 led me to abandon this route. For some numbers the Jetson ran Mixtral-8x7B at ~15 tps on llama.cpp, which is bad news for any dense model with 4x more active parameters.

This device is the spiritual predecessor to the Nvidia DGX spark, marketed extensively to the local AI community. With that in mind, Arm support for the underlying services has likely improved by now (Nov 2026).

Ryzen AI Max

Unified memory was still our best lead, but most available SoCs/Boards either ran Arm or something Apple-based. By late 2025, AMD had begun their rollout of the Ryzen AI Max devices and boards started appearing from vendors like AliExpress and SixUnited. These boards supported 128 GB of unified memory, ran x86 CPUs (i.e no Arm software-challenges) with enough compute to run 120B-MoEs above 10 tps. Before committing to full procurement from one of these OEMs, I needed to validate if this was true.

I purchased a 128GB box from a EU retailer, for less than our target COGS, which seemed very promising. Once AMD-specific compute drivers were set up, LLMs ran as expected and connected to the cluster as easily as with a regular computer. To ensure unified memory separation between CPU and GPU, the split had to be configured from the BIOS, effectively setting VRAM to 96 GB and RAM to 32 GB. Procurement of this hardware seemed possible on a per-board basis, meaning we could produce cases with our own branding one at a time.

Procurement challenges, pricing and timing

Getting ahold of an end user device and getting ahold of the boards were two different challenges entirely. At the time, the individual SoC was only listed with 2 manufacturers/sellers (Dec 2025). One required a MOQ of 200 units, at a listing 10% below the retail value of a fully assembled computer. The other provider was likely a middleman between the manufacturer charging 10% over the retail listing for the full device, but had a MOQ of 1. This type of scalper-seller would list the device at competitive prices, then change the price after you’d paid and demand you pay the difference. The serious manufacturer MOQ requirement meant ~5 million NOK of commitments + 1 million in import tax, which was off the table for Rafmer.

Framework, although technically a retailer, had a waitlist for this board and allowed utilizing it with your own branding. A good fit for our prototype. Deliveries didn’t happen until Q1 2026, and by that time hardware and memory-prices had risen drastically. End user devices had nearly doubled in price for the 128 GB configs, but Framework now offered a 64GB version. This version would still meet the performance requirements below our target COGS, but it would be at the expense of total memory. Hardware, especially memory-prices are still rising, and board availability is still a challenge, with Framework being the only possible retailer, constantly selling out.

Building the full node

With a board in hand, assembling it to a functioning computer was the easy part. I put together a modular prototype case using Claude Sonnet/Opus (for spatial reasoning and math) and CADQuery (for python-based CAD to version control and iterate). The screw configurations and bed-measurements for the board came from Framework’s open-source case documentation. The final case is printed as 6 parts, using PETG (for heat resistance) on a Creality K2 printer.

Once assembled, the device is set up with Ubuntu, validated with plain llama.cpp and joined into the cluster. It can also run a cluster-instance independently (although that needs a separate domain setup). Since it’s AMD, the GPU device can be mounted directly into underlying containers, and the embed / inference services don’t need the single-service workaround from Agent Infra. The design has a dedicated 400W PSU, and is kept cool with a single CPU-fan.

Results

The full system

The final result was this:

The final node runs frontier models fully locally at >10 tps. For Qwen3.8-27B at q4, it’s capable of a sustained 18 tps for a 12k token generation run whilst keeping CPU-compute free. With the model running, RAM has more than enough headroom for CPU and GPU loads, whilst leaving 16 x86 cores, allowing concurrent and flexible deployments. The final device is smaller than the size requirement and fits nicely on a desk or a rack. The total material COGS of this single build was 30 936 NOK (excluding labor cost, founders work for free), slightly above target.

Device Caveats

An issue I kept running into whilst designing the edge node and agent infra is the support discrepancy between weird hardware like Tegra or the early days of the Ryzen AI Max and “standard” hardware like the 3090s or cloud GPUs. Even projects like llama.cpp (built to handle any compute-backend) don’t support Tegra-optimized CUDA without specific forks or patches. For AMD, vLLM did not support the ROCm stack until recently. Even then, the ROCm stack for AMD is less mature than CUDA and only compatible with recent GPU-generations. In other words, software support for weird hardware, especially accounting for model optimizations and the underlying engines, is far from guaranteed if you rely on open-source projects (like Rafmer does).

Despite there being sufficient memory and GPU-compute, the entirety of the memory-bandwidth is consumed when generations are running. This can have consequences for other workloads running on the device, especially memory-intensive data-workloads for airflow.

Supply-chain problems and hardware roadmaps

For Rafmer, our hardware requirements meet first-generation “mid-end” AI compute (16 Cores, 64GB, etc.). This is the sweet spot where we can justify the cost of a single device against its capability. Hardware manufacturer roadmaps (AMD, Nvidia) are not at all aligned with this range of compute, because for them mid-end consumer AI is low margin. The current roadmaps for desk-sized AI SoCs reflect a 50% increase in memory-capacity (128 GB vs. 192 GB for AMD), with minimal increase in actual memory bandwidth (7%, AMD Ryzen AI Max 395 vs 495); this is the playbook for Nvidia as well. By adding more memory, you can justify a much higher price point, leaning high end, despite not solving the primary bottleneck (bandwidth). For reference, the next-gen Ryzen AI Max from Framework costs ~90k NOK, before import taxes.

Another key learning from building the agent infra and the edge node: reasonably priced AI-hardware doesn’t stay there. That means that when capable mid-tier devices are the only option left to the consumer, they will be harder to get ahold of and more expensive. I experienced this first hand when attempting to procure the initial board and balancing the target COGS. To survive long term, a product like the Edge Node would have to either push its price point out of reach for its target market or subsidize the hardware to incentivize software distribution for recurring revenue. A software licencing company is not the mission, and the world doesn’t need another enterprise server reseller.

Competing products and Total Consumer Cost

A trend that started this year (2026) is that OEMs are testing the market with mini-PCs utilizing a similar architecture to ours. Some also combine the SoC with mobile-GPUs, allowing for 2x the AI compute and outfit the devices with either uRAM or vRAM / RAM combos. Devices like Olares One (which also comes with kubernetes), DGX Spark, and other Ryzen AI Max variants have commoditized the hardware moat and become part of the standard B2B product offering. These products come from well-funded incumbents, who have much better manufacturing capability and list their products >50K NOK. Non-cloud hardware margins are already incredibly squeezed for these OEMs, pretty much guaranteeing our COGS would be pushed way beyond 30k once accounting for labor cost, current price hikes, availability and supplier roadmaps. On top of that, those prices are based on US listings; in the EEA / Norway you can expect this to be 25% higher. If it comes to a race to the bottom, which is near guaranteed for commodity hardware, Rafmer will lose.

What the nodes are used for today

Based on the Capex required, market need, risk and complexity, we’ve made the decision to not build a business that relies on a severely strained hardware supply-chain in a red-ocean market. It does not mean that productizing the PoC is impossible, it just doesn’t make sense in the near-to-medium term future.

With current OSS Agent offerings, our target market can simply buy a Mac mini and run a similar stack. Which is close to how we’re using the nodes today. They sit as independent servers used to quickly test new models and harnesses whose runtime requirements go beyond that of our Agent Infra. Some examples: