# Sebastian Ikin - Full Markdown > Sebastian Ikin - Senior Software Engineer and AI Founder. GPU inference, infrastructure and agentic systems. The full markdown content of every active page, bundled under the same section headers as llms.txt. Each page is also served separately as pure markdown at its page URL with a .md extension. ## Home HTML version: [https://sebikin.com/](https://sebikin.com/) ## Senior Software Engineer and AI Founder with 6 YOE Senior software engineer with five years in [Arm](https://sebikin.com/cv/arm/)’s GPU organization and co-founder, CTO and CEO since March 2024 of the AI company [Rafmer AS](https://sebikin.com/cv/rafmer/). Primary stack is Python, Linux and Docker within Data Engineering and MLOps Domains. Based in Oslo, bilingual Norwegian/English. If you're interested in my background; check out my CV [here](https://sebikin.com/cv/), [projects](https://sebikin.com/cv/projects/) or sign up for the closed alpha of [Agent Platform](https://rafmer.com/register-interest??source=sebikin.com). For other inquiries, refer to my [email](mailto:sebastianaikin@gmail.com) or [linkedin](https://www.linkedin.com/in/sebastianakin). #### About this site My goal here is to share a bit more of what I've been up to and what's on my mind. This site will be part portfolio and personal blog. I write about various tech, projects I've done, AI and the founder experience here (more posts coming soon). I'm also an avid reader and keep a reading list (books, movies and music). I'm half Australian, hence the english. Are you an LLM or AI Agent? See [llms.txt](https://sebikin.com/llms.txt) for a pure markdown version of this page. #### Disclamer _this is a work in progress and my opinions are my own._ _HTML proudly generated using [Rafmer LLM Infrastructure™®©](https://sebikin.com/cv/projects/rafmer-agent-infra/) through OpenCode, with all plain-text less proudly written by me._ --- © Sebastian Ikin 2026 --- ## CV # CV HTML version: [https://sebikin.com/cv/](https://sebikin.com/cv/) #### Downloads: [CV NOR](https://sebikin.com/assets/CV_Nor_Sebastian_Ikin_Master.pdf) | [CV ENG](https://sebikin.com/assets/CV_English_Sebastian_Ikin_Master.pdf) ## Experience --- ### [**March 2024 \- present:** *CEO / CTO / Co-founder, Rafmer AS*](https://sebikin.com/cv/rafmer/) Co-founder and technical lead in a two-person AI company developing software and infrastructure for local AI inference and agentic workflows. Started as Chair and CTO, becoming full-time CEO in June 2025\. Responsible for technical architecture, product direction and operations. #### **Technology and Product** - Designed and built an internal **Kubernetes-based GPU cluster** from hardware through the application layer for development, testing and operation of local language models. - Built inference deployments on consumer GPUs using **llama.cpp, vLLM, CUDA and Kubernetes**, including five RTX 3090-based deployments supporting eight concurrent clients and approximately 50 tokens/s per deployment with Qwen3.8-27B, as documented in the throughput table on [Rafmer LLM Infrastructure](https://sebikin.com/cv/projects/rafmer-agent-infra/). - Optimized LLM inference with a focus on **GPU throughput, memory usage, context length, parallelism and cost per request**, without enterprise GPU virtualization. - Designed and productized an **edge AI server based on AMD Ryzen AI Max**, Ubuntu, k3s, Vulkan and llama.cpp, including hardware selection, CAD design, prototyping and the software stack. - Developed backend and infrastructure services primarily in **Python and FastAPI**, deployed with Kubernetes, Helm and YAML-based workloads. - Designed and developed **REST APIs, MCP servers, API gateways, model routing and RBAC systems** for internal and customer-facing services. - Built internal tools and agent integrations for **cluster observability, project management, web search and developer workflows**. - Designed and deployed a **GitHub-integrated developer agent** running on the company’s own AI infrastructure, capable of working with internal repositories and development processes. - Designed and built customer-facing **ETL pipelines with Apache Airflow**, including customer-specific access control and processing of external datasets. - Built and delivered customer-facing datasets through **external REST APIs** with isolated projects and access control. - Developed a lightweight **OpenRouter gateway integrated with Supabase and Stripe** for subscription validation, API access and model routing. - Tested and optimized **local AI inference on Linux and Android**, with particular focus on hardware compatibility, throughput, model quality and memory constraints. - Contributed to the architecture and quality assurance of the company’s **Flutter-based Agent Platform**, including the inference runtime, API integrations and local model execution. #### **Leadership and Business** - Responsible for the company’s technical direction, product priorities and project management, primarily through GitHub Projects. - Led a strategic pivot from **B2B sales of AI hardware and licensing to local B2C inference** on consumer devices with a SaaS business model. - Completed and documented a **market validation project funded by Innovation Norway**, running digital marketing and landing customer letter of intent. - Worked with finances, including financial modelling, budgeting and cost management using Fiken and Google Sheets as well as engaging in VCs and other financing-methods. - Negotiated, drafted and coordinated agreements such as **NDAs, letters of intent and customer agreements**. - Responsible for the company’s work on **GDPR, information security and the EU AI Act**. - Managed GPU and AI hardware procurement from Norwegian and international suppliers, focusing on performance per krone. - Worked with market analysis, product pages, technical communication and digital marketing via Google Ads, LinkedIn and Meta. ### [**June 2020 \- May 2025:** *Senior Software Engineer \- GPU, Arm Trondheim*](https://sebikin.com/cv/arm/) Worked in Arm’s GPU organization developing and maintaining Python tools for GPU test automation, GPU bring-up, data processing and analysis. Started part-time during my studies, later moving full-time. Progressed to Senior Software Engineer, responsible for key internal tools, data deliveries and cross-team support. Selected contributions: - Built and maintained an **Apache Spark-based ETL pipeline** for automated issue detection in large test datasets. Reduced manual filtering and saved the team about **eight working hours per week**. - Built a research-based **image verification algorithm and data pipeline** automating a previously manual classification process. Achieved about **70% precision** and cut manual work by about **five hours per week**. - Established cross team standards and reporting channels for image verification issues. - Built and maintained an internal **React/Node application** for data classification/analysis. - Had full ownership of cross-team utilized **Python + IBM LSF tool**, stakeholder critical for data deliveries. Involved managing related Jira projects, adding features, improvements and ensuring uptime. - Improved testing and QA for internal data-distribution tools, helping reduce critical production errors to near zero over roughly three years. - Troubleshot complex issues across **hardware, drivers, test infrastructure and software** during GPU bring-up. - Identified and escalated cross-team test issues and contributed to root-cause analysis and coordination. - Managed and delivered analyses and datasets to four internal teams, technical leads and external partners. - Contributed code reviews, architecture discussions and improvements to internal development processes, inclusive **Jenkins based CI/CD**. - Ran **technical interviews** and helped recruit developers. - Mentored and onboarded new hires and supported less experienced developers. ## [Other Projects](https://sebikin.com/cv/projects/other/) ---- ## [Education](https://sebikin.com/cv/edu/) --- ### **August 2018 – June 2021:** *Bachelor of Engineering \- Computer Engineering, NTNU Trondheim* Specialization in software architecture. Bachelor’s thesis in **penetration testing and cybersecurity**. ## Technical Skills --- **Programming:** Python, TypeScript, SQL, JavaScript, Java, Bash, C++, Dart / Flutter **Backend & Software Engineering:** FastAPI, Node.js, REST APIs, microservices, MCP, RBAC, PostgreSQL, Supabase, serverless, distributed systems, system architecture, OOP, TDD, design patterns, code review **Data and ML:** Apache Spark, Apache Airflow, Pandas, NumPy, PyTorch, ETL / pipelines, data quality, validation, GPU inference **AI Infrastructure / MLOps:** Kubernetes, k3s, Helm, llama.cpp, vLLM, CUDA, ROCm, Vulkan, TensorRT-LLM, Triton Inference Server, Nvidia GPU, AMD Ryzen AI, Arm Mali, local LLM inference, model/GPU/memory optimization, huggingface **DevOps and Observability:** Git, GitHub, Gerrit, GitHub Projects, Jenkins, Rancher, Docker, Prometheus, Grafana, Loki, Linux / Ubuntu **Cloud:** AWS EC2, Lambda, RDS, Aurora, VPC, S3, SAM, GCP Firebase, Firestore, Supabase PostgreSQL, Auth, Edge Functions, Storage, Vault, RLS, Cloudflare Gateway **Leadership & Strategy:** Technical leadership/strategy, product development, project management, architecture, mentoring, onboarding, technical interviews, business development, financial modelling/cost management, market analysis, investor/stakeholder communication, GDPR, information security, EU AI Act, SCRUM, Kanban, Jira, Confluence, GitHub Projects --- © Sebastian Ikin 2026 --- ## Projects # Rafmer Internal Development Agent > Rafmer · Domain: Agentic Engineering, DevOps, Observability · Skills: Agent Design, Agent Orchestration, LLM/AI, Kubernetes, Docker, Python, OOP, Prompt Engineering, openai-agents, OpenCode, GitHub, Git, Project Management, Context Management HTML version: [https://sebikin.com/cv/projects/dev-agent/](https://sebikin.com/cv/projects/dev-agent/) ## TLDR - Built two iterations of an **agent-driven coding system** on the company's local LLM infrastructure, taking GitHub Project tickets from intake to reviewed PRs with minimal human involvement. - Designed a Python **dispatcher/worker pipeline** that sorted tickets into dependency chains, enriched them with repo context and ran an implement–review loop using **OpenCode, subagents and the openai-agents framework**. - Rebuilt the system around a **Hermes profile** with flow-step-specific skills and dedicated OpenCode slots, enabling asynchronous task intake over GitHub, Telegram and Signal with built-in observability and self-auditing. - Ran both versions on the internal **Rafmer GPU cluster**, with Qwen3.5-35B-A3B and Laguna-S 2.1 in v1 (~130 tps @ 300k context) and homogeneous Qwen3.8-27B in v2, with dense-model throughput as documented in the [Rafmer LLM Infrastructure](https://sebikin.com/cv/projects/rafmer-agent-infra/) throughput table. ## What? Built two iterations of an agent-driven coding system powered by local LLM infrastructure: #### Version 1 Pure Python job dispatcher and workflow, interacting with OpenCode exposed tools and repo: - A periodic dispatcher process listening and watching GitHub for ready tickets - The dispatcher sent tickets to the workflow - The workflow ran the dev flow - The resulting PR was pushed to GitHub #### Version 2 Hermes profile with flow-step specific skills and dedicated OpenCode slots: - A GitHub-connected Hermes gateway was the primary orchestrator - The Hermes gateway communicated with 3 dedicated OpenCode slots, dispatching tasks as they came in ## Why? Inside Rafmer, I was doing more and more agent-driven development requiring me to sit in a continuous dialogue with OpenCode, often lasting several hours. Rafmer had also moved to tracking tasks in tickets through several GitHub project Kanban boards. The Rafmer systems comprised several smaller repos, so I worked manually where necessary and context-switched to and from several active OpenCode instances doing vastly different work across different repos. As Rafmer's agent-side DevOps tooling became more mature, with added Prometheus, UI-Debug, and Kubernetes scripts/skills, my role as the human in the loop became more redundant. I was essentially answering mostly redundant questions and guiding OpenCode through the ticket, implement, validate, correct, validate loop. To preserve my own time and sanity, I wanted to at least automate part of this process so I could simply do code reviews remotely and only periodically validate the whole repo, without having to sit at my computer waiting for OpenCode to ask me random questions. #### Important caveat: Rafmer's infra was internal, and the apps were non-critical (for now). The infra (including [LLM and serving](https://sebikin.com/cv/projects/rafmer-agent-infra/)) was strictly Rafmer's own on a completely private network, so the leak surface was minimal. With the infra being Rafmer's own, token usage could also be maximized, which would have been a significant rolling cost with an external provider. Periodic backups of both code and data were taken, and as a two-person startup with minimal capital, getting other devs onboard was not an option. This was an option under Rafmer's conditions and not necessarily the right choice for a larger org with external infra. ## How? #### Version 1 Version 1 was a two-stage Python system: 1. Dispatcher: handled sorting and chain building of tickets stored on GitHub via LLM calls and dispatched them to the worker 2. Worker: ran the implementation loop on the connected repo The dispatcher pulled ready tickets from GitHub projects and sorted them into dependency chains, assigning dependencies to tickets and dumping the sorted state to disk. The dispatcher had a series of workers to which it delegated each chain. The worker picked up the next ticket in the chain. On a new ticket, the worker reset the repo state using basic git actions and checked out whatever dev branch it was responsible for. It then moved on to the dev flow, done in part by a Python library wrapping around OpenCode and subagents using the openai-agents framework: 1. Repo Validation: determine whether anything had to be set up before starting. Yes/Correct/Hard Failure 2. Enrich ticket with repo context: assess whether the ticket could be done with the current repo. Yes/Soft failure (ticket comment), next ticket 3. Plan: break down the enriched ticket into concrete steps 4. Implementation - Review: OpenCode implemented, the agent reviewed, committed and moved to the next step / returned to implement (4 iterations) / soft failure (ticket comment) 5. Review: checked the acceptance criteria, compared the actual code against the plan, pushed PR / global correct (3 iterations) / soft failure + pushed partial PR + ticket comment Architecturally, each step was its own class, subclassed from an abstract step with the per-step state management handled within the class parameters. The flow came together in a top-level caller (i.e. the worker) as a step-by-step flow iterating over each step instance. Each PR ended up in a PR-stack per chain to simplify the review, and was reviewed by me on GitHub. The LLM-stack for this version was Qwen3.5-35B-A3B and Laguna-S 2.1 running on a PoC version of the Rafmer Edge Node. Laguna was used in the planning and enrichment part to compensate for the Qwen MoE's lack of hard intelligence. It was hosted on an experimental node with 128 GB due to its size, with roughly 100k context going at 15 tps. The Qwen was responsible for the implementation and ticket sorting. It ran through the internal Rafmer cluster, primarily chosen for being the best LLM for throughput at the time, doing roughly 130 tps with 300k context per deployment. #### Version 2 Version 2 was a Hermes profile hooked up to a dedicated gateway, with isolated OpenCode slots that could work in parallel on separate tasks in the same repo. Hermes communicated with each OpenCode instance through the sloctl-mcp, and each instance had its own checkout of the repo that was fully reset as needed. The Hermes dispatcher decomposed tasks, either by reading tickets with a cron job, through a connected UI to the gateway, or by the user messaging it from Telegram or Signal. The dispatcher had the previous flow tasks decomposed as specific skills and sent queries directly to OpenCode and validated the results. The usage of skills and the scope of responsibility + heuristics were managed through the profile's SOUL.md. The whole system made use of the Rafmer cluster running only Qwen3.8-27B, on deployments running at the dense-model throughput documented in the [Rafmer LLM Infrastructure](https://sebikin.com/cv/projects/rafmer-agent-infra/) throughput table. If context, model quality and token speeds had been much lower, the Hermes approach would have been much less feasible. Hermes was also built to self-audit and improve, which meant I could interrogate the instance as a means of observability and that its traces were preserved. #### In Common Both designs were made to be GitHub-project, repo, LLM and coding-language agnostic, configurable from a simple .env. Both systems were deployable through Docker Compose and Kubernetes, and runnable on host. ## Results #### Version 1 Version 1 was a good start for problem scoping, but was bogged down by sub-agent overspecialization, large model bottlenecking, MoE output quality and too much rigidity. Some findings: 1. Flow worked very well for pure Python repos 2. Very hard to keep track of end results of prompt orchestration 3. Minimal and flaky observability through the OpenCode Python submodule (std.out being eaten) 4. Whole flow bottlenecked by the single-client larger model 5. Flow broke down for simple tickets like docs, requiring layers of special casing 6. Contradictory system-prompts across subagents and flow-steps 7. Pure code, big maintenance surface, especially for large prompt templates 8. Dumb MoE model requiring tool definitions and example use cases in each system prompt, further widening the maintenance surface 9. Spotty repo-state management by pure git Overall, Version 1 was a good case study in harness vs agent-flow engineering and a good throughput test for Rafmer's agent infra. #### Version 2 Version 2 was a significant improvement over Version 1, primarily due to the choice of harness and model capability: 1. Could communicate asynchronously across multiple channels and continue across them 2. Actual observability through Hermes' built-in capabilities 3. Much larger degree of freedom through Hermes and tools, allowing more dynamic task-solutions and ticket breadth 4. Ability to separate scheduled work and ad-hoc tasks, and multitask at different levels 5. Homogeneous models meant no single one was bottlenecking, both quality and performance-wise 6. Ability to communicate with Hermes about background tasks 7. A lot simpler to maintain, debug and audit 8. Hermes could modify tickets and answer questions about the project as well as the repo Some of the results could be attributed to improved models, but UX, observability and overall flexibility were the winners here. All in all, a much better agent as opposed to the pre-defined workflow Version 1 was. --- © Sebastian Ikin 2026 --- # Market Clarification Innovation Norway > Rafmer · Domain: Productization, MLOps · Skills: Leadership, Financial Modelling, Accounting, Market Validation, Grant Management, Stakeholder Management, Negotiation, LLM/AI, Marketing, Google Ads, SEO, Strategy, Business Development, Project Management Related assets: - [The clarified product line on the business site: Agent Platform, Edge Node and Enterprise offerings.](https://sebikin.com/assets/innovation/og_product_line.png) - [Google Ads paid search performance for the campaign period, including the reported 5.5% conversion rate. Note: reported conversions were inflated due to a bug relating to conversion tags; the real figures are based on actual entries in our lead database.](https://sebikin.com/assets/innovation/conversion_rate.png) - [Geographic targeting of the ad campaigns across the UK, the Nordics and Iceland.](https://sebikin.com/assets/innovation/our_targeted_market.png) - [Agent Platform PoC: Quick Chat working directly on a Python file, with common edits suggested in context.](https://sebikin.com/assets/innovation/agent_poc_1.png) - [Agent Platform PoC: the agent reading the inbox and composing an email analysis draft.](https://sebikin.com/assets/innovation/agent_poc_2.png) - [Agent Platform PoC: a scheduled daily lead update workflow with run status and history.](https://sebikin.com/assets/innovation/agent_poc_3.png) HTML version: [https://sebikin.com/cv/projects/market-clarification-innovation-norway/](https://sebikin.com/cv/projects/market-clarification-innovation-norway/) ## TLDR - Completed a **market validation project funded by Innovation Norway** (Grant 1) on time and under budget, with successful application, final reporting and full grant approval. - Produced the branding, BMC, pitch deck and stakeholder documentation that positioned Rafmer's local AI offering for B2B customers and investors. - Delivered the technical PoC behind the validation, including **Agent Platform v1**, the **Edge Node** and an initial **5-node cluster with multi-tenant Airflow and LLM serving**. - Validated market interest with a signed **Letter of Intent from a pilot business customer**, a **5.5% paid search conversion rate** (vs the 3.8% SaaS benchmark) and **3 VC/PE investor inquiries** after the brand rollout. ## What? Business validation for Rafmer through qualification for Innovation Norway Grant 1. The project covered building out a proof-of-concept local AI infrastructure to serve B2B and validate market interest. This involved getting branding ready, producing the necessary investor/stakeholder documentation, running paid ads across multiple channels, and producing a PoC product that could be marketed to businesses. Rafmer later pivoted from this B2B offering to local B2C inference in February 2026, following model and agent advancements. ## Why? The goal was to secure non-dilutive funding, test Rafmer's thesis, get the demo ready and improve the company's market presence. ## How? Rafmer already had the software partially ready, running on non-scalable hardware; this needed to be modified and finalized so it could be demoed. The initial product line was unfocused, but resulted in Agent Platform v1 and the Edge Node, with the on-prem infra serving as the demo for both. ![Rafmer product line page showing the Agent Platform, Edge Node and Enterprise offerings](https://sebikin.com/assets/innovation/og_product_line.png) The clarified product line on the business site: Agent Platform, Edge Node and Enterprise offerings. My co-founder was responsible for the design and UI/UX alignment of the frontend app across Web, Android, and Linux, as well as productizing copy around the Rafmer Agent Platform in the context of branding and investor relations. He also managed ad copy, wireframes, graphical design, and short-form video ads. My co-founder and I both contributed to pitching, applying, and final reporting to Innovation Norway. My responsibilities were: - Managing the cost, timing, and reporting of the grant project. - Running and adjusting marketing campaigns based on analytics. - Handling accounting, tracking, and auditor feedback. - Negotiating and securing Letters of Intent (LoIs). - Coordinating hardware procurement and gradually spinning up the [cluster](https://sebikin.com/cv/projects/rafmer-agent-infra/). - Finalizing backend software for the demo PoC. - Deploying a business site and managing presence through SEO. ## Results The project was completed successfully, on time, and less expensive than budgeted, producing the following results: #### Innovation Norway - Successful application and final reporting, approved by an auditor. - Created BMC, pitch-deck, and other relevant documentation. - Successful procurement of infrastructure hardware and marketing campaigns below the initial budget. - Project completed successfully and grant received. #### Branding and Product - Specification of a complementary product line. - Full rollout of the brand on web and socials with complementary analytics. - Pilot campaigns across LinkedIn and Meta. #### Market Validation - Letter of Intent signed by a pilot business customer. - Successful Google Ads campaign at a 5.5% conversion rate for paid search versus [3.8% for traditional SaaS](https://unbounce.com/conversion-benchmark-report/saas-conversion-rate/). - Received 3 VC/PE investor inquiries post brand rollout. ![Google Ads campaign overview showing paid search conversions and conversion rate](https://sebikin.com/assets/innovation/conversion_rate.png) Google Ads paid search performance for the campaign period, including the reported 5.5% conversion rate. Note: reported conversions were inflated due to a bug relating to conversion tags; the real figures are based on actual entries in our lead database. ![Map of ad targeting across the United Kingdom, Denmark, Sweden, Norway, Iceland and Finland](https://sebikin.com/assets/innovation/our_targeted_market.png) Geographic targeting of the ad campaigns across the UK, the Nordics and Iceland. #### Technical Results - Successful MVP delivery of the backend infrastructure software for the LoI. - Initial 5-node cluster with multi-tenant Airflow and LLM capabilities spun up. ![Agent Platform Quick Chat interface working on a mail_utils.py file with suggested edits](https://sebikin.com/assets/innovation/agent_poc_1.png) Agent Platform PoC: Quick Chat working directly on a Python file, with common edits suggested in context. ![Agent Platform Quick Chat composing an email analysis from inbox readings](https://sebikin.com/assets/innovation/agent_poc_2.png) Agent Platform PoC: the agent reading the inbox and composing an email analysis draft. ![Daily lead update workflow run in the Agent Platform workflow view](https://sebikin.com/assets/innovation/agent_poc_3.png) Agent Platform PoC: a scheduled daily lead update workflow with run status and history. #### Worth mentioning Rafmer pitched one of the funds, but they wanted more traction since the company was technically a pre-product hardware startup. Paid search was competing with much larger incumbents like OpenAI and Anthropic, battling over bids for phrases like "AI Platform," etc. --- © Sebastian Ikin 2026 --- # Image Classification > ARM · Domain: Machine Learning, Data Engineering, MLOps, GPU Bringup, Software Engineering · Skills: Python, C++, cv2, NumPy, PIL, PyTorch, TensorBoard, React, Jira, Computer Vision, Image Processing, Image Classification, Hyperparameter Tuning, Model Validation, Data Labeling, Dataset Creation, CLI Tooling, CI Integration, Drift Monitoring, Technical Presentations, Documentation, Cross-Team Standardization, Cross-Team Coordination, Stakeholder Management, Knowledge Transfer, Mentoring, Process Formalization, Project Management, Image Processing, Computer Vision HTML version: [https://sebikin.com/cv/projects/image-classification/](https://sebikin.com/cv/projects/image-classification/) ## TLDR - Built an **image difference classification system** to validate GPU driver quality during hardware bringup, replacing slow manual image comparison. - Progressed through **three iterations**, from a C++-ported PoC to a **Python CV pipeline** (cv2, NumPy, PIL), with later work moving toward a **PyTorch-based ML approach**. - Established a **cross-team image quality standard**, reference dataset and labeling process across **four hardware bringup teams**. - Achieved about **70% precision**, cut manual work by about **five hours per week**, and deployed the model into **production test pipelines**, with an internal classification UI and automated drift tracking for continuous improvement. ## What? This system classifies image differences used to validate GPU driver quality within the Hardware Bringup Flow. The final classification system is a CV pipeline written in Python, validated through confusion matrices against a human labeled test set. Later iterations utilized PyTorch. Precision is the share of flagged image differences that were confirmed as real differences. This project was completed at ARM Norway. Some technical details are omitted due to ARM Intellectual Property (IP) restrictions. ## Why? In GPU hardware bringup, it is necessary to evaluate driver quality across various platforms and thousands of workloads. For graphics workloads, the output is often a complex scene, featuring multiple geometries, different lighting, and various effects. The quality of this output needs validation, often by comparing it against a reference image. Human validation of these reference comparisons is possible, but it is time-consuming and expensive due to the sheer volume of tests. Direct pixel comparisons do not work due to precision differences between the driver/GPU, and traditional image noise metrics fail to capture low-pixel, high-contrast image artifacts. In the context of GPU validation, uncaught image generation errors can end up with the end customer when they are running the final tests on their end device. This negatively impacts the company's perception of quality, and in the worst case, can indicate uncaught underlying hardware issues. The project was undertaken to reduce engineering hours spent comparing images, minimize false negatives, and formalize an organizational methodology for defining "good" and "bad" images across bringup teams. Additionally, we needed systems for CI and to ensure the new approach could be plugged into production pipelines. ## How? The development was split over three iterations, with improvements achieved through automated drift tracking, formalized ownership, standardized processes, and standardized procedures for difference reporting. ### Establishing a baseline Across four different hardware bringup teams, each possessed a separate error standard and a unique approach to validation. Before any coding work began, I created a small reference set of image differences where all teams agreed on the labels, established a Jira Epic for reporting new findings, and designated "image experts" on each team. The reference set and the reporting from the image experts were used to test the iteration 1 approach. I served as my team's dedicated image expert and held full responsibility for the verification algorithm and setting the cross-team standard. This was a mostly isolated "skunkworks" project, which few people were willing to take on due to the abstract requirements. The goal was to build a solution viable for the pipelines of all four teams. ### Iteration 1 I rediscovered older pre-ML approaches used by animation studios, involving image kernels and contrast techniques. The existing example implementations were in pure C++, and I ported them to modern Python CV to serve as a Proof-of-Concept (PoC) using reference images. The initial approach showed promise and already outperformed existing pipelines. Prioritizing pre-ML pure Python was crucial to ensure portability into our existing pipeline. ### Data To move into the next step, I needed to establish a much broader reference set. To achieve this, I did the following: - Collected a large set of images from our production pipeline. - Developed a simple React application to quickly browse and label images. - Authored documentation on common errors and the "image expert" standard. - Persuaded my manager to secure team contributions for labeling based on the provided documentation. The resulting data was labeled after about a month. A training set derived from this data was subsequently used by iteration 3. ### Iteration 2 This iteration was based on the findings of the PoC and a key [research paper](https://research.nvidia.com/sites/default/files/node/3260/FLIP_Paper.pdf) from Nvidia. Iteration 2 was a complete rewrite, resulting in a clean image-processing and classification pipeline with fully configurable hyperparameters, tunable against the established reference set. The new approach primarily used image processing techniques combined with ML-adjacent computer vision methods focused on capturing contrast differences. Once tuned, iteration 2 performed at 70% precision, which was good enough to be integrated into our current production pipeline. The tool was also exposed as a CLI, utilized by several other teams, and integrated into another team's production pipeline. The findings and validation results were shared across the organization via an engineering presentation. Both iterations were primarily built using Python, cv2, NumPy, and PIL. ### Continuous Improvement In an environment where data flows continuously, work is never truly finished. The parameters of iteration 2 were adjusted as new human-provided cases were tracked via tickets, analyzed, and then run against the reference set. Due to the architecture of iteration 2, we could add new "filtering" steps for rare edge cases or batches of known errors. Automated data collection was also implemented in the production pipeline to continuously monitor classification drift and export new cases to a data directory for reference set expansion. Classification drift detection was built into the production pipelines, while data collection was a separate system entirely. Collected data was made available for human labeling through the classification UI. ### Iteration 3 During my final days at ARM, I coordinated the team's efforts to transition to a pure ML-based approach validated through TensorBoard and built with PyTorch. This was in conjunction with handing over the workflows and domain knowledge related to the iteration 2 algorithm and establishing future development directions. ## Results The project spanned roughly 1.5 years and was completed alongside my regular deliveries (e.g., workflow maintenance and test-related tasks). - Designed and implemented a classification system achieving approximately 70% precision. - Cut manual work by about **five hours per week**. - Presented implementation and findings to the organization. - Model taken into production. - Replaced faulty older methods in a pre-existing test pipeline with the new algorithm. - Maintained an organization-wide tool to run the algorithm and established feedback channels. - Led a CI project where test errors and user feedback were integrated into concrete algorithm improvements. - Built, ran, and maintained an internal classification UI to establish a baseline test set. - Guided and managed the direction of the novel ML-based approach as the final iteration. --- © Sebastian Ikin 2026 --- # Bug Classification > ARM · Domain: Data Engineering, Software Engineering, GPU Bringup · Skills: Python, PySpark, Apache Spark, Bash, ETL, Map-Reduce, Log Parsing, Data Standards, Schema Design, Heuristic Filtering, Issue Tracking, Test Automation, Performance Optimization, Documentation, Knowledge Transfer, Cross-Team Coordination, Stakeholder Management, Project Management, Process Formalization HTML version: [https://sebikin.com/cv/projects/bug-classification/](https://sebikin.com/cv/projects/bug-classification/) ## What? A bug classification pipeline that pulled logs from datacenter infrastructure and classified failing tasks. The underlying tool was a Python-based CLI, utilizing PySpark to distribute the workload and cross-reference it across known issues and symptoms. The pipeline took raw logs and turned vague errors into structured output, ready to be converted to tickets and assigned to the appropriate stakeholder. Built during my time at Arm, with MVP as an intern, and gradually expanded through the years. Some details are omitted to respect IP. ## Why? As part of my team's role in the GPU Bring Up process, we ran a large volume of per-GPU workflows running many different tests. Each of these tests was its own multi-step workflow, where each step could be owned by us or other external teams. In simple terms, the workflow itself ran build -> run -> validate -> extract for every test the customer or stakeholder required. As you can imagine, this produced GBs of logs and artifacts distributed across several locations. My team was responsible for monitoring said workflows for issues across the entire stack and notifying the appropriate teams. Due to the breadth and scale of the tests needing to be validated, doing it manually would consume the entire teams bandwidth. Some automation was happening through simple but inefficient shell scripts, but this approach bottlenecked the delivery pipeline and took hours. The goal was to structure the chaotic logs and artifacts, and distribute/parallelize the parsing in a Pythonic way using modern data engineering techniques. ## How? I started the project as an intern in 2020 and delivered a crude prototype based on porting the old shell scripts to plain Python. The initial design used the shell-based logic within multi-node Spark workers orchestrated by a central script handling the SparkSession. It quickly became apparent that Arm's LSF-based infrastructure had too much load to timely assign worker nodes, as it was used across the org. Another issue with running it on the infrastructure was that NFS sync timings ate up gains and introduced intermittent errors, due to having to sync the workflow's entire directory tree across all worker-nodes. To solve the issue, I moved the load onto our team desktops, caching only the interesting artifacts and using heuristic filtering of known open issues, sorted by priority and relevant tests, to pre-reduce the compute load before processing anything. When the scan ran, the ticket-relevant spark tasks targeted only the file and symptom described by the ticket. Once the ticket targeting was done, another task queue of ambivalent failures was scanned across the entire issue index. This approach minimized the really expensive computations to a few unaccounted tests at the end. To enable this, I developed and maintained a rolling structured standard for the ticket definitions, with a class-based Python schema with adapters between versions. Over time, the ticket-definition entries could be used two-ways, either as direct filters or to open tickets automatically. Adding these filters and indexes required internal discussions, alignment, A/B testing, and continual development to have sufficient coverage across the many different failure modes for test cases. This final result was a highly optimized ETL+MapReduce pipeline, sufficiently batched to run on a single desktop while the user enjoyed a coffee. The resulting pipeline outputs were standardized and documented, so that anyone within the team could easily pick up the work without extensive knowledge transfer. ## Results The final result was a desktop-runnable spark pipeline >10x faster than the original method. By receiving results an order of magnitude faster, running the scanning and validating the resulting tickets could be done by one person in roughly 20 mins - 1 hour, for workflows where up to 500+ tests could be run. Through the standard implemented, we could much more reliably open tickets and resolve ownership, and used the tool outputs to keep external teams updated on the state of tests as soon as they finished. This gave the team roughly 10 hours a week back (roughly 1 hour per person) through efficient tooling and standard triaging. I had ownership of this project going from graduate through senior, and it became a key tool in my team's delivery pipeline to a point where some key stakeholders began adopting it. Throughout my time at Arm, I managed projects, improvements, documentation, and data standards relating to this tool. --- © Sebastian Ikin 2026 --- # Rafmer LLM Infrastructure > Rafmer · Domain: AI Infrastructure, MLOps, DevOps, System Architecture, Hardware, Observability · Skills: Kubernetes, k3s, Docker, CUDA, vLLM, llama.cpp, Python, Quart, Bash, Hardware Procurement, Hardware Architecture, Nvidia GPU, GPU Utilization, API Gateway, OpenAI API, HAProxy, Traefik, NFS, OpenVPN, Rancher, Prometheus, Grafana, Loki, DCGM, pynvml, Observability, High Availability, Load Balancing, Network Security, Cost Optimization Related assets: - [The HERMES agent frontend served live through the OpenAI-conformant API gateway, backed by the GPU cluster (2x speed).](https://sebikin.com/assets/rafmer/cluster_demo.gif) - [Rancher used for day-to-day cluster management: nodes, workloads and health across the k3s cluster.](https://sebikin.com/assets/rafmer/rancher_landing.png) - [Three single-GPU inference nodes (4th out of commission), each running its own LLM deployment.](https://sebikin.com/assets/rafmer/single_gpu_instances.png) - [The two dual-GPU nodes used for the larger context, high-concurrency dense deployments.](https://sebikin.com/assets/rafmer/dual_gpu_instances.png) - [Grafana + NVIDIA DCGM dashboards: GPU temp, power, memory and utilization across the nodes, the view that exposed the power-surge drops.](https://sebikin.com/assets/rafmer/graphana_viz.png) - [A live LLM inference deployment viewed in Rancher: pods, replicas and readiness managed by k3s.](https://sebikin.com/assets/rafmer/model_deployment.png) HTML version: [https://sebikin.com/cv/projects/rafmer-agent-infra/](https://sebikin.com/cv/projects/rafmer-agent-infra/) ## TLDR - Designed and built an in-house **multi-node Kubernetes (k3s) GPU cluster** from bare hardware through the application layer, 8 RTX 3090s across 6 GPU nodes, serving highly available LLM and embedding deployments behind an **OpenAI-conformant API gateway**. - Built **vLLM and llama.cpp inference deployments** with internal routing that separated LLM and embedding serving, maximizing GPU utilization without enterprise virtualization. - Solved request dropping under concurrency by exposing **GPU utilization (pynvml)** from the deployments and layering an **HAProxy request queue**, achieving maximum concurrency with zero dropped requests. - Delivered enterprise-grade model capability and throughput on **~$10k of consumer hardware** that paid for itself in under four months, netting up to **2.7 billion tokens per month** across the cluster. ### Infrastructure Demonstration ![Sped-up demonstration of the HERMES agent frontend served through the Rafmer LLM API gateway](https://sebikin.com/assets/rafmer/cluster_demo.gif) The HERMES agent frontend served live through the OpenAI-conformant API gateway, backed by the GPU cluster (2x speed). ## What? An in-house multi-node Kubernetes cluster (using the k3s engine) housing highly available LLM / embedding / service deployments, accessible through an authenticated OpenAI-conformant API gateway. The cluster housed 8 3090s across 6 GPU nodes and 2 compute / control-plane nodes. Control of the cluster was achieved through remote kubectl and Rancher. Observability was achieved through Loki, Prometheus and the NVIDIA DCGM exporter on Grafana. The cluster was built from the ground up, hardware through software, on consumer desktops running Ubuntu and accessible remotely with OpenVPN. ## Why? #### Validation of Rafmer thesis and Product PoC The goal with Rafmer was to build a deployment-agnostic AI platform that, over time, could be spun up on site and run the same data / LLM offering any managed service could. The initial target was a software stack competitive with offerings like Astronomer, Zapier and of course ChatGPT. The hardware goal was to build something horizontally scalable (with multi-node capacity as an upsell) and compute intensive enough to run the software offering on one or more nodes. Translated into sysadmin terms: - Must run distributed Apache Airflow (Astronomer / Zapier) - Must efficiently run self-hosted LLMs - Must efficiently run both services on a single node - Nodes must be added and removed efficiently with minimal downtime - Each service must be API-accessible, meaning servers, routing and proxying were required Building the on-prem cluster from the ground up achieved several things: Learning the actual tech to bring about the Rafmer product goal, and building out demo infrastructure for later customer engagements. #### Low Cost Infra On top of being a plain product demo, it doubled as the hosted infrastructure for the SaaS version of the Agent Platform. This way, I could work on both the software and hardware angles while capitalizing on the on-prem capabilities. The initial build-out had roughly 100 monthly active users (MAUs) in mind, but later optimizations and model releases made it capable of hosting closer to ~1000, through estimated improvements to peak concurrency. #### How it was used later As Rafmer's products and business evolved, the customer infra requirements decreased drastically. The cluster was later repurposed for internal work. Among the workloads were: - compute intensive distributed data-pipelines - highly concurrent llm-serving - [specialized agents](https://sebikin.com/cv/projects/dev-agent/) - build systems - a data lake - MCPs, APIs and other support services ## How? #### Building the Hardware The initial step was to settle on what kind of hardware to use. After estimating cost, reviewing model requirements and prowling socials for builds and performance, I settled on the RTX3090. The 3090 had 24 GB of VRAM and was the best in class GPU at its release time and popular amongst gamers. This meant they were relatively easy to get ahold of, and sold at competitive prices on the second hand market. Pretty early on in the process, I found that buying entire gaming rigs with the cards was very cost efficient, costing nearly the same as a single card itself. Two of the rigs had sufficient space for two cards and larger power supplies, and they became the dual GPU nodes. I had settled on the 20B - 40B model range, which fit snugly on a single card and meant I could free both RAM and CPU for regular compute workloads. A happy accident of the gaming rig approach was that most of them had 32 Gigs of RAM, more than enough to handle some load + model deployment (moving model from RAM to VRAM) at the same time. The dual GPU nodes needed new RAM to support larger deployments and new power supplies. The cluster build-out happened gradually over several months and on each new device the computer was set up with the following: - Ubuntu with NVIDIA drivers - NFS, Git, ssh and Docker - nvidia-container-manager, set as default for Docker and Podman - k3s-agent The k3s agent was then connected to the control plane, a dedicated node that housed the VPN and served as the cluster entrypoint. The nodes ran on a private network, only remotely accessible through the VPN (managed with certs and appropriate profiles) and with specific auth for cluster management. The whole process of adding a new node to the cluster took roughly 1.5 hours, mostly spent waiting for Ubuntu to install. #### Running k3s k3s was chosen as the Kubernetes engine simply for cross compatibility with other engines, simplicity of install and an opinionated stack. I preferred the opinionated stack so I could have routing, container repos and distributed storage working out of the box. It's no secret that kubectl is painful to use at times, and over time I mostly replaced it with Rancher, the native k3s cluster management interface. ![Rancher cluster overview showing the Rafmer k3s cluster nodes and workloads](https://sebikin.com/assets/rafmer/rancher_landing.png) Rancher used for day-to-day cluster management: nodes, workloads and health across the k3s cluster. The first infra to be deployed was the inference engine llm service, allocating each discrete GPU using the NVIDIA k8s-device-plugin. With the plain Kubernetes approach, the resulting deployment yaml, containers and code could be deployed on any other future infra, cloud or otherwise. ![Rancher view of the four single-GPU k3s machine pools](https://sebikin.com/assets/rafmer/single_gpu_instances.png) Three single-GPU inference nodes (4th out of commission), each running its own LLM deployment. ![Rancher view of the two dual-GPU k3s machine pools](https://sebikin.com/assets/rafmer/dual_gpu_instances.png) The two dual-GPU nodes used for the larger context, high-concurrency dense deployments. #### Building useful Inference The first requirement to deal with was maximizing GPU utilization, with support for both plain LLMs and embeddings to support Chat and RAG within the Agent Platform. Generally, embeddings were exposed by the underlying inference engine, but swapping models periodically also meant swapping the embedding model. From a database POV, this was the equivalent of swapping out your index every time the server deployment changed. In other words, I needed to separate the embedding serving from the model serving. It was out of the question to commit a full 24 GB GPU to a 1 GB model, which would also make the embedding deployment non-redundant. Running the embed on CPU would be orders of magnitude slower than GPU and contend with data-workloads, and adding dedicated nodes with smaller cards would increase the bare-metal maintenance surface for a single software feature. Normally the solution here would be to utilize GPU-virtualization to separate the deploys, but with this being a consumer card it didn't come with that. The solution that added minimal complexity was to run 2 instances of the inference engine and route internally within the deployment (as opposed to just running the bare inference engine). This allowed me to bake both vLLM and llama.cpp into the single deploy, with the engine chosen per model through deployment envs: ```python # From servers/model_server/server_nvd.py (engine selection per deployment env) ENGINE = os.environ.get("ENGINE", "llamacpp") EMBED_ENGINE = os.environ.get("EMBED_ENGINE", "llamacpp") class ModelLoader: async def load_llm(self): if not self.llm_worker: if ENGINE == "vllm": self.llm_worker = await asyncio.create_subprocess_exec( self.VLLM_CLI, *self.llm_vllm_args()) else: self.llm_worker = await asyncio.create_subprocess_exec( self.LLAMA_CPP_SERVER, *self.llm_llama_cpp_args()) logger.info("LLM Started") async def load_embed(self): if not self.embed_worker: if EMBED_ENGINE == "vllm": self.embed_worker = await asyncio.create_subprocess_exec( self.VLLM_CLI, *self.embed_vllm_args()) else: self.embed_worker = await asyncio.create_subprocess_exec( self.LLAMA_CPP_SERVER, *self.embed_args()) logger.info("Embed started") async def start(self): await self.load_llm() await self.load_embed() ``` The internal routing was plain Python: a Quart app proxying by endpoint, gated by a semaphore: ```python # Internal routing: LLM requests proxied to the model engine, # embeddings served by the dedicated engine instance alongside it. app = Quart(__name__) model_loader = ModelLoader() @app.route("/v1/chat/completions", methods=["POST"]) async def completion(): if not await try_acquire_sem(llm_semaphore): logger.debug("LLM Caught busy") # encourage LB retry return Response( response=json.dumps({"error": "busy", "detail": "no local capacity"}), status=503, content_type="application/json", ) async with llm_in_flight_lock: globals()['llm_in_flight'] += 1 method = request.method headers = {key: value for key, value in request.headers if key.lower() != 'host'} data = await request.get_data() params = request.args cookies = request.cookies logger.debug("Forwarding request") return await model_loader.proxy_request( f"http://localhost:{model_loader.LLM_PORT}/v1/chat/completions", method, headers, data, params, cookies) ``` The engine approach turned out to be a stroke of genius in the following year: with an arms race between open-model providers, the vLLM and llama.cpp projects resorted to pure patches and hotfixes as opposed to long-term releases to support the plethora of models, architectures and configs. From the Rafmer POV this meant I could download a model the day it released, build the inference engines from source in a Docker image and A/B test the performance of vLLM vs. llama.cpp and deploy the day a new model was released. #### High Availability and Concurrency I exposed the k3s deployments using a service and ingress-rule. This ingress was initially managed by Traefik, the built-in proxy router of k3s. Traefik used round robin to distribute traffic across the underlying deployments, but for long running, compute intensive LLM-calls this quickly became an issue. When a request was routed to an inference engine, it could usually handle the requests it was configured to process in parallel; anything above that and the request was either dropped or returned an error. When using round robin for long compute intensive requests, even with retries and at minimal GPU utilization, requests will drop by the time they have reached the inference engine. The deployment could be locked down at max concurrency using semaphores that returned errors, but Traefik would still round robin the retry. My solution to this was to bake pynvml into the deployment and use the actual GPU utilization to inform about the busyness of the instance. At a certain utilization threshold, the deployment returned a 503 signaling the proxy to try somewhere else. Live GPU utilization and memory readings came from a small pynvml helper in the deployment: ```python @staticmethod def get_gpu_util_mem_nvidia(): n = nvmlDeviceGetCount() assert n > 0, "No GPU found" h = nvmlDeviceGetHandleByIndex(0) util = nvmlDeviceGetUtilizationRates(h).gpu # percent mem_used_mb = nvmlDeviceGetMemoryInfo(h).used // (1024 * 1024) return mem_used_mb, util ``` The readiness endpoint then combined the semaphore state with those readings: ```python # Deployment readiness: reports busy state from in-flight requests # and real GPU utilization/memory via pynvml. @app.route("/readyz", methods=["GET"]) async def ready(): try: async with llm_in_flight_lock: busy = llm_in_flight >= 1 if busy: return {"status": "busy", "llm": "LLM Busy"}, 503 mem, util = model_loader.get_gpu_util_mem_nvidia() if util is None: raise RuntimeError("nvidia-smi not available, problem with GPU metrics") if mem < MEMORY_THRESHOLD_READINESS: return {"status": "not ready", "llm": "llm not fully deployed"}, 503 if util <= 80: return {"status": "ok", "llm": "ready"}, 200 return {"status": "busy", "llm": "LLM Busy"}, 503 except Exception as e: return handle_unexpected_exit(e) ``` The round robin problem and request queueing was still not solved by this, so another proxy was added between the client and llm-instance. This proxy was a simple HAProxy instance configured to match the parallelism of the underlying engine, re-route on 503 and queue everything else. With this approach, I was able to achieve maximum concurrency without dropping any requests. #### Observability As inference became more mature, problems became more ephemeral and hard to investigate across the entire stack. Moving from kubectl to Rancher helped debug the basic deployments and PVC issues, but still had some clear gaps. The key recurrent issues were: - Ephemeral deploy logs, gone on restart. - Lack of an overview of GPU state over time and deploys. - Log monitoring across all inference deployments. - Random GPU drops across multiple nodes The solution to this was to plug NVIDIA DCGM into the cluster and hook it up to Rancher's built-in Prometheus, with which I built dashboards to monitor temp, heat, memory and util. Based on this I was finally able to see the bigger picture and discovered the drops were caused by power surges. The solution here was to add power limiting to each node as a scheduled system service outside the cluster. ![Grafana dashboards of GPU temperature, power, memory and utilization across the cluster](https://sebikin.com/assets/rafmer/graphana_viz.png) Grafana + NVIDIA DCGM dashboards: GPU temp, power, memory and utilization across the nodes, the view that exposed the power-surge drops. Combined with the Loki plugin for Grafana, I was finally able to preserve logs, whack recurrent issues and debug clearly. With this in place, each new model deploy was stable, and any issues were observable and dealt with fast. Using these standard cluster observability tools, I was able to create agent-skills dedicated specifically to this cluster debugging. #### Bringing it all together Every open model and inference engine has different parameters, prompt templates and thinking-specific parameters rarely supported by frontend clients without extensive config. A common thread amongst frontend clients is that they always support the OpenAI-API as specified by OpenAI. Modern inference engines also support the rough specification, but often have per-model or architecture specific parameters that have to be included per request. On top of this, many frontend clients will fail hard if API-keys or model names did not conform to the ones provided by OpenAI. In order to bridge the gap between clients and engine, I built an API Gateway that translated the standard client requests into the engine-specific per-model version of said requests. The gateway translated a standard OpenAI-conformant payload into its engine-specific, per-model form: ```python # Use model name to map to specific qwen thinking and non-thinking params def inject_model_params_qwen3(body: dict): model = body.get("model", "no-think") defaults = { "temperature": 0.7, "top_p": 0.8, "chat_template_kwargs": {"enable_thinking": False} } if model in "think": defaults = { "temperature": 1.0, "top_p": 0.95, "chat_template_kwargs": {"enable_thinking": True} } for k, v in defaults.items(): body.setdefault(k, v) def route_to_ai_deploy(headers: dict, data: bytes, fullpath: str): body = json.loads(data.decode("utf-8")) original_stream = False model = body.get("model", "assistant") candidates = None if "/v1/chat/completions" in fullpath: # Injects Rafmer specific system-prompt if body["messages"][0]["role"] != "system": body["messages"].insert( 0, {"content": get_system_prompt_models(model), "role": "system"}) # injects thinking related and qwen based model params inject_model_params_qwen3(body) # Model and context based subservice routing candidates = build_route(body, fullpath) if "model" in body: del body["model"] # Always respond by streaming backend for standard forwarding method # Reconstruct streamed response on batched request. if "stream" in body: original_stream = body["stream"] return json.dumps(body).encode("utf-8"), candidates, original_stream ``` On top of that it also handled context-based routing to the dual GPU nodes, and authentication with dedicated user-scoped API-keys (part of a separate system). This way, the infra could connect to any agent, framework or frontend supported by OpenAI. #### Final Architecture: The final llm-infra was three layers: - API Gateway - HAProxy - Per GPU LLM-deployment The API Gateway handled Auth, model name to parameter translation and deployment routing. The high availability proxy handled the request queue and forwarding to non-busy deploys. The LLM Deployment handled GPU mem / util monitoring to report to the ha-proxy server. The deploy image had the necessary cuda libraries, vLLM and llama.cpp built from source and read engine parameters as injected deployment env variables. A second inference instance dedicated to a standardized embedding model ran alongside the llm engine routed within the deploy with plain python code. Each engine was managed with asyncio piping to stdout (to be captured by Loki) and a single engine crash was registered as a deployment failure for self healing. The LLM model was read from a mounted PVC connected to a central NFS server on the control plane node. ![Rancher deployment detail view of a Rafmer LLM inference deployment](https://sebikin.com/assets/rafmer/model_deployment.png) A live LLM inference deployment viewed in Rancher: pods, replicas and readiness managed by k3s. This meant the initial model deploy for new models took a very long time, waiting for the model to propagate. However, the simplicity of just downloading the model to a single dir on release made up for it. ## Results This was easily the most challenging technical undertaking of my career. The project spanned multiple years and had to adapt to several external factors, like model releases, consumer limitations and business requirements. This was a living project, maintained and improved over time. Across that timespan the following was achieved: #### Integrations and Security - Built a fully OpenAI-conformant API Gateway, allowing integrations into nearly all clients - Supported completions, embeddings and responses where the underlying engine allowed - Locked down the API Gateway with a user system with JWT-tokens - Model routing through gateway, bridging front and backend #### Deployment Flexibility - Deployment with configurable context and concurrency for the underlying engine - Inference and model flexibility - Architectural choices resulting in new models being day 1 deployable with underlying engine patches - Separated model and embed, allowing standardized fields for the embed-db without sacrificing model flexibility #### Utilization and Scheduling - Per deploy GPU utilization semaphore allowing accurate utilization reporting - Separate high-availability proxy configured to route and queue llm and embed requests on actual GPU utilization - Separation of llm and embedding models with internal deployment routing, enabling homogenous service deploys and maximizing GPU utilization #### Cost Achievements As a mostly bootstrapped and cash-strapped start-up, the cluster build-out saved Rafmer a lot of capital. Running a similar stack in the cloud, with the same capability (distributed compute, storage and llm), would have cost close to 3000 dollars monthly (numbers from Sept 2025). The final infra cost approximately 10000 dollars in CapEx and roughly 300 dollars in recurring power cost (being in Norway helps, but this depends on utilisation and power prices). The hardware cost justified itself in less than 4 months against the estimated cloud equivalent (Sept 2025 estimate), and with late 2025 GPU price hikes the impairment loss (lost hardware value) was also minimal. In terms of model capability, the consumer hardware combined with the internal deployment routing approach enabled Rafmer to squeeze out enterprise capability, throughput and concurrency from the whole cluster. An enterprise system running with similar capability would set you back between 30k - 54k dollars (comparing dual NVIDIA A100 and H100, and just the cards). #### Model Performance and Inference Value Rough performance numbers across Mixture of Experts and Dense llm deployments. | Model | Node type | Concurrent requests per deployment | Context length (tokens) | Output tokens/s per deployment | Measurement month | | --- | --- | --- | --- | --- | --- | | Qwen3.5-35B-A3B q4 (MoE, llama.cpp) | 4 single, 2 dual GPU nodes | 4 | 300k | 130 | May 2026 | | Qwen3.8-27B q4 (Dense, vLLM) | 2 dual GPU nodes | 8 | 500k | 80 | Aug 2026 | | Qwen3.8-27B q4 (Dense, vLLM) | 4 single GPU nodes | 4 | 200k | 40 | Aug 2026 | Single = one RTX 3090 per node; dual = two RTX 3090s per node. All figures are per deployment instance. The dense 500k-context figure was measured at peak concurrency across the two dual-GPU nodes. The MoE deployment and the 200k-context dense deployment each also served a separate embedding model alongside the LLM. The MoE setup netted an estimated 2.7 billion tokens per month across the cluster. Monthly totals are estimates, not metered usage: they extrapolate the measured per-deployment throughput to around-the-clock saturation. The estimated OpenRouter value of the monthly output token throughput was roughly 2000 dollars, calculated using llama.cpp serving behaviour as the prior, i.e. throughput per client drops as concurrency rises. The two dense deployment modes (2 dual-GPU nodes and 4 single-GPU nodes) each netted an estimated 415 million tokens per month, for a total of 830 million per month across the cluster. The estimated OpenRouter value of that monthly throughput was 1800 dollars. OpenRouter prices were taken from the equivalent models' output tokens on Oct 22, 2026. --- © Sebastian Ikin 2026 --- # Other > Self-Directed · Domain: Cloud Engineering, Backend Development, Web Development, FinTech, Quantitative Finance · Skills: AWS, Serverless Architecture, Backend Development, Database Management, Event-Driven Architecture, TypeScript, Web Development, Frontend Development, 3D Graphics, Algorithmic Trading, Quantitative Finance, Python, Financial Modeling, Backtesting, API Integration, Software Design HTML version: [https://sebikin.com/cv/projects/other/](https://sebikin.com/cv/projects/other/) This page details various smaller projects and technical endeavors, ranging from AWS backend development to algorithmic trading systems. * **Work-Shift Scheduler:** An event-driven system built with AWS SAM and Lambda to automate work-shift scheduling and notifications. * **Bli Med Oss:** A web-based 3D game developed using React and Three.js to report on indoor climate data. * **Financial Algorithmic Systems:** Developed backtesting scripts, trade executors, and portfolio management systems using Python and Interactive Brokers APIs for swing trading and ETF allocation. ## Automated Work-Shift Scheduler (2023) Built an AWS backend and event-driven notification/SMS system for scheduling work-shifts. It was part of a broader product where the Admin and Client functionalities were handled by other teammates. It was a side project done with a 3-person team, and my deliveries was the entire backend, DB, and related business logic. Project was shelved after MVP due to the team moving on to other projects. #### Stack AWS SAM, Lambda, Aurora DB, AWS EventBridge, AWS SNS, Prisma ORM, TypeScript, and Twilio ## Bli Med Oss (2019) A web-based 3D game for reporting indoor climate, developed as part of my Research Assistant role at [NTNU](https://sebikin.com/cv/edu/). You can interact with the project [here](https://www.blimedoss.com/). #### Stack React, Three.js, HTML, CSS, JavaScript, Squarespace, and CAD ## Finance Related Projects (2023 - 2026) As an investor, I have periodically utilized algorithmic order execution and backtesting. Done part-time over several years as a way to understand the underlying logic behind investing better. I use data from multiple sources, and Interactive Brokers due to the maturity of its commonly available software integrations. **Broker Stack** [IB GW through this project](https://github.com/gnzsnz/ib-gateway-docker), TWS, ib_insync, Python For regular joe investing, I use Nordnet since it's reporting fits well with my approach and they provide Norway specific accounts. #### Backtesting As part of both swing trading, options-trading, ETF allocation, and general ideation, I usually run some backtests before committing to a strategy. Usually as simple as a single Python script taking in data from both yfinance, IBKR, DataBento depending on the type of instrument and its availability to do some transformations and plot it out. My general risk metrics are historical drawdown and drawdown correlation between holdings. My general performance metrics are backtested returns (using Sortino) against the benchmark index across favorable and unfavorable market regimes. The approach is very simple and flawed, but allows for direct comparison against commonly available ETF and mutual fund data. **Backtesting Stack**: Python, pandas, yfinance, numpy, and seaborn #### Trade Executor Initially built to algorithmically handle order entries and exits, such as staggered stock buys with multiple limits, or a sell order if some non-price condition is met. It was later expanded to do both buy, sells, and monitoring the state of each strategy within an event loop. The rough architecture was entry and exit strategies implementing a BaseStrategy and wrapping listeners and order submissions around ib_insync, all ran by a top-level worker process. Additionally, at the top level, there are separate strategy-dependent data feeds (usually just fetching the price) and handling of per-step conditionals to move the worker state from entry through exit. The system has been tested with paper-trades as part of backtesting and been used live twice. It has not been stress-tested at scale. **Portfolio System Stack**: Python, AsyncIO, ib_insync #### Full Portfolio System There have been attempts at a broader top-down portfolio/rebalancing/order scheduling system. Versions of this system have only ever been used for paper trading. **Rough Design**: * **UI / API / Broker**: User submits orders to the system and maintains a regular portfolio with the broker. * **Manager**: Watches the portfolio from the broker and maintains a queue of user orders. * **Rebalancer**: Plans a series of orders to rebalance the portfolio based on the user's requests, maintaining overall metrics described in the backtesting section. * **Order Scheduler**: Enqueues the rebalancing strategies and submits them to the executor. Applying a fully automated system to real trades is hard to justify as opposed to ad-hoc trades, when most API-accessible brokers exist outside the Norwegian deferred-tax account regime. A prior would also be for the executor to be properly field-tested. **Portfolio System Stack**: Python, Docker, FastAPI, AsyncIO --- © Sebastian Ikin 2026 ---