Image Classification

ARM · Domain: Machine Learning, Data Engineering, MLOps, GPU Bringup, Software Engineering · Skills: Python, C++, cv2, NumPy, PIL, PyTorch, TensorBoard, React, Jira, Computer Vision, Image Processing, Image Classification, Hyperparameter Tuning, Model Validation, Data Labeling, Dataset Creation, CLI Tooling, CI Integration, Drift Monitoring, Technical Presentations, Documentation, Cross-Team Standardization, Cross-Team Coordination, Stakeholder Management, Knowledge Transfer, Mentoring, Process Formalization, Project Management, Image Processing, Computer Vision

TLDR

Built an image difference classification system to validate GPU driver quality during hardware bringup, replacing slow manual image comparison.
Progressed through three iterations, from a C++-ported PoC to a Python CV pipeline (cv2, NumPy, PIL), with later work moving toward a PyTorch-based ML approach.
Established a cross-team image quality standard, reference dataset and labeling process across four hardware bringup teams.
Achieved about 70% precision, cut manual work by about five hours per week, and deployed the model into production test pipelines, with an internal classification UI and automated drift tracking for continuous improvement.

What?

This system classifies image differences used to validate GPU driver quality within the Hardware Bringup Flow. The final classification system is a CV pipeline written in Python, validated through confusion matrices against a human labeled test set. Later iterations utilized PyTorch. Precision is the share of flagged image differences that were confirmed as real differences.

This project was completed at ARM Norway. Some technical details are omitted due to ARM Intellectual Property (IP) restrictions.

Why?

In GPU hardware bringup, it is necessary to evaluate driver quality across various platforms and thousands of workloads. For graphics workloads, the output is often a complex scene, featuring multiple geometries, different lighting, and various effects.

The quality of this output needs validation, often by comparing it against a reference image. Human validation of these reference comparisons is possible, but it is time-consuming and expensive due to the sheer volume of tests. Direct pixel comparisons do not work due to precision differences between the driver/GPU, and traditional image noise metrics fail to capture low-pixel, high-contrast image artifacts.

In the context of GPU validation, uncaught image generation errors can end up with the end customer when they are running the final tests on their end device. This negatively impacts the company’s perception of quality, and in the worst case, can indicate uncaught underlying hardware issues.

The project was undertaken to reduce engineering hours spent comparing images, minimize false negatives, and formalize an organizational methodology for defining “good” and “bad” images across bringup teams. Additionally, we needed systems for CI and to ensure the new approach could be plugged into production pipelines.

How?

The development was split over three iterations, with improvements achieved through automated drift tracking, formalized ownership, standardized processes, and standardized procedures for difference reporting.

Establishing a baseline

Across four different hardware bringup teams, each possessed a separate error standard and a unique approach to validation. Before any coding work began, I created a small reference set of image differences where all teams agreed on the labels, established a Jira Epic for reporting new findings, and designated “image experts” on each team. The reference set and the reporting from the image experts were used to test the iteration 1 approach.

I served as my team’s dedicated image expert and held full responsibility for the verification algorithm and setting the cross-team standard. This was a mostly isolated “skunkworks” project, which few people were willing to take on due to the abstract requirements. The goal was to build a solution viable for the pipelines of all four teams.

Iteration 1

I rediscovered older pre-ML approaches used by animation studios, involving image kernels and contrast techniques. The existing example implementations were in pure C++, and I ported them to modern Python CV to serve as a Proof-of-Concept (PoC) using reference images. The initial approach showed promise and already outperformed existing pipelines. Prioritizing pre-ML pure Python was crucial to ensure portability into our existing pipeline.

Data

To move into the next step, I needed to establish a much broader reference set. To achieve this, I did the following:

Collected a large set of images from our production pipeline.
Developed a simple React application to quickly browse and label images.
Authored documentation on common errors and the “image expert” standard.
Persuaded my manager to secure team contributions for labeling based on the provided documentation.

The resulting data was labeled after about a month. A training set derived from this data was subsequently used by iteration 3.

Iteration 2

This iteration was based on the findings of the PoC and a key research paper from Nvidia.

Iteration 2 was a complete rewrite, resulting in a clean image-processing and classification pipeline with fully configurable hyperparameters, tunable against the established reference set. The new approach primarily used image processing techniques combined with ML-adjacent computer vision methods focused on capturing contrast differences.

Once tuned, iteration 2 performed at 70% precision, which was good enough to be integrated into our current production pipeline. The tool was also exposed as a CLI, utilized by several other teams, and integrated into another team’s production pipeline. The findings and validation results were shared across the organization via an engineering presentation. Both iterations were primarily built using Python, cv2, NumPy, and PIL.

Continuous Improvement

In an environment where data flows continuously, work is never truly finished. The parameters of iteration 2 were adjusted as new human-provided cases were tracked via tickets, analyzed, and then run against the reference set. Due to the architecture of iteration 2, we could add new “filtering” steps for rare edge cases or batches of known errors. Automated data collection was also implemented in the production pipeline to continuously monitor classification drift and export new cases to a data directory for reference set expansion. Classification drift detection was built into the production pipelines, while data collection was a separate system entirely. Collected data was made available for human labeling through the classification UI.

Iteration 3

During my final days at ARM, I coordinated the team’s efforts to transition to a pure ML-based approach validated through TensorBoard and built with PyTorch. This was in conjunction with handing over the workflows and domain knowledge related to the iteration 2 algorithm and establishing future development directions.

Results

The project spanned roughly 1.5 years and was completed alongside my regular deliveries (e.g., workflow maintenance and test-related tasks).

Designed and implemented a classification system achieving approximately 70% precision.
Cut manual work by about five hours per week.
Presented implementation and findings to the organization.
Model taken into production.
Replaced faulty older methods in a pre-existing test pipeline with the new algorithm.
Maintained an organization-wide tool to run the algorithm and established feedback channels.
Led a CI project where test errors and user feedback were integrated into concrete algorithm improvements.
Built, ran, and maintained an internal classification UI to establish a baseline test set.
Guided and managed the direction of the novel ML-based approach as the final iteration.