Workflow System Testing Framework

ARM · Domain: Software Engineering, DevOps, Testing & QA, GPU Bringup · Skills: Python, unittest, OOP, TDD, mocking, API, Regression Testing, CI/CD, Git, DevEx Tooling, Documentation

TLDR

What?

A central delivery managed by my team at Arm was a workflow submission tool that defined steps, tests, runtime environments, and pushed to a central cluster running Arm’s compute infra. This project was to establish a comparison baseline across the breadth of flows that could be submitted, build a validation system where the workflow requests could be compared against that baseline, and integrate the checks as part of our CI/CD flow.

Some details are omitted to respect IP. Most of the work for this project was done in 2022.

Why?

To give a sense of the breadth of workflows, we were running ~20 substeps with tooling developed across 4 teams, including their specific environments, patches, and env-vars/tooling args, with specialized steps being a dot product of test requirements (say raytracing, upscaling, or just plain compute workloads), the features of the GPU, and the GPU runtime (FPGA vs software model). Each workflow run could batch up to 100+ tests, with each flow being quite complex on its own. There are thousands of tests available to run, and each test + workflow configuration can potentially be unique. Every step, although extensively unit-tested, imports and exports from the previous step, with a list of expected files to carry over. The compute cluster was both NFS-constrained and compute-constrained, so we didn’t blanket-export everything. For every step, the imported tool path and its appropriate env had to be correctly defined and picked up in our runtime environment. This meant that every input and command-line chain had to be defined in the workflow request, whether based on Docker, virtualization, plain binaries, or Python scripts (with different version support depending on the GPU). For reference, the workflow definition was a JSON several thousand lines long, defining task dependencies, tool env arguments, runtime environments, etc.; that JSON was then sent as an API request to the cluster scheduler.

Given that breadth and complexity, just checking the unit-test logic and looking at individual substeps of the flow was not enough. We needed to ensure every valid output combination was functioning for every test we supported (and we’re talking thousands). During my first years at Arm, the team was struggling with maintaining workflows that weren’t run on a weekly basis, and new features or requests for legacy data would almost always fail in production. I was given the not-so-trivial task of putting together an output validation system that, without running on the cluster, could validate feature support from the workflow definition all the way through the generated submission request.

A goal I set for myself was to build something that would be as painless as possible, meaning our DevEx of extending, modifying, and supporting brand new workflows within that system should be version-controllable, diff-comparable, exclude per-run junk data (timestamps, IDs, etc.), and be human-readable. A clearly templated workflow benchmark to easily compare and test against.

How?

Baseline Generation

First, a baseline for every unique flow variant needed to be established. I classified every type of test triggering its own behavior, along with every subflow variant, and wrote a script that would iterate over each variant and dump the request before it was sent to the cluster scheduler. The script then sanitized the request body, removing things like timestamps, IDs, and prefixes/suffixes, leaving just the “definition” of the workflow. This definition includes the full end-to-end task flow as defined in the original JSON, plus its static internals and dependencies. The resulting script was a simple pipeline calling a per-baseline wrapper around subprocess, easily extendable with new baselines. Baselines were then version-controlled within the repo, leaving an audit trail in case any patch led to structural updates of the underlying result.

Testing Framework

In order to utilize the baselines in a systematic manner, they needed to be integrated into our CI/CD flow. Changes like cross-team version or tooling updates could break things outside of my team, and minor changes to components (changes in filenames, suffixes, etc.) could result in imports/exports breaking. Having validation run every time a patch was pushed was a no-brainer to mitigate this failure mode and catch external problems early.

This meant that for every baseline, I needed to generate and compare the generated workflows in an efficient manner and in a way that was easily re-usable for future extensions of the tests. To make this happen, I built a unittest.TestCase (Python’s builtin) parent class, wrapping data operations like on-demand sanitization, diffing per task, and diffing entire subflows. The generation command was an abstract parameter of the base class, allowing us to utilize polymorphism and inheritance to run basic testing of every test-case specialization (without writing the same test-logic over and over, very nice for legacy maintenance), with per-flow special-casing targeting individual task comparisons or subflows. For each specialized TestCase instance, we only run the workflow generation once, reusing the result across the per-task / subflow tests. A nice plus of doing it this way is that you can run each case straight from regular Python IDE functionality. This allowed easy rolling validation and extension from a DevEx POV. I documented the framework extensively to enable this further.

The workflow definitions take a while to generate, as they need to connect to a number of external services (container repos, test indexes, etc.). To avoid doing this many times over for the automated tests, I batch-fetched and cached the necessary files on disk once. This all came together into a separate TestSuite with a custom runner that monkey-patched connections and other singleton classes on a per-“workflow generator”-instance basis, allowing plain TestCase parallelism.

When a test case fails, it dumps both the reference and generated workflow definitions, as well as a numbered diff file, for comparisons and corrections. This gave the team a better overview of the side effects their patches might have, with a clear before-and-after.

Results

The resulting design had the following structure:

By finally getting the full workflow definition pipeline validated in CI/CD, production errors relating to structural issues on our end dropped significantly. The resulting framework covered ~50 TestCases (running parent and child class-tests) at ~5 min of dedicated CI runtime, with verified extensibility for both new internal and external tooling. By my final days at Arm, the tooling was part of critical production validation and had been a key resource during a full rewrite of the workflow submission tool where there had to be a one-to-one match between input and output.

The biggest gap the system has is that it validates the request contract, not runtime behavior: it can’t catch or validate what actually happens after submission. Any attempt to reach beyond that would mean validating against static snapshots of real runs. It’s also worth noting that if the underlying submission tool drifts too much (i.e., new external connections, or changes to the task definition schema), the test framework will break. In these cases a failure is more of a feature than a bug: baselines are version-controlled, so any structural change surfaces loudly in the diff and forces a deliberate review before it goes into a baseline, including its consequences for the other teams whose tooling feeds these flows.