Bug Classification

What?

A bug classification pipeline that pulled logs from datacenter infrastructure and classified failing tasks. The underlying tool was a Python-based CLI, utilizing PySpark to distribute the workload and cross-reference it across known issues and symptoms. The pipeline took raw logs and turned vague errors into structured output, ready to be converted to tickets and assigned to the appropriate stakeholder.

Built during my time at Arm, with MVP as an intern, and gradually expanded through the years. Some details are omitted to respect IP.

Why?

As part of my team’s role in the GPU Bring Up process, we ran a large volume of per-GPU workflows running many different tests. Each of these tests was its own multi-step workflow, where each step could be owned by us or other external teams. In simple terms, the workflow itself ran build -> run -> validate -> extract for every test the customer or stakeholder required. As you can imagine, this produced GBs of logs and artifacts distributed across several locations.

My team was responsible for monitoring said workflows for issues across the entire stack and notifying the appropriate teams. Due to the breadth and scale of the tests needing to be validated, doing it manually would consume the entire teams bandwidth.

Some automation was happening through simple but inefficient shell scripts, but this approach bottlenecked the delivery pipeline and took hours. The goal was to structure the chaotic logs and artifacts, and distribute/parallelize the parsing in a Pythonic way using modern data engineering techniques.

How?

I started the project as an intern in 2020 and delivered a crude prototype based on porting the old shell scripts to plain Python. The initial design used the shell-based logic within multi-node Spark workers orchestrated by a central script handling the SparkSession.

It quickly became apparent that Arm’s LSF-based infrastructure had too much load to timely assign worker nodes, as it was used across the org. Another issue with running it on the infrastructure was that NFS sync timings ate up gains and introduced intermittent errors, due to having to sync the workflow’s entire directory tree across all worker-nodes.

To solve the issue, I moved the load onto our team desktops, caching only the interesting artifacts and using heuristic filtering of known open issues, sorted by priority and relevant tests, to pre-reduce the compute load before processing anything. When the scan ran, the ticket-relevant spark tasks targeted only the file and symptom described by the ticket. Once the ticket targeting was done, another task queue of ambivalent failures was scanned across the entire issue index. This approach minimized the really expensive computations to a few unaccounted tests at the end.

To enable this, I developed and maintained a rolling structured standard for the ticket definitions, with a class-based Python schema with adapters between versions. Over time, the ticket-definition entries could be used two-ways, either as direct filters or to open tickets automatically. Adding these filters and indexes required internal discussions, alignment, A/B testing, and continual development to have sufficient coverage across the many different failure modes for test cases.

This final result was a highly optimized ETL+MapReduce pipeline, sufficiently batched to run on a single desktop while the user enjoyed a coffee. The resulting pipeline outputs were standardized and documented, so that anyone within the team could easily pick up the work without extensive knowledge transfer.

Results

The final result was a desktop-runnable spark pipeline >10x faster than the original method. By receiving results an order of magnitude faster, running the scanning and validating the resulting tickets could be done by one person in roughly 20 mins - 1 hour, for workflows where up to 500+ tests could be run.

Through the standard implemented, we could much more reliably open tickets and resolve ownership, and used the tool outputs to keep external teams updated on the state of tests as soon as they finished.

This gave the team roughly 10 hours a week back (roughly 1 hour per person) through efficient tooling and standard triaging. I had ownership of this project going from graduate through senior, and it became a key tool in my team’s delivery pipeline to a point where some key stakeholders began adopting it. Throughout my time at Arm, I managed projects, improvements, documentation, and data standards relating to this tool.