Patchwork project
The first year of Patchwork's Dashboard
The Patchwork Dashboard provides regular glimpses into what kind of traffic is crossing the network of the FABRIC testbed. The Dashboard is itself an ongoing experiment, and this post describes a milestone in the Dashboard's development.
Last Friday's entry in the Patchwork Dashboard took place exactly once year after the first entry on the Dashboard!
To mark the occasion, I extended the Dashboard's analysis pipeline to provide additional graphs. Compared to the existing graphs on the Dashboard, these graphs provide additional information from the regular Patchwork runs that feed the Dashboard. I also improved the appearance and accessibility of the Dashboard. While we're on the topic, this post shares some anecdotes from running the Dashboard. At the end of the post, I outline how the updates to the Dashboard dovetail with some other ongoing research.
Acknowledgement
Reaching this point would not have been possible without the many people who helped build and run Patchwork! Collaborators (including students who worked on Patchwork), FABRIC, and its community all made valuable contributions to Patchwork, and they are acknowledged in the Patchwork paper (PDF).
What's new
Two changes have been made to the Dashboard on its anniversary. First, the analysis and visualization pipeline was extended to produce more graphs, as seen in Fig 1. This extension relates to ongoing work on increasing what is shared through the Dashboard—skip to the end of this post to find out more. Second, the Dashboard has been updated to improve its accessibility, simplify its dependencies, and spruce up its appearance. This update is shown in Fig 2.
How the dashboard came about
Why have a dashboard? Within the Patchwork project, we had kicked this idea around for some time. A dashboard would provide several benefits: (1) provide a means to share findings with our research community; (2) provide a continuous stream of data, showing how FABRIC's network traffic changes over time; (3) force us to run Patchwork regularly and improve its resilience and usability.
The Dashboard idea didn't have enough substance to give it to students who I was working with at the time, who were mostly interested in low-level systems research. That's to say that, while I'm sure a student would have done a great job with producing a dashboard, the process of producing a dashboard didn't require (or help develop) deep technical skills, and therefore risked short-changing a student for their time.
So I decided to cobble this together myself. Although the Dashboard went live in September 2025, there was an early prototype of it in March 2025. Between March and September 2025 we were running Patchwork on FABRIC, only not streaming that data onto the dashboard. It took time to find the time to finish the dashboard!
How the dashboard works
The Dashboard's setup is kept as simple as possible. It relies on three cron jobs. The first cron job runs every few days to run Patchwork and gather data from FABRIC. The second cron job runs regularly to notice when a Patchwork run has completed and new data is available. The cron job's script then collects this data and instantiates a new entry in the Dashboard. That entry provides information about the Patchwork run—such as how many sites were involved and how much data was collected. This script also places the placeholder ("Still Looming") images for graphs, since the graphs are generated by the next step. Finally, the third cron job runs the analysis and visualization scripts on the collected data, and adds graphs to that run's entry in the Dashboard.
The scripts run by the cron jobs are set up to accommodate the variable amount of time that they can take to complete. Sometimes the analysis and visualization pipeline terminates quickly because only a few hundred megabytes of data is collected (and therefore analyzed), but other times it takes longer because tens of gigabytes of data is collected. (This variability depends on how busy is FABRIC's network during the run, and how many collection points could be established.)
Anecdata and Meta-anecdata
Building and running the dashboard requires very different thinking than what I use in my other research activities, but I find the two types of thinking to be complementary. Gathering data over a long period requires a different sort of dedication when compared to the focus needed to build a research prototype, but in turn that gathered data is serving as a foundation for new research.
One key difference in the two types of thinking is that the Dashboard should continue to be available and must navigate resource variability on FABRIC and intermittent failures, whereas a research prototype typically must serve to evaluate an idea or hypothesis (usually in an environment whose conditions are less variable).
Patchwork was matured through several alpha and beta releases, and that helped provide a bedrock of reliability for the Dashboard, but several other failure modes show up when running the Dashboard. We needed a way to consistently document and deal with failures.
From the start, I decided never to delete entries from the Dashboard—even when those entries are about failed or degraded runs. Instead, I'd add notes explaining why those runs failed. You can see examples of such notes related to degradation from incidental reasons unrelated to Patchwork and attempting to reuse FABRIC resources too rapidly, and related to failure from configuration bugs in Patchwork and from expired tokens.
Other issues we encountered while running the dashboard include sustaining the dashboard's scalability and availability.
For scalability, we need to ensure sufficient storage and computing resources. The computing resources needed to process a single Patchwork run usually far outstrips what we can handle on our equipment, therefore we reduce the data aggressively. I have an idea of how to overcome this, but that'll have to wait for the future. (Incidentally, I also have ideas of additional tweaks we could do to gather more data, but that's not core research at the moment.)
For availability, we need to mitigate inherent and external sources of instability. An inherent (though low) source of instability involves reacting to changes in FABlib's API and its semantics over time. You can see an example of this in Fig 10 of the Patchwork paper (PDF). Another inherent (but tolerable) source of instability is the variability in FABRIC resources that Patchwork can use for its data gathering. As explained in the Patchwork paper (PDF), we designed Patchwork to tolerate having low resources.
Examples of external instability include evolving Python dependencies and the location of the server on which the Dashboard's processing is done. For example, the original server room incurred occasional power blinks, therefore the server was configured to boot into a state where Patchwork can run. More recently, I heard that our department will likely be moved to a different building, so I took advantage of FABRIC's August maintenance window to move the server to a new datacenter and test the setup there, to avoid disruption when the department moves.
Did you say dAIshboard?
Updating the Dashboard also presented an opportunity to think through the use of AI. Up to now, everything in Patchwork—including the Dashboard's scripts—was completely and exclusively hand-written. (And all the words in this post are mine alone!)
In this update, I made use of AI to explore improvements to the appearance and accessibility of the Dashboard and the project webpages, and to explore variations of data extractions and graphs. The use of AI was limited to presentation and to an extension to the analysis and visualization pipeline. I didn't find the AI to be useful for ideation, but I found it very useful to navigate the minutiae of producing compliant and accessible HTML, and synthesize invocations of mathplotlib's API. For each graph, I worked through what it was showing—similar to how I would during a research meeting—since I believe that, left unchecked, it's easy to get nonsense results when using AI. The actual data-gathering, analysis, and visualization code we distribute with Patchwork continues to be AI-free.
I tried to be sparing in the use of AI for several reasons. First, we don't have a clear picture about its resource usage and environmental cost, though it's difficult to overlook the systemic costs we're incurring from it. Second, there are ongoing debates and policy changes about whether AI-generated code should be included in open-source projects or hosted in repo servers. Third, I observe that AI-generated code tends to be overengineered, and I worry that this creates a dependency to continue using AI to maintain that code, since the code becomes difficult to maintain manually. (See also Kernighan's Lever.)
What's next
One of my motivations to revamp and extend the dashboard is to complement new research into how to share network data safely. Working on Patchwork has heightened my appreciation of the difficulty and importance of data sharing, and I'm working on new techniques to facilitate that further. Those techniques are being trialled using Patchwork data. On that note, if you have any requests for graphs, analyses, or any other types of outputs related to Patchwork-gathered data, please reach out to me (homepage). Of course, all data requests will be cleared through the appropriate process, including ensuring compliance with FABRIC's policies.