The Starship of Theseusthree days beforeThis framing of the Starship of Theseus has been on my mind…The Plumber Age

Stages of project forking2023I’ve been thinking recently about how to design an open source project,…This framing jumped out of a conversation with the fellow FLUX-ers and I just gotta write it down. It appears that every software project, upon hitting its stride of success, eventually arrives at what I call the “plumber age”. Here in this story, I use “plumbing” to refer to the work of improving the quality of the code – as opposed to “feature work” that adds new functionality.

The Plumber Age according to Midjourney

The Plumber Age is the period of time when making further progress depends primarily on the quality/velocity of refactoring and cleaning up the code base.

Separating cargo from the train2023I’ve been puzzling over a problem that many engineering teams face and…The rewrite count2021As a software engineer, I often see a struggle that both teams…Forces of the pace layering confusion2021I had a super-insightful conversation this week with a colleague about approaches…I like to think of it as a season in the lifecycle of a software engineering adventure. This season is characterized by the rapid rise in importance of refactoring and rewriting existing code. At some point, a software project encounters this interesting moment, where the number of new features and capabilities slows down to a trickle, because adding them takes longer and longer. Everyone on the team wants to keep going, and there’s a lot of frustration and tension, but the code seemingly begins to actively resist being changed. Every new bit of code breaks some existing behavior, blows up performance metrics, or other shenanigans.

Build a thing to build the thing2023When building new products, there’s always a weird tension between making something…Most useful software is born out of a bunch of hacks. Sometimes they are really solid hacks, and sometimes they are nearly literally scotch tape and popsicle sticks. And it makes sense. We want to go fast, get something in the users’ hands, and iterate like crazy, listening to what our users tell us. In the process, we accumulate hacks – also colloquially known as “accruing technical debt”. Sometimes, the hacks aren’t born as hacks. We might conceive of a rather elegant design at the start, but the contact with reality unveils our modernist naivete, and what seemed like a paragon of software engineering turns rapidly into an annoying vestige – now covered in hacks.

Shadows and Curses2022The second curse can be seen as the interplay between “ 🔮…At some point, the hacks congeal into a solid mass that becomes nearly impossible to reason about. Veteran engineers will stare carefully at the code and move with artful precision, only for the code to laugh back at them. Often, crossing into the Plumber Age happens rather suddenly, almost like being locked out of the house. It feels like just yesterday we were able to land new bits and then – bam! – a Cthulhu is staring back at us through our code editors. The pile of hacks filigreed into a tangled network and now, this network transitioned into its next phase.

Aircraft Carriers and Zodiac Boats2024An insightful conversation with a colleague inspired me to articulate the distinction…Enter the Plumber Age. When embraced, this season is marked by the emergence of the demand for, well, plumbers. People who enjoy (or at least tolerate) moving the code around become more and more common – and sometimes prominent – within the organization. The nature of the project changes. That crazy idea of tests and test infrastructure, previously mostly ignored, is considered a critical investment. Talk of build and development metrics crop ups everywhere. Words like “services” and “layering” start showing up in everyday work conversations. Plumbing becomes a full-time job, and there may even pop up a team whose job it is to maintain and modernize some stable “core” of the software. They allow other teams to continue to move forth with feature and capability development.

Organizational cartography2022Given that this team is thriving, the various units within it collaborate…Most interesting to the leaders of an organization that needs to embrace the Plumber Age is the change in how the teams are structured. The loose and free-spirited flat setup that works so well to get our enterprise going becomes a hindrance. In the Plumber Age, code ownership and clear delineation of roles and responsibilities becomes a necessity. Hierarchies and RACI matrices start popping up all over the place. This may give us a queasy feeling – are we becoming a rigid bureaucracy?! Not necessarily. If we are to continue on this path and keep increasing the user value of our software, a more structured organization is needed. How carefully and thoughtfully we structure it is up to us. My intuition is that it is the fear of becoming bureaucratic that actually turns engineering organizations into bureaucracies. Adding more structure intentionally is not bad. However, pretending to not be bureaucratic will guarantee the feared outcome.

Builder and Gardener Mindsets2022Briefly, the distinction emphasizes the difference in mindsets with which we approach…The miracle count2021While working on a technical design this week, my colleagues and I…Some organizations try to reject the Plumber Age, and attempt to skirt around it in various ways. Few build second systems, and are rarely heard from again. It’s hard to see all the miracles that led our horrifying pile of hacks to become a successful first system. Others push brashly forward, chewing through talent and becoming more and more challenging to move – until they keel over and die. There are probably other paths and combinations, each leading to similar outcomes.

The Layering Principle2023This might seem counterintuitive, but start writing code without explicitly putting these…Making Fiftythree days afterThis one will be a bit unusual. Instead of my typical sermons…Limits of applying AIlinks here, 2023Looking at the LLM bounding box – or more accurately, the lack…My guess is that the Plumber Age is inevitable. It is something that every team will encounter in due course. And just like the seasons, it is best experienced when prepared. This does not mean that each new team must start with a thoroughly designed infrastructure or hire a cadre of plumbers. That’s a whole other trap that I will save for later discussion. However, as the team starts to feel the lift of successful engagement with its users, it needs to anticipate the transition. Start having regular conversations about architecture and layering. Draw sketches of where the core of the system begins and feature work ends. Consider how features and capabilities can be added without affecting the core. Think about how the team might be structured. Are there tech leads emerging who could lead the sub-teams? Oh yeah – almost forgot: teach your code to be testable. All of these will come handy, kind of like those coats and hats you’re storing for winter.