Month: January 2023

When we don’t recognize that growth and control are in tension with each other, we tend to develop a bias that prevents us from spotting problems with the decisions we make. To better make sense of how this bias shows up – and how to account for it – I will rely on the dandelions and elephants framing. As you may recall, the whole point of dandelions is growth, and elephants are all about holding on to value in a well-defined niche.

The more dandelion-like the environment, the more likely we are to exhibit bias toward control, having higher than warranted confidence in our ability to predict the shape of the product niche that is yet to be discovered. This bias is entrenched in the usual “brilliant engineer/scientist” stereotype: the plucky hero who can see the solution way before anyone else. My experience is that these brilliant minds are narratives: they are stories of the lucky survivors whose being in the right place at the right time is reinterpreted – and entrenched in myth – as some superhuman ability.

In a dandelion environment, everyone is just stumbling around to find out. The less firmly we hold on to our conceptions about the final form of the product, the more we invite serendipity. The more optionality we embed into our own stumbling around, the more likely we are to hit the vein of something interesting. The early pivots of products that led to Flickr and Slack are great examples of doing this well. Poor Stewart Butterfield. He keeps trying to build a video game, and these successful businesses pop out instead.

One typical pattern emerging from the control bias in a dandelion environment is placing early big bets. When we see a large organization spun up for a focus area whose shape is not fully known, we are likely seeing the “control as cost” scenario: the mere size of the organization and the need to justify its existence will prevent it from embracing serendipity. The book Showstopper! is a great read for an inside story of the suffering this anti-pattern causes.  I call these kinds of products “elephantine dandelions” – they are simply too big to be agile among other dandelions in the wind of change, yet they continue to exist, buoyed by their size.

Conversely, the more elephant-like our environment, the more likely we are to bias toward growth, presuming that it is something we can simply rely on. There are a couple of patterns here that I am familiar with. One shows up as a belief that continued growth is the organization’s birthright – despite the fact that most of the effort is expended eking out the last bits of value under the curve in the well-defined niche. Thankfully, Clayton Christensen’s concept of disruptive innovation provides an antidote to this bias – though given that the pattern keeps repeating across all industries, I am not sure how effective this antidote is.

Another pattern of growth bias is what I call the “never-ending long tail.” Every so often, a new contender comes up to disrupt a well-established niche, with the tantalizing promise of breaking through the mold of the well-optimized niche. In such cases, it’s very important to watch for the long tail of consumer expectations: are they still there? When the newcomer shows promise and inspires a compelling new vision of what’s possible, are they playing in an entirely new space, or are they still on the hook to deliver on all existing, mostly implicit contracts with the customer?

A good example here is the emergence of new Web rendering engines. I have seen a couple of announcements crop up here and there, and I am totally rooting for them as a fellow engineer, but as a strategist, I have very little hope that they’ll survive under the crushing weight of all the quirks and barnacles that a modern Web rendering engine must support. After initial success and admiration of colleagues (“look! it’s so much faster than others!”), a never-ending long tail of conforming to the same shape within the same niche awaits them.

I alluded to this tension before, but it was somewhat buried in a fairly obscure context of compounding loops. Here’s a somewhat unkempt riff on the concept.

When introducing a product into the hands of our customers, we will nearly always face the tension between growth and control. Here, growth reflects the number of people who choose to use our product, and control is the degree to which we get to decide what this product will be and how it might be used.

Basically, think of these as two extreme ends of a line that can never move close to each other. We can travel along the line with our product strategy, but never bend it to make control and growth meet. We can either lean toward growth and thus relinquish more control, or we can lean toward control and thus give up on more growth.

Interestingly, the quests for growth and control both stem from the same intention toward value. We seek growth when we are looking to acquire value, and we seek control when we seek to protect it. And almost always, we want to have both: we are intent on holding on to some value and are looking to acquire more. Truly blessed are those who don’t have attachments: they are blissfully unaware of this tension. If one has nothing to lose, they have no need to grow. For the rest of us, the growth/control tension is something we experience every day.

For example, if we’re shipping an API, we usually want its usage to go up (growth) and we want developers who use this API to only use it the way we want to (control). When we imagine a new service, we want it to be enjoyed by as many users as possible (growth), yet we also have very firm ideas on what we want to build (control).

Often, control isn’t something we have due to value we already hold, but rather due to a debt we carry. Writing software or making hardware takes time. In the gap between us starting the project and it becoming available to our customers, we crystallize our ideas into code and circuitry. Think of this crystallization as a form of control: we make decisions about what the product should be, and by making these decisions we exercise control over its shape. 

This crystallization is also an unrealized value – we won’t start accruing this value until the product is released to the users. When we do, many of these decisions are somewhat irreversible: even if we want to relinquish control and adapt our product to stimulate its growth, we can’t do so easily. Especially with hardware, a single decision made somewhere in depth of the multi-year gap between ideation and shipping can severely limit a product’s capacity to grow.

The outcome of growth has a similar trappy quality. If our product is blessed with exponential growth, after reaping its benefits for a while, we will eventually arrive at the weirdest situation. No matter how much we try, we can’t do anything but optimize for what is already expected by our massive user base. Any attempt at deviation triggers the allergic reaction of irate customers: “why are you moving my cheese and how soon can you put it back?” In this trap of successful growth, control is no longer ours, no matter how much we try to wrestle for it.

I realized that I never captured what I learned about the nature of open source projects anywhere, so here’s one attempt to rectify this omission.

The way I understand the concept of “open source” is that it’s a way of running a software project. In this definition hides a gazillion of options, and I will try to lay them out in a spectrum. Understanding these options as a spectrum has helped me quite a bit in reasoning about software project strategy, and I hope this will be helpful for you, too.

As bookends for our spectrum, let’s position two radical alternatives: fully closed-source and fully open source projects. As an example that might resonate with Web developers, we can see that the late Internet Explorer fits very close to the fully closed-source extreme and Firefox is at the other side of the spectrum.

To place other projects on the line between these bookends, we need a couple of notches. I will define these notches loosely, and give them numbers. So, any project that you’re evaluating for their degree of open source, you can use the whole part of the number as the rough estimate of where a project sits within the spectrum, and then the decimal part to adjust more finely.

We will also have two numbers! They are both located on the same scale. One number will reflect visibility and the other participation. Project visibility describes how much the general public can see into the project, and participation shows how much anyone can engage with the project. So, to describe Internet Explorer, we can firmly give it two zeroes: zero for visibility and zero for participation, or v0p0. I will use this convention from here on: a string of “vNpM”, where N is the number indicating the scale of visibility, and M is the number indicating the scale of participation for a project.

How high can these numbers go? I propose that we use a what/how/why framing. It seems to fit well. Think of the spectrum that defines the degrees of open source as separated by three veils of invisibility.  

🏠 The what

The first veil is the “what” veil. Outside of it, all you get is the final artifact – the product that resulted from compiling/running/hosting the source code. Anything that’s outside of this veil will get a grade of zero (0) in our scale on both participation and visibility.

Once this veil is lifted, we can start seeing into the “what” of the project. For example, we could start seeing the source code itself, even if it’s just some file with source code in it. We can examine what was written, and try to understand what the code does. Anything that’s between this veil and the next gets a one (1) in visibility. 

To evaluate participation, we ask the question: “how can a random developer who comes by this code provide feedback on it?” For example, does the project have the means to allow commenting on code or perhaps making suggestions or filing bugs? A good example of an OSS project that is at v1p0 is Apple MacOS OSS distributions on Github: you can see the code, but there’s literally nothing else you can do to participate. Should this project gain some means of communication (a mailing list, a Discord server, a bug tracker, etc.) that allow the general public to contribute their insights, it would move to a v1p1.

🏗️ The how

The next veil is the “how” veil, outside of which we can see what the source code is, but can only guess about how it’s been written.

After we lift this veil, we can start seeing how the code is being written. We can start getting a sense of the units of work (bugs, commits, etc.) and how they come together as the source code. Inside this veil, we start seeing things like actual commits by actual contributors, rather than automated bot code dumps. Importantly, we start seeing how these commits are connected to particular bugs and feature requests, and the actual engineering/product discussions about them are revealed.

Participating at this level means that there is a path for anyone to introduce their own units of work into the project: to start contributing to these discussions, to start and guide them, to create pull requests and have a path to merge them, etc. Depending on the kind of set up the project chooses, there are many different ways to do so, but the key property of achieving this degree of open source participation is that anyone can walk up to the project and contribute meaningfully to how the project’s source code comes together.

Projects that have visibility and/or participation beyond the veil of “how” will have a grade of two (2). For example, a v2p1 project will have its bug tracker and commits open to the public, but no way to actually become a committer to the project, outside of filing bugs or providing ad hoc feedback. Lots of open source projects that come from large companies are run this way. With a v2p1 project, I as a random developer can check out the code, follow the steps to get it running, and even fork it. What I can’t do is join the circle of project developers and say things like: “Oh, I really love what you’re doing with <foo>! Here’s a PR to refactor <foo> for clarity”.

To get to v2p2, a project must have a clear path to do so. Typically, there’s some section on the project site that describes how one can start contributing, outlining the development environment, and the basic processes that go into the “how”. Apple’s Webkit is a great example of such a project, with a clear outline on how to get started with the source code, and how to begin to contribute. There’s even a diagram outlining the process flow as well as the famously elegant code style guidelines. Effectively, it’s a batteries-included participation toolkit: as much as possible, the tacit knowledge of the team is documented clearly and concisely. If one has the necessary programming skills, they can jump right in. 

📐 The why

The final veil is the “why” veil. Beyond that veil, we find the project’s motivations and strategy. Why is this project even here? What are the leaders of the project thinking? Why are we choosing to work on this unit of work and not the other?

This is the realm of project governance. Giving the general public the ability to understand (visibility) and/or influence (participation) project direction is the ultimate degree of openness for a project, and we give it the degree of three (3) on our scale.

Many open source foundations take this approach. It’s not for the faint of heart, since the “why” can contain significant value. A project that adopts a v3p2 stance is effectively accepting that the value of the project is not within the confines of the organization that runs it, but rather in the community that gathers around the project.

Such projects will openly state their intentions, outline their strategy and approach, and eagerly look for feedback. The whole idea is to lean onto the wisdom of the crowd to get it right.

An even more radical stance is v3p3, where there is a path for anyone to contribute to the governance itself – not just evaluating and helping shape directions and priorities, but also change them. In projects like this, in addition to clear roadmaps and open strategy, the governance is fully transparent, and there isn’t a point at which the capacity to make decisions goes into a tunnel.

✨ The Open Source Scale

Now we have our four-grade scale, with the numbers going from zero (0) to three (3). When visibility and participation are combined, I presented them in a bit of a pattern: the visibility goes one in front of participation, with participation catching up.

Are there other combinations of grades? Are there v3p0 or v0p2 projects out there? Maybe, although my experience tells me they are rare. Typically, a successful open source project only has one degree of difference between the grades, and the participation trails the visibility. One can’t participate at a certain grade, without having visibility at the same grade. Similarly, the setups where participation trails by too much (like v2p0) are rarely sustainable: eventually the project owners arrive at “what’s the point?” question and either close down visibility or further open up participation.

The key insight for y’all: the difference between the two grades nearly always results in additional overhead and friction. The larger the difference, the more likely it is that the project will be overwhelmed by this overhead, which will inevitably lead to a change in its grade.

So, when you’re planning a new open source project, please make sure to:

• clearly understand the grade you’re starting with;
• if this grade is unsustainable, understand the grade you’re aiming at and devise strategies to get there as quickly as possible;
• put in place project operating principles that hold it within this grade.

Hope this helps and good luck on your adventures in open source! 

There’s this gap that I’ve been struggling with for a long time, and I often feel like I am no closer to finding a way to bridge it. It’s a gap between nuance and action. I’ve talked about many variants of problems that this gap causes, but at their core, they all share the same dynamic.

Very loosely, the dynamic is that understanding nuance requires so much work in addition to the important, urgent work that already needs to be done, that folks who need to understand this nuance rarely have energy/time to do so.

Suppose you have invested a bunch of time into studying a problem space – like growing developer ecosystems, for example. You have acquired admirable depth of nuance in this problem space. You understand how APIs evolve. You understand the layering fundamentals of opinion. All that good stuff. Now, you would like to convey this depth to others. These others are typically your team, but could be just random people on the Internet.

You choose the option of carefully writing words and drawing diagrams to describe the nuance in the problem space as concisely and clearly as you can. Then you share the resulting artifact with your team. And… nothing happens. At best, people diligently flip through your artifact and go: “This is really nice. But why so abstract? How does this translate to action?”

This question has become somewhat triggering for me and my fellow ecosystem adventurers. Nuance often feels too abstract, even to a most receptive audience. 

I wonder if it’s because nuance points at how to think, rather than what to think. This might be an important distinction.

We act on what we think, so “what to think” is close to the ground, and an artifact that outlines what to think will tend to feel more concrete. Do this. Don’t do that. Go here. Stay there.

All the while, the “what we think” is situated inside of how we think. Put differently, how we think defines all possibilities for action. When we change “what we think”, we just pick another of those possibilities. When we change “how we think”, we start seeing another set of new possibilities. Until then, our field of view is limited to the initial set.

Because nuance affects how we think, it will nearly always feel detached from action. Modifying “how we think” means introducing different mental models, a whole new space from which new actions could be drawn. And there’s a lot of effort that needs to go into accommodating these new mental models, integrating them into one’s understanding.

Only after that happens, new actions will emerge – but there will be an inevitable delay between conveying the nuance and the actions appearing. I bet there was a puzzled look on the face of the proverbial man who was expecting the fish, and instead got a fishing rod thrust into his hands and an invitation to walk to the lake.

When everyone is busy to the point of delirium, stopping to learn how to fish feels both impossible and foolish. So what do we do?

I don’t know. But I will keep trying to find different ways of sharing nuance with others – and of course, keep sharing what I learned with you.

I stumbled into this framing when talking about product/platform thinking with my friends. To put this framing together, we have to employ a certain way of thinking about software products – viewing them as a graph.

Imagine a graph – a bunch of vertices and edges that connect them. In this particular graph, bits of software we write are represented vertices, and the formats, protocols, and APIs we use to glue these bits together are represented by the edges. Both are important and necessary.

Represented like this, the two groups (the software and the glue) can be discussed separately. For example, when I talk about an API to get some data (like current time) from a particular server, I can see what appears to be one thing – the API to get current time – as two distinct entities. In our graph representation, there will be the vertex representing the code that is required to obtain current time and format it according to some requirements, and the edge, which represents the actual API contract that defines the requirements to be implemented.

C++ has this separation built into the language: to write useful code, a developer has to produce both the header file (the edge) and the source file (the vertex). Other languages have the construct of an interface that serves a similar purpose, though most of the time, this distinction between edges and vertices is more of a practice. One of the more familiar ones is test-driven development (TDD), where we are encouraged to start by describing the behavior of the code we want to write in tests, and then writing the code itself.

This distinction reveals two different thinking modes that appear to be complementary, but independent of each other: the vertex-wise thinking and the edgewise thinking.

 When I have my vertex-wise thinking hat on, I am usually contemplating the actual building blocks of the thing I intend to make. This one is very common and a bit of a habit for software folks: when asked “what are we building?”, the answer shows up as a list or a matrix of things, a to-do list of sorts. “To get <foo>, I will need <bar> and <baz>” and so on. This approach is super-familiar to any maker of things, professional engineer or not. While in this thinking mode, the vertices are at the center of attention and edges are an emergent artifact. We are focused on constructing the blocks, and how they are connected is something we figure out as we go. 

The situation inverts with edgewise thinking. When I have the edgewise hat on, I am primarily focused on the edges. I want to understand how the vertices connect – before figuring out what they’re made of. Here, the glue between the blocks is the subject of my attention, and the blocks themselves are an emergent outcome. When asked “what are we building?”, the answers that the edgewise thinkers give usually come in the form of control flow or other kinds of interaction diagrams, defining how the vertices will communicate with each other. Edgewise thinkers obsess about formats and protocols.

One of my mentors had a habit of designing through mocks: sketching out an entire project as a collection of dummy classes that do the absolutely minimal amount of work to glue together. You could build this project and run it, and it would actually print out sentences like “contacting bluetooth stack” and “getting results”, none of it real: it was to be implemented later.

Such edgewise thinking exercises allowed us to see—and mess with—the shape of the entire thing long before it existed. The mocks served as the edges. Once we figured them out, vertex work was just a matter of typing out code that conformed to the shapes the edges defined.

The inevitable question in an article of this sort arrives: which one is better? It depends on the situation. A colleague of mine captured the rule-of-thumb well: edgewise thinking tends to lead to divergent effects and vertex-wise thinking to convergent.

Vertex-wise thinking works exceptionally well – and is called for – when we know pretty well what we’re looking for as the end result, and the biggest obstacle we’re facing is proper sequencing of work. Program managers can do wonders turning lists and matrices of building blocks into burn-down charts and shipping schedules, and help the team click through the milestones.

Edgewise thinking can’t compete with that. In fact, folks who favor edgewise thinking tend to struggle with calendars. However, one of the superpowers of edgewise thinking is the ability to surprise ourselves. If we leave vertices as an emergent outcome, we create the potential for something we didn’t anticipate: different ways of using the edges to build a whole different thing.

For example, imagine that we are Tim Berners-Lee back in the 1980s and we want to organize the documents across departments at CERN. Applying vertex-wise thinking, we end up with a pretty cool database that any CERN member can use. Applying edgewise thinking, we end up inventing the Web.

Platform and ecosystem work is first and foremost edgewise thinking. We start by imagining how various bits will connect and what the value of these connections might be. We find ways to ensure that this value is preserved and increased by designing the edges: interfaces, formats, and protocols that define the connections. We intentionally leave the vertices blank and invite ecosystem participants to invent them by relying on the connection mechanisms we designed. 

For me, a really useful marker in a software design conversation is when the notion of “reusable building blocks” comes up. It’s a good sign of a thinking hat confusion: the participants are engaged in vertex-wise thinking when they need to think edgewise. A “reusable building block” is not really a block, but rather the glue that makes it reusable. When in this conversation, it might be good to pause and reorient, switching hats: discuss interfaces, formats, and protocols, rather than the actual code that will fill out the block. 

Here’s another little experiment I built over the break. Like pretty much everyone these days, I am fascinated by the potential of applying large language models (LLMs) to generative use cases. So I wanted to see what’s what. As my target, I picked a fairly straightforward use case: AI as the narrator.

While I share in the excitement around applying AIs as chatbots and pure generators of content, I also understand that predictive models, no matter how sophisticated, will have a hard time crossing the chasm of the uncanny valley – especially when pushed to its limits. As a fellow holder of a predictive mental model, I am rooting for them, but I also know how long this road is going to be. We’ll be in this hollow of facts subtly mixed with cheerful hallucination for a while.

Instead, I wanted to try limiting the scope of where LLM predictions can wander. Think of it as putting bounds around some text that I (the user) deem as a potential source of insights and asking the AI to only play within these bounds. Kind of like “hey, I know you’ve read a lot and can talk about nearly anything, but right now, I’d like for you to focus: use only these passages from War and Peace for answering my questions”.

In this setup, the job of an LLM would be to act as a sort of well… a language model! This is a terrible analogy, but until I have a better one, think of it being a speech center of a human brain: it possesses incredible capabilities, but also, when detached from the rest of the prefrontal cortex, produces conditions where we humans just babble words. The passages from War and Peace (or any other text) act as a prefrontal cortex, grounding the speech center by attaching it to the specific context in which it offers a narration.

Lucky for me, this is a well-known pattern in current LLM applications. So, armed with a Jupyter cookbook, I set off on my adventure.

A key challenge that this cookbook overcomes is the limited context window that modern LLM applications have. For example, GPT-3 is only capable of just over 4000 tokens (or pieces of words), which means that I can’t simply stuff all of my writings into context. I have to be selective. I have to pick the right chunks out of the larger corpus to create the grounding corpus for the narrator. So, for example, if I have questions about Field Marshal Kutuzov eating chicken at Borodino, yet I give my narrator only the passages from Natasha’s first ball, I will get unsatisfying results.

I won’t go into technical details of how that’s done in this post (see the cookbook for the general approach), but suffice to say, I was very impressed with the outcomes. After I had the code running (which, by the way, was right at the deca-LOC boundary), my queries yielded reasonable results with minimal hallucinations. When in the stance of a narrator, LLM’s hallucinations look somewhat different: instead of presenting wrong facts, the narrator sometimes elides important details or overemphasizes irrelevant bits.

Now that I had a working “ask ‘What Dimitri Learned’ a question” code going, I wondered if the experience could be more proactive. What if I don’t have a question? What if I didn’t know what to ask? What if I didn’t want to type, and just wanted to click around? And so the Wanderer was born.

Here’s how it works. When you first visit wanderer.glazkov.com, it will pull a few random chunks of content from the whole corpus of my writings, and ask the LLM to list some interesting key concepts from these chunks. Here, I am asking the narrator not to summarize or answer a specific question, but rather discern and list out the concepts that were mentioned in the context.

Then, I turn this list of concepts into a list of links. Intuitively, if the narrator picked out a concept from my writings, there must be some more text in my writings that elaborates on this concept. So if you click this link, you are asking the wanderer to collect all related chunks of text and then have the narrator describe the concept.

Once you’ve read the description, provided by Wanderer, you may want to explore related concepts. Again, I rely on the narrator to list some interesting key concepts related to the concept. And so it goes.

When browsing this site (or “wandering”, obvs), it may feel like there’s a structure, a taxonomy of sorts that is maintained somewhere in a database. One might even perceive it as an elaborate graph structure of interlinked concepts. None of that exists. The graph is entirely ephemeral, conjured out of thin air by the narrating LLM, disappearing as soon as we leave the page. In fact, even refreshing the page may give you a different take: a different description of the concept or a different list of related concepts. And yes, it may lead you into dead ends: a concept that the narrator discerned, yet is failing to describe given the context.

The overall experience is surreal and – at least for me – highly generative. After wandering around for a while, it almost feels like I am interacting with a muse that is remixing my writing in odd, unpredictable, and wonderful ways. I find that I use the Wanderer for inspiration quite a bit. What concepts will it dig up next? What weird folds of logic will be unearthed with the next click? And every “hmm, that’s interesting” is a spark for new ideas, new connections that I haven’t considered before.