Why AI orchestration

Why do I find the problem of AI patterns and more generally, AI orchestration so interesting that I literally started building a framework for it? Why do we even need graphs and chains in this whole AI thing? My colleagues with a traditional software engineering background have been asking me this question a lot lately.

Put very briefly, at the height of the current AI spring that we’re experiencing, orchestration is a crucial tool for getting AI applications to the shipping point.

To elaborate,  imagine that an idea for a software application takes a journey from inception to full realization through these two gates.

First, it needs to pass the “hey… this might just work” gate. Let’s call this gate the “Once” gate, since it’s exactly how many times we need to see our prototype work to get through it.

Then, it needs to pass through the “okay, this works reasonably consistently” gate. We’ll call it the “Mostly” gate to reflect the confidence we have in the prototype’s ability to work. It might be missing some features, lack in polish and underwhelm in performance benchmarks, but it is something we can give to a small group of trusted users to play with and not be completely embarrassed.

Beyond these two gates, there’s some shipping point, where the prototype – now a fully-fledged user experience – passes our bar for shipping quality and we finally release it to our users.

A mistake that many traditional software developers, their managers, and sponsors/investors make is that, when looking at AI-based applications, they presume the typical cadence of passing through these gates.

Let’s first sketch out this traditional software development cadence as a sequence below.

The “Once” gate plays a significant role, since it requires finding and coding up the first realization of the idea. In traditional software development, passing this gate means that there exists a kernel of a shipping product, albeit still in dire need of growing and nurturing.

The trip to the “Mostly” gate represents this process of maturing the prototype. It is typically less about ideation and mostly converging on the robust implementation of the idea. There may be some circuitous detours that await us, but more often than not, it’s about climbing the hill.

In traditional software development, this part of the journey is a matter of technical excellence and resilience. It requires discipline and often requires a certain kind of organizing skill. On more than one occasion, I’ve seen brilliant program managers brought in, who then help the team march toward their target with proper processes, burndown lists, and schedules. We grit our teeth and persevere, and are eventually rewarded with software that passes the shipping bar.

There’s still a lot of work to be done past that gate, like polish and further optimization. This is important work, but I will elide it from this story for brevity.

In AI applications, or at least mine and my friends/colleagues’ experiences with it, this story looks startlingly different. And definitely doesn’t fit into a neat sequential framing.

Passing the “Once” gate is often a matter of an evening project. Our colleagues wake up to a screencast of a thing that shouldn’t be possible, but somehow is. Everyone is thrilled and excited. Their traditional software developer instincts kick in: a joyful “let’s wrap this up and ship it!” is heard through the halls of the office.

Unfortunately, when we try to deviate even a little from the steps in the original screencast, we get perplexing and unsatisfying results. Uh oh.

We try boxing the squishy, weird nature of large language models into the production software constraints. We spend a lot of time playing with prompts, chaining them, tuning models, quantizing, chunking, augmenting – it all starts to feel like alchemy at some point. Spells, chants, and incantations. Maaaybe – maybe – we get to coax a model to do what we want more frequently. 

One of my colleagues calls it the “70% problem” – no matter how much we try, we can’t seem to get past our application producing consistent results  more than 70% of the time. Even by generous software quality standards, that’s not “Mostly”.

Getting to that next gate has little resemblance to the maturation process from traditional software development. Instead, it looks a lot more like the looping over and over back to “Once”, where we rework the original idea entirely and change nearly everything.

When working with AI applications, this capacity to rearrange everything and stay loose about the details of the thing we build, this design flexibility is what dramatically increases our chances of crossing to “Mostly” gate. 

Teams that hinge their success on adhering to the demo they sold to pass through the “Once” gate are much more likely to never see the next gate. Teams that decide that they can just lay down some code and improve iteratively – as traditional software engineering practices would suggest – are the ones who will likely work themselves into a gnarly spaghetti corner. At least today, for many cases – no matter how exciting and tantalizing, the “70% problem” remains an impassable barrier. We are much better off relying on an orchestration framework to give us the space to change our approach and keep experimenting.

This is a temporary state and it is not a novel phenomenon in technological innovation. Every new cycle of innovation goes through this. Every hype cycle eventually leads to the plateau of productivity, where traditional software development rules.

However, we are not at that plateau yet. My intuition is that we’re still climbing the slope toward the peak of inflated expectations. In such an environment, most of us will run into the “70% problem” barrier head-first. So, if you’re planning to build with large language models, be prepared to change everything many times over. Choose a robust orchestration framework to make that possible.

Makers and Magicians

I want to finally connect two threads of the story I’ve been slowly building across several posts. I’ve talked about the rise of makers. I’ve talked about the magicians. It’s time to bring them together and see how they relate to each other.

First, let’s paint the picture a little bit and set up the narrative.

The environment is ripe for disruption: there’s a new software capability and a nascent interface for it, and there’s a whole lot of commotion going on at all four layers of the stack. Everyone is seeing the potential, and is striving to glimpse the true shape of the opportunity, the one that brings the elusive product-market fit into clarity.

As I asserted before, there’s a brief moment when this opportunity is up for grabs, and the ground is more level than it’s ever been. Larger companies, despite having more resources, struggle to search for the coveted shape quickly due to the law of tightening aperture. Smaller startups and hobbyists can move a lot faster – albeit with high ergodic costs – and are able to cover more ground en masse. Add the combinatorial power of social networks and cozywebs, and it is significantly more likely that one of them will strike gold first.

For any larger player with strategic foresight, the name of the game is to “be there when it happens”. It might be tempting to try and out innovate the smaller players, but more often than not, that proves to be hubris.

Instead of trying to be the lucky person in the room, it is more effective to be the room that has the most exceptionally lucky person in it – and boost their luck as much as possible.

When the disruption does finally occur and the hockey stick of growth streaks upward, such a stance reduces the chances of counter positioning and improves the larger player’s ability to quickly learn from the said lucky person.

Put simply, during such times of rapid innovation, the task of attracting “exceptionally lucky people” to their developer ecosystems becomes dramatically more important for larger companies.

If the story indeed is playing out like so, then the notion of magicians is useful to identify those “exceptionally lucky people” – because luck compounds for those who explore the space in a way that magicians do.

But where do makers fit in? A good way to think of it as overlapping circles of two groups: developers and makers.

We’ll define the first circle as people who develop software, whether professionally or as a hobby. Developers, by definition, use developer surfaces: APIs, libraries, juts, tools, docs, and all those bits and bobs that go into making software.

The second circle is broader, because it includes folks who both develop and interact with software in a way that creates something they care about. Makers and developers obviously overlap. And since “maker” is a mindset, the boundary between makers and developers is porous: I could be a developer during the day and a maker at night. At the same time, not all developers are makers. Sometimes, it’s really just a job.

Makers who aren’t developers tend to gravitate toward becoming developers over time. My intuition is that the more engaged they become with the project, the more they find the need to make software, rather than just use it. However, the boundary that separates them from developers acts as a skill barrier. Becoming a developer can be a rather tough challenge, given the complexity of modern software.

Within these two circles, early adopters make up a small contingent that is weighted a bit toward makers. Based on how I defined maker traits earlier, it seems logical that early adopters will be primarily populated by them.

A tiny slice of the early adopter bubble on the diagram is magicians. They are more likely to be in the developer circle than not, since they typically have more expertise and skill to do their magic. However, there are likely some magicians hiding among non-developer makers, prevented by the learning curve barrier from letting their magic shine.

I hope this diagram acts as a navigational aid for you in your search for “exceptionally lucky” people – and I hope you make a room for them that feels inviting and fun to inhabit.

Zones of LLM predictability

As you may know, large language models (LLMs) are smack dab in the middle of my tangle of interests presently, so you can bet I spend a lot of time talking with my friends and colleagues about them. One lens that seems to have resulted in fruitful conversations is the one related to predictability of output.

In this lens, we look at the LLM’s output as something that we can predict based on the input – and the reaction we might have on the outcomes. If we imagine a spectrum where the results are entirely unpredictable at one extreme, and can be predicted with utter certainty at the other – then we have a space to play in.

For a simple example, let’s suppose we’re asking two different LLMs to complete the sentence “roses are red, violets are …”. If one LLM just returns a bunch of random characters, while the other consistently and persistently says “blue”, we kind of know where we’d place these models on the spectrum. The random character one goes closer to an unpredictable extreme and the insistent blue one goes closer to the perfectly predictable end.

For ease of navigating our newly created space, let’s break it down into four zones: chaotic, weird, prosaic, and mechanistic. 

🌫️ Chaotic

In the chaotic zone dwell the LLMs that basically produce white noise. They aren’t really models, but random character sequence generators.  By the way, I asked Midjourney illustrate white noise, and it gave me this visage:

(It’s beautiful, Midge, but not what I asked for)

This zone is only here to bookend the very extreme of the spectrum. Suffice to say that we humans tend to only use white noise as means to an end, mostly judging it as useless on its own.

🐲 Weird

The adjacent zone is where the model outputs something that is weird and bizarre, yet strangely recognizable and sometimes even almost right. Remember the whole “hands” thing in the early generative imagery journey? That’s what I am talking about.

(“A normal human hand with five fingers” – whoopsie!)

This zone is where LLMs are at their creative best. Sure, they can’t count fingers, and yes, some – many! – outcomes are creepy and disturbing, but they also produce predictions that are just outside of the norms, while still retaining some traits that keep them outside of the chaotic zone. And that stirs creativity and inspiration in those who observe these outcomes. This is the zone where a model is more of a muse – odd and mysterious, and not very serious. Yet, when paired with a creative mind of a human, it can help produce astounding things.

📈 Prosaic

The prosaic zone is where an LLM produces mostly the results we expect. It might add a bit of flourish in bursts of creativity and insert an occasional (very safe) dad joke, but for the most part, that’s the zone that I also sometimes call the “LLM application zone”. If you ever spend time getting your retrieval-augmented generation to give accurate responses, or only return code results that can actually run – you’ve lived in this zone.

(“a happy software engineer working, stock photo” – oh yes, please! More cliche!)

My own explorations are mostly in this zone. The asymptotes I outlined earlier this year are still in place, and holding. If anything, time has shown that these asymptotes are firmer than I initially expected.

⚙️ Mechanistic

Another bookend of the spectrum is the mechanistic zone. At this point, LLM output is so constrained and deterministic that we become uncertain if using an LLM is even necessary: we might be better off just writing “old school” software that does the job.

The mechanistic zone is roughly the failure case for the current “AI” excitement. Should the next AI winter come, we’ll likely see most of the use cases shift toward this zone: the LLM either constrained, significantly scaled down in size, or entirely ripped out, replaced with code.

💬 A conversation guide

Now that we have the zones marked in the space, we can have conversations about them. Here are some interesting starter questions that generated insights for me and my colleagues:

  • How wide (or narrow) is each zone? For example, I know a few skeptics that don’t even believe that the Prosaic zone exists. For them, its width is zero.
  • How much value will be generated in each band? For instance, the Prosaic zone  is where most of the current attention seems to be. Questions like “Can we make LLMs be useful at an industrial scale? How much value can LLMs produce?” seem to be on everyone’s mind.
  • How will the value generated look for each band? What type of value comes out of the Weird zone? What about the Prosaic zone?
  • What kind of advancements – technological or societal – would it take to change the proportions of the zones?

For more adventurous travelers, here are more questions that push the boundaries of the lens:

  • What does “predictable”even  mean? If I know English, but don’t have the cultural background to recognize the “Roses are Red” ditty, I might find the “blue” perplexing as a completion. Violets are kind of purplish, actually.
  • What do judgments about predictability of the LLM output tell us about the observer? What can we tell about their expectations, their sense of self, and how they relate to an LLM?
  • What is it that LLMs capture that makes their output predictable? What’s the nature of that information and what might we discern about it?

As you can tell, I am pretty intrigued by the new questions that large language models surface to us. If you’re interested in this subject as well, I hope this lens will be useful to you.

Doers, Thinkers, and Magicians

I’ve been reflecting on my experiences of working with developers and developer ecosystems, and I realized that there’s a really interesting twist on the typical “early adopter” story that’s been hiding in the back of my mind.

Let’s suppose that you and I are spinning up a new developer experience project. We have a fledgling developer surface that we’re rapidly shaping and growing, and trying to to get it right by making contact with its intended audience.

The very first of these developers are commonly called early adopters, which originated from Everett Roger’s book Diffusion of Innovations. It is my experience that these early adopters can be further broken into three subgroups: doers, thinkers, and magicians – and the presence of all three is required for the developer surface to successfully navigate toward broad adoption and bring forth our hopes and dreams for it. 

More than that, the mix of these subgroups heavily influences the arc that the project will follow, the pattern into which the developer ecosystem around the developer surface will settle into – should it succeed.

To explore this notion, let’s zoom in on each subgroup.

💪 Doers

The doer early adopters are typically the most populous sub-group. They are very easy to identify: they do stuff with our developer surface, making things with it, poking at it here and there.

Doers bring energy and create the sense of a bustling community emerging around technology or products, powered by technology. They are eager, excited, typically with some tinkering time to spare. They are boisterous, peppering technology or product builders with questions and suggestions. Most of their questions and feedback of a very practical nature: they just want to make our thing do their bidding.

Doers often don’t have enough technical skills to just start doing what they want – not just because our developer surface is new, but because there might be gaps in their understanding of the surrounding technologies. As such, they need patient and consistent investment of hand-holding, be that tutorials, hackathons, or individual support.

If our project has doers in the early adopter mix, we have a key ingredient. We have the potential energy that can be transferred into forward progress. This subgroup of early adopters provides valuable insights on the usability of the technology or product, and their contagious enthusiasm attracts new customers.

If our project doesn’t have doers, we might as well not have a project. The absence of doers in the early adopter mix is a warning sign that we might have come up with something that is deeply uninteresting, incomprehensible, or otherwise impossible to access by doers.

🧠 Thinkers

The thinker early adopters usually come in much smaller numbers than doers. In some ways, thinkers can be seen as a subset of doers, with a key distinction: they actually spend time imagining possibilities and exploring the possibilities of the technology they are studying. They might be playing with the developer surface themselves, but they could also just be observing doers and identifying interesting potentialities in the churning soup of ideation that the doers produce.

One of my first encounters with thinkers was back in the early 2000s, when blogs.msdn.com was introduced as part of Microsoft Developer Network. I was a fairly new doer-inclined developer myself, and I was fascinated by the blog posts from Dare Obasanjo or Nikhil Kothari on the then-nascent .NET framework. They moved from the pragmatic “here’s how you do <blah>” to open-ended cross-blog conversations about second order effects and implications of technology they were using, as well as introduced various completely new ideas into how it might be used. For me, whole new frontiers opened up and connections were made between concepts that I viewed as entirely unrelated – all the while making me ever more energized about the technology.

This is the role of the thinkers: they hold our developer surface in their hands lightly, turning this way and that, and applying intellect and curiosity to consider its potential.

When our project has thinker early adopters, we have acquired a source of more durable energy. While doers do introduce the initial energy, their explorations, being mostly pragmatic and practical, often peter out and lose steam without the influx of new ideas. Thinkers are the ones who introduce these new ideas, and reinvigorate the excitement and enthusiasm.

Not having thinkers as early adopters means that the project is in danger of getting stuck in a premature local maxima. When the doers uncover all the obvious use cases, these might not be the ones that propel our developer surface toward our intended destination – and we’ll have to contend with being stuck in what we view as “mediocre success” or just breaking down our camp and admitting defeat. It is the thinkers who help move doers move beyond the initial local maximas into adjacent areas that are more likely to hold the value we’re looking for.

✨ Magicians

The final ingredient in the early adopter mix are the magicians. The magician early adopters are even more rare. Even having one is an incredible stroke of fortune, and something that we are obliged to cherish.

The magicians are both doers and thinkers, but they have this weird knack for building amazing things that blow your mind. I wish I knew how that works. In the past, I attempted to grow magicians out of thinkers and/or doers, but there doesn’t seem to be a path from here to there.

The magicians are usually experienced and seasoned developers. They grasp the idea behind our developer surface in seconds, and intuitively see the landscape of opportunities. Then, they reach for the simplest path toward the opportunity that appears most valuable – and for some unfathomable reason – they are usually right. They connect bits and pieces of our stuff into something that suddenly looks solid and – this is a common effect – blindingly obvious. “What the hell?! How?! … Oh… Why didn’t I think of this before?!” is a common reaction to a magican’s artifact.

When I worked in Chrome Web Platform, I was very lucky to have a handful of these magicians around me. For some reason, the Chrome team’s DevRel contingent was rife with them. In a more recent memory, Simon Willison’s work on llm has the same magician quality.

The presence of magicians significantly strengthens our project’s chances of broad adoption. Like thinkers, the early adopter magicians uplevel the current understanding of what’s possible – but they do it in an explosive, revolutionary way. “Now <bar> is possible, here’s some code” – and everyone freaks out, dropping all the previous work, being able to not just imagine the potential of the next frontier, but actually try it. This explosive amount of energy that the magicians inject into a project can catapult it way beyond our initial intentions. We just need to be patient, hang on to our dear lives while the starship of a new idea streaks forward, and be ready to explore the crazy new planet it will land on.

Not every developer surface gets magician early adopters. A project can still be moderately successful even in the dearth of magicians. A very common side effect of that is that the developer community grows large and appears vibrant, but the outcomes it produces tend to be on the lower side of our expectations. Low-magician developer ecosystems tend to have very thin long tails, with only a few well-settled participants forming a narrow head.

🧪 The right mix

A reasonable question might be: what is the right mix for our project? Disappointingly, there is no satisfying answer. Developer-oriented projects, at least in my experience, all tend to follow roughly the same shape of proportions: lots of doers, a few thinkers, a couple – if any – of magicians. It is usually the presence of magicians and thinkers that significantly improves the chances of our project going somewhere good. So, if I could offer any advice to budding developer experience makers, it would be this: seek out the thinkers and the magicians. They are the key to passing through the early adoption stage.

Placing and wiring nodes in Breadboard

This one is also a bit on the more technical side. It’s also reflective of where most of my thinking is these days. If you enjoy geeking out on syntaxes and grammars of opinionated Javascript APIs, this will be a fun adventure – and an invitation.

In this essay, I’ll describe the general approach I took in designing the Breadboard library API and the reasoning behind it. All of this is still in flux, just barely meeting the contact with reality.

One of key things I wanted to accomplish with this project is the ability to express graphs in code. To make this work, I really wanted the syntax to feel light and easy, and take as few characters as possible, while still being easy to grasp. I also wanted for the API to feel playful and not too stuffy.

There are four key beats to the overall story of working with the API:

1️⃣ Creating a board and adding kits to it
2️⃣ Placing nodes on the board
3️⃣ Wiring nodes
4️⃣ Running and debugging the board.

Throughout the development cycle, makers will likely spend most of their time in steps 2️⃣ and 3️⃣, and then lean on step 4️⃣ to make the board act according to their intention. To get there with minimal suffering, it seemed important to ensure that placing nodes and wiring them results in code that is still readable and understandable when running the board and debugging it.

This turned out to be a formidable challenge. Unlike trees, directed graphs – and particularly directed graphs with cycles – aren’t as easy for us humans to comprehend. This appears to be particularly true when graphs are described in the sequential medium of code. 

I myself ended up quickly reaching for a way to visualize the boards I was writing. I suspect that most API consumers will want that, too – at least at the beginning. As I started developing more knack for writing graphs in code, I became less reliant on visualizations.

To represent graphs visually, I chose Mermaid, a diagramming and charting library. The choice was easy, because it’s a library that is built into Github Markdown, enabling easy documentation of graphs. I am sure there are better ways to represent graphs visually, but I followed my own “one miracle at a time” principle and went with a tool that’s already widely available.

🎛️ Placing nodes on the board

The syntax for placing nodes of the board is largely inspired by D3: the act of placement is a function call. As an example, every Board instance has a node called `input`. Placing the `input` node on the board is a matter of calling `input()` function on that instance:

import { Board } from “@google-labs/breadboard”;

// create new Board instance
const board = new Board();
// place a node of type `input` on the board.
board.input();

After this call, the board contains an input node.

You can get a reference to it:

const input = board.input();

And then use that reference elsewhere in your code. You can place multiple inputs on the board:

const input1 = board.input();
const input2 = board.input();

Similarly, when adding a new kit to the board, each kit instance has a set of functions that can be called to place nodes of various types on the board to which the kit was added:

import { Starter } from “@google-labs/llm-starter”;

// Add new kit to the existing board
const kit = board.addKit(Starter);

// place the `generateText` node on the board.
// for more information about this node type, see:
// https://github.com/google/labs-prototypes/tree/main/seeds/llm-starter#the-generatetext-node
kit.generateText();

Hopefully, this approach will be fairly familiar and uncontroversial to folks who use JS libraries in their work. Now, onto the more hairy (wire-ey?) bits.

🧵 Wiring nodes

To wire nodes, I went with a somewhat unconventional approach. I struggled with a few ideas here, and ended up with a syntax that definitely looks weird, at least at first.

Here’s a brief outline of the crux of the problem.  In Breadboard, a  wire connects two nodes. Every node has inputs and outputs. For example, the `generateText` node that calls the PaLM API `generateText` method accepts several input properties, like the API key and the text of the prompt, and produces outputs, like the generated text.

So, to make a connection between two nodes meaningful, we need to somehow capture four parameters:

➡️  The tail, or node from which the wire originates.
⬅️ The head, or the the node toward which the wire is directed.
🗣️ The from property, or the output of the tail node from which the wire connects 
👂 The to property, or the input of the head node to which the wire connects

To make this more concrete, let’s code up a very simple board:

import { Board } from "@google-labs/breadboard";

// create a new board
const board = new Board();
// place input node on the board
const tail = board.input();
// place output node on the board
const head = board.output();

Suppose that next, we would like to connect property named “say” in `tail` to property named “hear” in `head`. To do this,  I went with  the following syntax:

// Wires `tail` node’s output named `say` to `head` node’s output named `hear`.
tail.wire(“say->hear”, head);

Note that the actual wire is expressed as a string of text.  This is a bit unorthodox, but it provides a nice symmetry: the code literally looks like the diagram above. First, there’s the outgoing node, then the wire, and finally the incoming node.

This syntax also easily affords fluent interface programming, where I can keep wiring nodes in the same long statement. For example, here’s how the LLM-powered calculator pattern from the post about AI patterns looks like when written with Breadboard library:

math.input({ $id: "math-question" }).wire(
  "text->question",
  kit
    .promptTemplate(
      "Translate the math problem below into a JavaScript function named" +
      "`compute` that can be executed to provide the answer to the" +
      "problem\nMath Problem: {{question}}\nSolution:",
      { $id: "math-function" }
    )
    .wire(
      "prompt->text",
      kit
        .generateText({ $id: "math-function-completion" })
        .wire(
          "completion->code",
          kit
            .runJavascript("compute->", { $id: "compute" })
            .wire("result->text", math.output({ $id: "print" }))
        )
        .wire("<-PALM_KEY", kit.secrets(["PALM_KEY"]))
    )
);

Based on early feedback, there’s barely a middle ground of reactions to this choice of syntax. People either love it and find it super-cute and descriptive (“See?! It literally looks like a graph!”) or they hate it and never want to use it again (“What are all these strings? And why is that arrow pointing backward?!”) Maybe such contrast of opinions is a good thing?

However, aside from differences in taste,  the biggest downside of this approach is that the wire is  expressed as a string: there are plenty of opportunities to make mistakes between these double-quotes. Especially in a strongly-typed land of TypeScript, this feels like a loss of fidelity – a black hole in the otherwise tight system. I have already found myself frustrated by a simple misspelling in the wire string, and it seems like a real problem.

I played briefly with TypeScript template literal types, and even built a prototype that can show syntax errors when the nodes are miswired. However, I keep wondering – maybe there’s an even better way to do that?

So here’s an invitation: if coming up with a well-crafted TypeScript/Javascript API is something that you’re excited about, please come join our little Discord and help us Breadboard folks find an even better way to capture graphs in code. We would love your help and appreciate your wisdom.

Traits of a maker mindset

The whole concept of makers has been on my and my colleagues’ mind a lot, especially with their current rise in prominence with generative AI.  

A fun question popped out in one of the conversations: “How might one tell a maker from a non-maker?” The idea of a “non-maker” immediately felt a bit ridiculous. Since we’ve already established that “maker” is a mindset, it’s pretty clear that shifting out of that mindset is going to land us in a “non-maker” territory. But when we are in the maker mindset, what are the traits and characteristics that might stand out?

After talking to a few folks about this, a rough list started to emerge. It’s still not quite right, but I thought I would share it early for your perusal.

So far, I have four traits that seem to resonate whenever I talk about a “maker mindset” to others. These traits are option-seeking, craving creative friction, zagging, and optimism.

🔢 Option-seeking

When in the maker mindset, we tend to seek more options. If offered a single way to solve a problem, no matter how simple and elegant, a maker in me will perceive it with skepticism. There’s something about optionality and preserving the agency to choose these options that is highly important to a maker mindset. The more knobs, the merrier. The more choices, the more exciting. This is plainly in conflict with the mainstream theories of user experience for the common consumers. When I am in a non-maker mindset, I want a simple single solution that gets the job done. When wearing a maker hat, I will steer away from it.

This might be one of the reasons why open source projects and modular solutions are attractive to makers. Being able to pick and choose whichever pieces I want and combine them in whatever way I want – and have an option to change mind – are a big part of the whole maker experience.

💪 Craving creative friction

Very related, a maker mindset frowns on well-solved problems. We will rarely find makers tinkering with obvious or fully understood problem spaces. For makers,  things have to be difficult and challenging to be attractive. Too much polish is a bit of a letdown – it means that someone already solved all the fun problems. Maker mindset cherishes creative friction – the presence of a challenge in the process of making is what stirs creative juices.

This is why makers don’t mind messing with stuff that isn’t yet fully baked. One of my colleagues put it as “makers are in it for the problems, not solutions.” This is a bit too blunt, but I can’t disagree with the sentiment. For makers, it’s about the journey, though the tantalizing promise of a destination definitely helps.

🔀 Zagging

Makers love to zag when everyone zigs. When in the maker mindset, we tend to look for opportunities that are odd-shaped compared to what everyone else is seeking.

Makers rejoice when it looks like they’re doing something weird.  Being outside of the curve means that there’s a chance we’re ahead of it. Makers wholeheartedly take this chance. Even if zagging doesn’t pay off, the thrill of exploring the wilderness is a powerful force that animates makers.

When everyone is making an AI-powered chatbot, makers are playing with meta-reasoning and autonomous agents. When everyone finally catches onto the agents, makers move on to something else.

☀️ Optimism

This one I am least certain about. It definitely rings true, but I don’t know if the word “optimism” captures the gist. When in the maker mindset, we are driven by a belief that our actions will lead to some outsized outcomes. Somehow, somewhere, we will hit that exponential curve, and things will truly get out of control. There’s a sense of “it’s definitely not working now, but just wait and see” that is like fresh air for makers.

Many makers are techno-optimists, who – often implicitly – believe that technology will solve all problems and do more good than evil over the long run. After all, making something often means creating new technology – be it physical, organizational, or social. And definitely, most makers believe that making something is better than not making it. Making is art, and all maker’s art has purpose, animated by often completely unfounded confidence in better outcomes.

📐 Designing for makers

Despite this list being so unkempt, we can start gleaning some interesting insights about designing user experiences that attract makers.

Makers flip the script on the conventional wisdom of delivering polished, simple experiences to users. Steve Krug’s “Don’t make me think” turns into “Give me an interesting puzzle!” and sometimes into “Ooh, this mess of a product looks perfect for my project”. For makers, rough edges signal exciting possibilities. I am still learning what this all means, but it’s starting to feel that product design for makers is dramatically different from design for users not in the maker mindset.

If such a difference does indeed exist, it’s interesting to consider how a product might be perceived by the same person, but from different mindsets. And perhaps even more granularly: some tools I want to have lots of knobs and options and rough/unexplored edges, and some of them I just want to work, even when I am in the maker mindset.

It seems overwhelming – and likely foolhardy – to establish a precise taxonomy here. The only recipe I know is to have the maker’s intuition. To design for makers, one has to have accumulated a lot of experience of being a maker. There doesn’t seem to be any way around that.

The engine and the car

The whole large language model space is brand new, and there are lots of folks trying to make sense of it. If you’re one of those folks, here’s an analogy that might come handy.

Any gasoline-powered car has an engine. This engine is typically something we refer to as a “V8” or “an inline 4” or sometimes even a “Wankel Rotary Engine”. Engines are super-cool. There are many engine geeks out there – so many that they warrant a video game written for them.

However, engines aren’t cars. Cars are much more than their engines. Though engines are definitely at the heart of every engine, cars have many additional systems around them: fuel, electrical, steering, etc. Not to mention safety features to protect the passengers and the driver, and a whole set of comforts that we enjoy in a modern car. Pressing a button to roll down a window is not something that is done by the engine, but it’s definitely part of the whole car experience.

When we talk about this generation of AI systems, we typically talk about large language models (LLMs). In our analogies, LLMs are like engines. They are amazing! They are able to generate text by making inferences from the massive parametric memory accrued through training over a massive corpus of information.

However, they aren’t cars. One of the most common mistakes that I see being made is confusing engines (LLMs) with cars (LLM-based products). This is so common that even people who work on those products sometimes miss the distinction.

When I talk to the users of the PaLM API, I see this confusion show up frequently in this manner: developers want to reproduce results from the LLM-based products like Bard or ChatGPT . When they try to get the same results from the API, they are disappointed that they don’t match. Factuality is lacking, API can’t go to the internet and fetch an article, etc. 

In doing so, they confuse the engine with the car: the API, which offers access to the model, is not the same as the products built with it. With an LLM API, we have a big-block V8. To make it go down the road, we still need to build the car around it.

 To build on this analogy, we live in the early age of cars: the engines still figure prominently in the appearance and daily experience of a vehicle. We still have to turn the crank to start the car, oil the engine frequently, and be savvy enough to fix minor problems that will definitely arise.

As our cars become more refined, the engines get relegated into a well-insulated compartment. Users of cars rarely see them or operate on them directly.

This is already happening with LLM-based products. Very few current offerings that you might encounter in public use are LLMs that are directly exposed to the user.

So, when you use a chat-based system, please be aware that this is a car, not the engine. It’s a tangle of various AI patterns that are carefully orchestrated to work as one coherent product. There is likely a reasoning pattern at the front, which relies on an LLM to understand the question and find the right tool to answer it. There is likely a growing collection of such tools – each an AI pattern in itself. There are likely some bits for making sure the results are factual, grounded in sources, and safe.

As the LLM products become more refined, the actual value niches for LLMs become more and more recognizable. Instead of thinking of one large LLM that does everything, we might be seeing specialization: LLMs that are purpose-designed for reasoning, narration, classification, code completion, etc. Each might not be super-interesting in itself, but make a lot of sense in the overall car of an LLM-based product.

Perhaps unsurprisingly, the next generation of cars might not even have the same kind of engine. While the window control buttons and the steering systems remain the same, the lofty gasoline engines are being replaced with electric motors that fit into a fraction of space. The car experience remains more or less the same (aside from the annoying/exhilarating engine noise), but the source of locomotion changes entirely.

It is possible that something like this will happen with LLMs and LLM-based products as well. The new open space that was created by LLMs will be reshaped – perhaps multiple times! – as we discover how the actual products are used. 

Steady winds, doldrums, and hurricanes

It just so happened that this year, many of my friends and colleagues ended up looking for new opportunities, and in our conversations, I ended up shaping this metaphor. As most metaphors, it’s not perfect, but hopefully, will stir some new insights for you.

We kept trying to describe the energy within organizations and the animating forces that move them. These forces can make our lives inside these organizations a delight – or a complete and utter misery. It seemed like a good idea to understand how these forces might influence us and find ways to sense these forces early. Preferably, even before committing to join a new team.

The idea of presenting these forces as winds seemed rather generative. If we look at the innovation S-curve, we can spot three different kinds: steady, doldrums, and hurricanes. They don’t exactly match the stages I outlined back in the original article. Instead, these winds follow the angle of the S-curve slope.

⛵The steady winds

Steady winds are consistent. We can feel them going in one direction and they change infrequently. Apparently sailors love them, because they provide a predictable way to navigate. Even if it’s not a tailwind, a steady wind can be harnessed through tacking.

Similarly, organizations that are in the upslope of their development tend to have a relatively consistent animating force that feels like a steady wind. Usually, there’s some big idea, some intention, and a group of highly-motivated individuals who set the direction of this wind.

We can feel it as soon as we step into an organization. It usually appears as the ambition of the  charismatic leader/founder, who knows exactly what they want and is doing everything they can to make it possible. More rarely, it might also appear as a set of ideals that depict some future state of the world – and this team has the fire (and funding) to bring it forth.

Steady winds aren’t always great. Sometimes, a steady wind’s direction is simply incompatible with where we want to go. It might trigger aversion in us, or be in discord with our own principles. The leader might be charismatic, yet have character quirks we deem appalling. The big idea might indeed be big, but no matter how much we try to suspend disbelief, we keep finding it laughable.

At the same time, steady winds bring clarity. They give us a very good idea of what this team is about and where they are going. These folks are going someplace. It’s on us to choose to go there with them.

When considering a new team and sensing a steady wind that moves it, ask yourself: is this wind aligned with what I myself want to do? Does it stir fire in my belly? At the very least, can I tack into this wind in a way that moves me where I want to go? And of course: am I at the place where I want to go on an adventure?

Because joining steady-wind teams definitely brings adventure. It might be glorious and awesome, or it might be like the Donner party, with all the fixin’s of freezing to death, scurvy, and/or dysentery. Only time will tell.

If the wind is favorable and adventure is what you seek, such a team might be a good fit.

⛳ The doldrums

Prior to the invention of motors, doldrums were a terrifying thing for sailors. Doldrums meant that to go anywhere, we have to break out our oars and turn our own sweat into motion. There is no wind to help us go anywhere.

Organizations tend to experience doldrums at the top of the S-curve. Once the niche is fully explored and the product or service is optimized to fit it exactly, it is really not clear where to go next. All successful products end up experiencing this. We can see this as fewer interesting changes in them, and a deluge of incremental improvements that may sound exciting, but don’t actually add up to anything like the stuff the organizations used to produce at the upslope.

To get anything done in this organization requires some form of consensus. There are usually processes. Approvals. Reviews. Endless, exhausting discussions. When in doldrums, there’s a prevailing sense of powerlessness, often accompanied by a weird combination of comfort and toil. Everything is hard, but at least it’s exactly the same as yesterday.

Leaders who used to produce the steady wind at the upslope typically leave when they encounter the doldrums. We won/lost. Why stay? Instead, they are replaced by sailors. These leaders concentrate more on preserving what was accumulated so far. Risk is frowned upon. 

It’s not like nothing gets done in organizations stuck in doldrums. There’s always activity, and an appearance of movement. To create this appearance, there’s a syndrome of chronic bigness: every new initiative is bigger than the previous one, ever more bombastically described and painted in brighter colors. Underneath is the same dull surface of still water.

Doldrums aren’t necessarily a red flag for joining. If what you’re looking for is the steady stillness of boring, yet never-ending work, that might just be the place. Large bureaucracies like government agencies and corporate giants have large organizational swaths that live in the doldrums – and necessarily so. Not everything needs to be an adventure. Sometimes, the slow and steady beat of the oars is the only thing that keeps the grand ship inching forward.

However, if you’re seeking something to fill your sails, please keep walking. Committing to a doldrums team will suck the soul out of you and is not worth it.

🌀 The hurricane

The final part of our story is hurricanes. Sailors caught in storms just hang on to their life, trying to survive and keep the ship afloat.

Similarly, organizations find themselves in turbulent waters. This typically happens on the downslope of the innovation S-curve, when the quiet ride through the doldrums is eventually replaced by contact with reality.

In the hurricane, there’s lots of wind. It’s blowing in all directions. To continue our metaphor, the wind is the animating force that is usually created by organization’s leaders and their intentions. In the hurricane, this intention is chaotic and unpredictable. And it’s usually reactive,  spurred by some external threat.

The downslope of the S-curve isn’t fun. The collective anxiety of leaders who got used to the doldrums creates a vicious cycle, exacerbating the situation further. The overall direction is unclear, but not for the lack of effort. There’s lots of movement, and lots of force, all going in circles.

On very, very rare occasions, a new leader emerges and manages to set the steady wind, bringing the team out of chaos. I have seen it happen, but haven’t experienced it myself. 

Unless you’re a total glutton for punishment or have a severe savior complex itch, it is difficult to recommend joining an organization in the hurricane. The trouble is, it’s often hard to tell. It is in nobody’s interest to reveal the true state of disorder to the candidates. So the hurricane-embattled team might appear as either doldrums or steady winds, depending on who you ask.

One of my colleagues recommended this approach: find someone on the inside. Someone who might still be there or left recently. Ask them candidly: “is this a sh*t show?” Watch their reaction and prod a bit. Look for stories that sound like aimless grasping for straws and high anxiety among the team’s leaders. Those are the telltale signs of the hurricane.

Diving into unpredictability

My previous essay on the topic of unpredictability generated a few insightful comments from my colleagues and friends. One of them led to this vignette.

It is very tempting to imagine that some people are just generally less susceptible to the discomfort of unpredictability than others. It might even feel like coming up with a way to gauge one’s ability to thrive in unpredictable situations would be a useful tool.

My intuition is that this stance needs a bit more nuance. As humans, we all abhor unpredictability. We rarely actually “thrive” in it, at least over the long run. The metaphor that comes to mind is diving.

Some people are great divers. They can spend a significant amount of time under water. They can go deep and explore the parts of the seabed inaccessible to anyone  else. At the same time, nobody would claim that great divers can actually live in the depths of the sea. We all need to come up for air.

In this metaphor, unpredictability is water. If we stay in it for too long, we drown. I see the desire for predictability – or homeostasis – as a gravity-like force that animates all of us. It isn’t something we can completely detach from – though stoics and buddhists try. Just like air that we need to breathe, predictability is something that is essential for nourishing our minds. Our minds are predictive systems. Unpredictability is anti-mind.

Great divers – those who can endure unpredictability better than others – are those who invest generously into techniques and strategies that enable them to stay in the deep longer and even enjoy it. However, prolonged exposure to it will still take the toll, and the need to come up for air will always win over.

Diving into predictability is hard work. Just like with any good diver, if they are making it look effortless, we can bet that a lot of effort was put in before. And just like with any good diver, the “true pirates” who appear as thriving in unpredictability are nearly always those with the decades of practice, with all the blood, sweat, tears, and scars such a practice entails. One of the foundational elements of this practice is finding a way back to the fresh air of a predictable environment.

Sometimes you gotta hit the wall

I’ve probably written about this a few years back, but I still find this mantra useful and worth repeating. It applies to the situations where we’re stuck but we don’t know that we’re stuck – not yet.

When we’re in this state, we have a sense that we’re still moving forward, and we’re making all the right moves. We get upset when our friends or colleagues cautiously share with us that we might be spinning our wheels. Yeah, there’s some loss of traction, but if we just keep going, we will figure this thing out. Just one more push.

Particularly for technologists and other admirers of modernist thinking, the likelihood of becoming stuck in this way somewhere along our careers is pretty high. The idea that if we know what we’re doing and we’re doing everything right, then things should work out according to our plans – it’s just so damn seductive.

We can last quite a bit of time in this purgatory of delusion. There are just so many options to choose from. It’s the environment around us that is all wrong. Someone is actively conspiring against us. There are some indicators that show clearly that we’re still moving forth as planned. The more clever and quick-thinking we are, the more likely we are to come up with a story that keeps us stuck.

Inevitably, there’s a moment when it all comes apart. We finally hit the wall. We’re in shock,  feeling injured by the cruel reality and betrayed by it. But – it is only when we hit that wall do we get the chance for self-reflection. There’s an opportunity, when the shell of self-delusion is cracked, to actually gain some clarity. We might remember our colleagues’ gentle hints and worried faces, the early signs of stuckness we’ve chosen to ignore, and the now-obviously illustory stories we’ve told ourselves.

Should we experience it, this moment is a significant milestone. It allows us to create a little space between reality and the stories we tell ourselves. It allows us to hold our stories as objects instead of being subject to them. Experienced once, it’s a perspective that can be temporarily lost, but never fully forgotten. Next time the allure of modernism tempts us, we might still feel the pull – but think twice about answering the call. Once we’ve hit that wall, we’ve learned that “knowing what we’re doing” and “doing everything right” are just stories we tell ourselves, and they have little to nothing to do with reality.

The somewhat sad part is that this lesson can not be taught. No amount of explanation or teaching will bring one closer to the precious insight without the painful experiential part. This particular bit of wisdom can only be gained by face planting into the unyielding, uncaring reality at full speed. Sometimes you just gotta hit the wall.