Category: What Dimitri Learned

After writing a bit about principles and tensions, I thought I’d switch tracks and go with something slightly different. Philosophy meets strategy in this story about how we see ourselves in relation to the world around us.
I’d like to start by making a distinction between two different ways in which we place ourselves in our environment. The first one we will call the ODA stance and we’ll name the second the OODA stance. I’ll explain the names shortly.

📈 The ODA stance

The ODA stance  is fairly common and broadly applied. In this stance, we see ourselves situated in our environment and able to perceive our environment directly. When we act on our environment, we observe changes, make decisions, and act again. This cycle – let’s call it the ODA loop (Observe, Decide, Act) – is our way to navigate the environment.

In many situations, the ODA loop surprisingly well. We observe customer behavior, we decide on the change we need to make to our product to adapt to this behavior, and then we make that change. Because we can see the environment around us (the Observe part), we can make plans, create OKRs, make Gantt charts, and proceed forth with confidence.

When in the ODA stance, it is all about making the right decisions. To make sure our actions yield the outcomes that we desire, our decisions have to be correct. If our organization’s narratives rotate around effectiveness of decision-making, it’s very likely that we’re in the ODA stance.

Because of that, organizations in the ODA stance are usually program-heavy. They trend toward rails and cranks. Rails are processes and practices that ensure that only correct decisions are made. Cranks are tools that to make sure that following the right processes and practices is as easy as turning the crank. When I am in the ODA stance, I have this firm belief that if I put down enough rails and create ergonomic enough cranks, I can solve any problem at scale.

The ODA stance will start to feel insufficient and missing ingredients when we get blindsided by an unanticipated change. When our plans and projections get swept away by a disruptive innovation or a newly discovered challenge, we begin to struggle.

Something feels off. It often feels like someone or something is deceiving us. Organizations tend to react to this feeling in various ways. Sometimes, the guilty parties are identified and dismissed. Sometimes, the external entities are deemed antagonistic and troops are rallied to defend the fort. Very often, the disruption is dismissed as a blip, an outlier that’s not worth paying attention to. In most cases, the whole ordeal is forgotten. Until it happens again.

Technological innovation really messes with the ODA stance. Every new change brings chaos, the uncomfortable time when things stop making sense. The best thing we can do, it seems, is to hang on for our dear lives and wait out the turbulence. Unfortunately, the frequency of storms only seems to increase.

Unsurprisingly, the ODA stance organizations experiencing the storms intuitively lean into even more rails and cranks. They strive to create better decision-making processes, document practices more precisely, and enforce more stringently that they are followed.

🌀The OODA stance

After getting bruised by the turbulence a few too many times, a new stance tends to emerge. Speaking from experience, we rarely grasp the idea and adopt this stance immediately. It takes actual experiential scars to arrive at it.

The OODA stance discards the idea of perceiving the environment directly. Instead, we insert this concept of a mental model between our observations and our decisions. This mental model is what, back in the ODA stance, we mistook for the environment.

In this stance, observations stream into our model, and continuously update this model, causing us to orient – to change our understanding of our environment. When we see an observation, we orient, then make decisions, then act. Helpfully, a fellow named John Boyd already coined the term for this cycle: the OODA loop.

The addition of an extra “O” to the ODA loop from the previous stance implies a more humble posture toward the environment. We do not know what the environment looks like. We may never have a full grasp of it. We only have observations as clues. We try to cobble these observations as best we can into a model of the environment, and then try to guess what the environment will do next by using our model to make these predictions.

It’s a productive kind of humility. Instead of beginner’s humility, where we are too timid to make predictions because we are alarmed by our own ignorance, with productive humility we admit that the environment is likely too complex for us to grok it fully, yet we have tools to navigate this complexity.

Organizations operating in the OODA stance focus on bettering their mental models. They recognize that the key to making good decisions stems from their ability to maintain mental models that result in good predictions.
This recognition usually leads to three types of investments:

1️⃣ Sensing. The OODA stance craves observation. Productive humility dictates that external sensing is paramount. We put in a lot of effort to ensure that information from outside of the organization (and the inside!) flows as freely as as abundantly as possible. Sensing is not a responsibility limited to a small group of folks with the right job titles, but a daily practice for everyone.

Organizations with the OODA stance encourage the culture of reading and writing, of seeking out novel trends. There are spaces for robust conversations about these trends and tools to plug into the flow easily. 

2️⃣ Flexibility. When we’re in the OODA stance, we want to ensure that we’re able to orient flexibly. No matter how meticulously maintained and useful so far, mental models are only based on past observations. Any new disconfirming evidence must be treated as a signal that the model might need an update. 

Organizations in the OODA stance have a healthy fear of seeing it all wrong. They build tripwires and warning lights into their processes, and treat challenges to established mental models as gifts, rather than annoyances. Coming from this organization’s leader, “Prove me wrong” is not a threat, but a plea.

In contrast with the ODA-stance organizations, it’s not the speed of the decision-making, but rather the agility of adjusting course that is valued and developed.

3️⃣ Shared mental model space. Organizations that favor the OODA stance nurture shared mental models. A funny thing: when we adopt the OODA stance, we recognize that everyone has their own mental model that they use to make predictions. Lived experiences, culture and upbringing, all contribute to a massive kaleidoscope of mental models of individuals across the organization. When looking at the same environment, we are seeing slightly different things, often without knowing it.

Contemplating this notion can feel overwhelming. The whole post-modernist movement might have risen out of that. To get past the “is there really a real reality?” line of inquiry, we lean into productive humility. It is clear that a collection of individuals of greater diversity of perspectives will likely have parts of a larger mental model amongst them. Everyone has a piece of the puzzle.

The trick is to assemble this puzzle. In the OODA stance, we expressly cultivate spaces where people can share their experiences, and have confidence that their perspective will be listened to and incorporated into the bigger picture. Collectively, we learn how to overcome the discomfort of others seeing something entirely different from what is plainly obvious to us.

☯️ Compare and contrast

Both stances have their utility. The OODA stance takes a lot more self-discipline and accumulated experience – the scars! – to acquire and hold. Because of that, the ODA stance is the default choice for most organizations.

Particularly in environments that are very familiar, don’t change very much, or change in monotone ways (like seasons), the ODA stance can work great. A good way to think of it is that when we don’t need to Orient ourselves a lot within an environment, we can simply omit the middle “O” from the OODA loop.

The ODA stance brings industrial scale. I might even argue that one can’t have industrial scale without leaning toward the ODA stance. We can only contemplate large-scale endeavors only when our model of environment is so sound and well-embedded across the organization that we don’t even think about it. To make a well-functioning bureaucracy, one needs a well-established system of governance.

On the other hand, in novel environments and environments that change rapidly, where the existing mental models keep failing to predict useful outcomes, the OODA stance becomes necessary. The ODA stance is simply blind to all the novelty, experiencing it as utter chaos. This is when it becomes necessary to decouple our mental models from the environment – and embrace the weird and wild ride of the OODA loop.

It seems pretty obvious that there is some sort of cyclical nature to technological progress. There seems to be a rhythm to how things happen, and many words were written to discern the structure of the cycle. My introduction to the cycle came by way of Tim Wu’s The Master Switch. In that book, the author proposed a cyclical rhythm of open and closed ecosystems alternating as new innovations emerge. Echoes of similar dynamics are found in Crossing the Chasm, The Innovator’s Dilemma, and other books. It’s like everyone is pointing at the elephant and trying to glimpse the full shape of it.

A tiny part of this elephant is the rise of makers. The rise of makers is the time in this cycle of progress when those who tinker and prototype emerge as the best candidates for finding the Next Big Thing. 

Every story of a tech giant’s humble beginning in a garage is the story of makers. My assertion is that the reason why we hear fewer of these stories in some periods of time than others is not because we run out of bright, entrepreneurial minds and certainly not because we run out of garages. Rather, the prominence of makers is a function of the cycle of technological progress.

Put differently, makers’ contributions result in significant outcomes in one part of the cycle and shift to the margins otherwise.

📻 Who are makers?

Perhaps I could back up a bit and start by describing what I mean by “makers”.

Makers are tinkerers and prototypers who are engaging with a new technology not because they have to as part of their job, but because they find it irresistibly interesting and fun. I like to say that developers are “9-to-5-ers” and makers are “9pm-to-early-morning-ers”.  This might not be an entirely accurate description, but it captures the spirit. It also hints at the key property of a maker: it’s not a kind of person. It’s a mindset. The same person who puts in their work hours at their day job becomes a maker extraordinaire in the evening or weekends.

Makers make stuff. They aren’t here for entertainment, fame, or other promise of future boon. They get their hands dirty. They build stuff. In the language of Crossing the Chasm, they are the kind of early adopter who doesn’t just adopt the tech. They make new things with it.

Makers play with technology, rather than apply it to achieve business goals. They delight in finding weird quirks and twists, the same way gamers love finding glitches that lead to a speedrun. They find all of your design flaws and unintended side effects – and turn them into a feature. The whole process is messy and often in the “life, uh… finds a way” manner – what makers make may look very different from the technologists’ intended range of use.

Makers write crappy code and wire things on breadboards. They rarely care about the future enterprise strength of their project. They explore ideas non-linearly, random-walking. They scrap half-finished projects and repurposing them into the new ones. All this contributes to the seeming disarray of the maker scene. A good sign of a maker project are fix-forward collaboration practices. 

Makers are here to discover something new, to bravely explore. They crave being first to uncover some way to make technology do a thing that nobody else had seen before. Makers become increasingly more disinterested with a particular technology as it matures and becomes polished. Polish and reliability mean that the tech has become mainstream – and thus, less likely to yield a “holy crap!” moment.

Being a maker means being in constant search of that moment. When the thing finally works and goes viral on Twitter, and investors come knocking – it’s a maker’s dream come true. Often, it’s also the end of a maker’s journey. Once the new big thing is found, makers shift to become businesspeople. The fun hobby project grows into a rapidly growing team around the newfound thing. Not all makers choose that path. After all, the thrill of exploration does get replaced by the mundane concerns of running the business. However, those who do, they wield power to reshape and create industries.

… At least, when the conditions are right.

🧫 Conditions for makers’ rise

When a novel technological capability moves forth to acquire an interface, and a broader audience begins to interact with it, there’s a question that hangs in the air: “What is this thing actually good for?” 

This is the value question. Whoever answers this question first gains a temporary advantage: until everyone else also figured it out, they can seize the opportunity to acquire this value.

Makers arrive at the scene right about then. They start poking at the interface and make things with it. The amount of power makers will have at this point depends on whether or not they can answer the value question sooner than anyone else.

What are the properties of the technological capability (and the environment it is introduced into) that put makers in the driver’s seat?

I’ve been thinking about this a bit, and I keep coming back to these three: 🔒 access to technology, 🏞️ openness of space, and 🚀 iteration velocity.

All these properties interact with each other, so they aren’t exactly orthogonal.

🔓Access to technology is both the property of new technological capability and its environment. It is probably best measured in the number of makers who could practically start using the capability.

It is the property of the capability because technologists who introduce the interface can choose to make it more or less accessible. It is also the property of the environment, because other, adjacent technological advances may make the capability more accessible than before.

A good example of the capability becoming more accessible due to shifts in environment is the introduction of widely available high-speed Internet. Without any discernible change in how the Web worked (no change in capability itself), the increase in bandwidth created new opportunities for makers to spur what is known as “Web 2.0”.

🏞️ The openness of space is reflected by the number of new opportunities created by the introduction of the technology. A good marker of a wide-open space is that a typical market-sizing exercise keeps collapsing into a fractal mess, creating more questions than answers. It’s not just one thing that suddenly becomes possible, but a whole bunch of things – and there’s this feeling that we have only scratched the surface.

Open spaces favor makers, because they require random-walk and large quantities of participants to facilitate the ergodic exploration of the space. Well-established players tend to be subject to their embodied strategies, leaving them unaware of vast portions of the space, and thus highly vulnerable to your usual innovator’s dilemma and counter-positioning.

🚀 Finally, the iteration velocity is what gives makers the edge. Makers rule tight feedback loops. The shorter the lead times, the more likely makers will show up in the leaderboards. If something can be put together quickly, count the makers to stumble into a thing that actually works. Conversely, if the new technology requires lengthy supply chains and manufacturing processes, makers would play at best supporting roles.

Iteration velocity is also influenced by the level of stakes in the game. The higher are the stakes, the less velocity we’ll see. For example, we are unlikely to see makers playing with passenger airplanes or power grids. Those are the areas where the lead times are necessarily long, and no matter how exciting the innovation, makers won’t play a pivotal role in those spaces.

📈 Makers rising

There is no doubt in my mind that we’re experiencing another moment of makers rising to prominence. The galaxy-sized spaces opened by generative AI, its accessibility, and the velocity with which one could put together an impressive prototype – all are pointing at the notion that perhaps the next big thing will come from a tinkerer’s garage. My intuition is that we’re in an historic moment that we haven’t seen since the birth of the Internet – or perhaps even larger than that.

We are living in the age of makers rising. I am not sure how long it will last. After all, the prominence of makers is relatively short-lived, just a moment in the large story of the technological advance. But oh boy, is this moment significant.

I may have written about this before. It’s a trick that I’ve learned intuitively over the years, and I’ve found it tremendously helpful. It tends to work best in situations where the outcome is uncertain.

When adventuring forth into the unknown, we usually focus on the thing that we want to achieve. Even if the chances of it happening are slim, we set our eyes on the prize and commit to persevere through whatever challenges are thrown at us.

It is just before this point that I usually pause and spend a bit of time contemplating the gifts of failure. The question I ask myself is simple: “What good can come out of failing in this venture?” There are usually plenty of platitudes about learning from failure and all that wonderful stuff that is talked about incessantly in books about innovation. This is not what I am looking for.

When I ask that question, I am looking for tangible benefits that might arise as a result of the failure. Once we’ve reached the threshold where we definitely know we’ve been defeated – by circumstances, a worthy foe, or our own limitations – what are we left with?

Put differently, when all we have is to pick up the broken pieces of our failed enterprise, how quickly can we repurpose these pieces to start anew?

If we don’t contemplate this question earnestly, the answer is usually “not much”. In such cases, the collapse is dramatic and total. Very little of the effort put into the project is recoverable. If anything good comes of it, it’s sheer luck.

The stepping stones tactic is one way through which we can maximize this luck. In Greatness Cannot Be Planned, the authors argued that small increments just outside of the adjacent possible – the stepping stones! – are what ultimately feeds innovation and big discoveries.

By reflecting on the gifts of failure, we can sketch our path as a series of stepping stones that we create for ourselves. What are the stepping stones will we need to produce anyway as we walk toward our vision? Which ones can take on the life of their own, and serve us even after our initial project has failed? Through this lens, the product of our endeavor will no longer look like a monolithic thing. Instead, it will be a modular construct that can flexibly rearrange itself into something completely different, becoming highly resilient.

As another tactic more useful for smaller projects, I sometimes maintain a small portfolio of secondary benefits that could be promoted as primary. These benefits must be somewhat independent of each other, so that if the initial attempts prove fruitless, there’s another path forward that opens up by refocusing on another potentiality.

Both of these tactics arise from reflecting on the original question. When we accept that in highly uncertain, volatile environments, any exploration is much more likely to fail than to succeed, we stop seeing the target of our adventure as the mythical “end all be all”, and focus on accumulating opportunities and potentialities along the way.

I have been lucky enough to be part of the process of shaping several organizations, yet only uncovered this nugget of insight very recently. Very briefly, it helps to discern the forces of the broader environment that influence us without our awareness.

It is very rare that the leaders of a newly formed organization want it to behave exactly the same as the surrounding environment. If that is the case, we have a distinction without a difference, a label rather than an organization. Teams and organizations are created to do something (even if slightly) different from the rest of the environment. At the same time,  they are still in that environment and are influenced by the forces of this environment.

If I want to create a team that innovates daringly while situated within an organization that is designed for keeping an important chunk of infrastructure running. I can speak all I want about inspiring creativity and fearless questing for ideas to my team – but if I don’t account for the fact that most processes around me are designed and incentives are structured for a “keeping” mission, none of that may matter. Implacably, my attempts to encourage innovation will come out shaped like infrastructure improvement projects. It will feel like an invisible force being exerted on the newly born team,  veering it off our intended course.

More often than not – especially in larger organizations – such forces are invisible until we start deviating from their direction. These forces are meant to organize, to bring various people’s intentions into alignment in service of the mission. Only when we try to do something different do we start experiencing friction.

Two questions might help you start making these more visible.

The first question is “what is our gravity?” This will take a little bit of self-reflection. We rarely ask ourselves why our feet stick magically to the ground. Similarly, organizations have gravity-like forces that everyone is used to. They tend to hide in structural and cultural bits of the org. If you’d like, use my fourplex of constraints to investigate. Which ones of these do we need to counteract when deviating from the default organization configuration? OKRs are probably my favorite example here. No matter how much we try to overcome it, the language of OKRs inexorably drives us to short-term thinking. If we try to build an organization that thinks in a longer timeframe than the rest of its OKR-driven environment, we will have a difficult time placing ourselves within it. “Wait, these people don’t have OKRs?  Do they even do anything?”

The second question is “what is our water?”, echoing the parable presented by David Foster Wallace. What are the things that we take for granted to such a degree that we don’t even notice them anymore? How do these things influence how we think and act? Where are they out of alignment with our intentions? If our team has a strong engineering culture, we will naturally have biases that will filter out less “engineer-ey” ideas – and that might be alright, unless these ideas are essential for making progress toward our intended mission.

The answers to these two questions are nearly always tentative and wrong, at least initially. It is important to keep examining them as the time goes on.

A while back, one of my colleagues was reflecting on their first year on a team at which we both worked at the time: “At first, I thought the hardest part will be to get creativity and inspiration going. Surely, the problem was that we weren’t thinking cleverly enough. Oh, boy was I wrong. It’s exactly the opposite! Everyone here is incredibly smart and is bursting with creative energy. Now I believe that where we struggle the most is in applying all that energy in a way that leads to shared learning that persists.” My colleague’s guess was even more refined and nuanced  the next year.

Understanding what is stopping us from where we want to go is key to getting there. If we don’t stop to examine the forces that hold us in place, the best we can hope for is an external shock that jolts us out of our trap.

One of my colleagues quipped recently that instruction tuning might not be about teaching large language models (LLMs) what to do, but rather about reminding LLMs of the wealth of wisdom they already have at their foundation. Here’s a rather unkempt vignette on what wisdom I would like an LLM to recall.

I would like to help a model remember how to solve problems that are a bit more complex, and require some rigor in reasoning. We are well underway in our first generation of LLM reasoning explorations, and I am looking a bit behind the current horizon.

Here is my hypothesis: an LLM that is capable of reasoning about a broad variety of problems would be biased to structure its reasoning in a certain way. That is, when we ask it to “think step by step”, it will rely on some predefined structure of these steps while performing a text completion.

If this hypothesis is true, reasoning structures will be at the core of dramatically improving the reasoning ability of the models. Put differently, I anticipate that the next cycle of model evolution will be about exploring the space of reasoning structures and finding more and more capable local maximas of such structures. 

The future of instruction tuning might not be about the quantity or even quality of the data, but rather about the rigor of the overarching reasoning structure in the data sets. 

To make my “reasoning structure” concept a bit more clear, I tried to conjure one up relying on the problem understanding framework that I’ve created a while back. This is just one rough sketch, but it should give you some idea of what the heck I am talking about.

🎱 Guessing the problem class

To quickly recap, I posit in my framework that there are three classes of problems: solved, solvable, and unsolvable.

LLMs are already able to address solved problems with moderate success. The implicit code execution shows a straightforward way to approach them: once the problem is deemed as something that was already solved before, we ask an LLM to write the minimal bit of code to produce the solution. Crunching numbers, reversing words, making charts, and various similar kinds of problems fall in this class.

LLMs are making decent progress on tackling solvable problems. Solvable problems have no well-established solution, but we know what the solution looks like – also known as “the ground truth”. The presence of this ground truth is what enables reasoning boxes, and their iterative approach to finding the solution.

The yet untapped potential of LLMs is in making sense of unsolvable problems. Unsolvable problems are highly adaptive, and often appear to be solving us rather than the other way around. Things like relationships, organizations, politics, and ecosystems all fall into this class of problems – and usually, we don’t end up solving them. Instead, we navigate through them as best we can. A good way to picture an unsolvable problem is thinking of an open ocean. We can’t solve the high seas, but we can navigate them.

The key to navigating an unsolvable problem is in investing a bit of time into sensing the various forces inside of the problem that make it impossible to solve once and for all. I call this process “framing the problem”. A successful framing produces one or more solvable problems that when solved, help us navigate the unsolvable problem.

Let’s use this simple taxonomy as the foundation of a reasoning structure.

The structure will likely begin with making a guess at the class of the problem: are we looking at a solved, solvable, or unsolvable problem? I experimented a bit with asking LLMs about Cynefin, and was pleasantly surprised by the quality of these guesses. Perhaps this is the foundational wisdom of the model that my colleague was talking about? 

Based on the outcome of the initial guess, the structure will guide the reasoning to branch and engage with the problem differently based on its class.

🔨 Handling solved problems

For solved problems, an LLM would attempt to just answer it, or write a bit of code as I mentioned above – just as most question-answering LLM-based applications do today. Like so:

Question: What is the square root of 38495?
Response: The problem of determining a square root of a number is a solved problem. 

Here is a program that returns a square root of 38495:
```javascript
console.log(Math.sqrt(38495));
```
 The answer is the output of the program.

For this problem class, this is not as much as solving the problem as retrieving the solution from the depth of the model’s wisdom.

🧩 Actually solving problems

If the problem is solvable, the reasoning structure will assume an iterative stance with two interlinked lines of reasoning.

The first line of reasoning tries to answer the question of “Has the problem been solved?”. This is the actual process of solving the problem. 

The second line of reasoning tries to answer the question “Do I know everything I need to know to verify that the problem has been solved?”. This is the process of establishing the problem context, which determines the constraints that could be used to verify that the solution is correct. This context serves as the ground truth for the problem.

Establishing problem context is a problem in itself, and the reasoning structure will branch further into various techniques that could be effective. For example, the LLM might decide to write a suite of tests that, when run, will prove or disprove that the answer is correct. It could also opt to ask the user to provide feedback on the answer and learn from that feedback. 

It might even choose an “inner monologue” technique, where the ground truth is present only in a socratic sense, with the LLM constantly questioning its own assumptions and making progress by spotting inconsistencies in them. As long as the assumptions are visible, there is an opportunity to question and correct them.

Whatever the choice, the reasoning structure will need to presume that both solving and context-finding processes are iterative. The LLM will no longer attempt to solve the problem in one completion, but rather focus on making progress toward the solution in multiple iterations and prefer frequent contact with the ground truth. 

This stance is different from the typical “question-answer” one of the solved problem class. I’ve been able to get the current generation of models to do so with mixed results, and it feels like an important capability to develop, perhaps through instruction tuning.

The model will also need to be capable of keeping track of where in the reasoning structure it is. Given that solving a problem might take multiple completions, there needs to be something that helps the model orient in relation to the overall progress of solving the problem.

Question: Write a to-do list app
Response: Writing a to-do list app is a solvable problem, because it is very clear when the app is built and whether it allows viewing the to-do list, as well as adding and removing items from the list. 

What is the information am I missing to solve this problem? 

I need to ask the following questions: 
- Other than showing a to-do list, adding, and removing items from the to-do list, what other features are important to you? 
- Should this app support multiple lists or is just one list sufficient?
- On what device will you use this app?

Then, a few interactions later:

Question: Write a to-do list app
Context: This is a solvable problem, and we are currently in the process of determining requirements. Previous interactions established the following requirements: 
- app will support only one to-do list
- app will run on an iPhone
- app will allow viewing one to-do list
- it should be possible to add and remove items on the list
- it should be possible to sort items on the list. 
Response: <completion continues here>

Today, this tracking is primarily done by recording the history of the problem-solving interactions, like a chat transcript. Most chat models will accept the history of the previous messages and are tuned to treat it as guidance for the text completion. I can’t wait to see the new, more effective methods of retaining problem context be discovered.

For instance, a more sophisticated reasoning structure might guide a model to detect that the problem could be split into multiple sub-problems that could be solved more easily, in parallel or in sequence. In such cases, the problem context would map into multiple sub-problems and eventually reduce back to one, once the sub-problems are solved. For example:

Question: Write an app that allows me to create a to-do list
Response: Writing a to-do list app is a solvable problem, because it can be tested if the app:
- successfully starts
- allows viewing the to-do list
- supports adding and removing items from the list.

To simplify the problem, I will break it up into two sub-problems:
- the backend that provides the to-do list API for storing and retrieving the list
- the frontend that consumes this API.

The context for the backend sub-problem will be:
- write a backend for a to-do list app
<detailed backend spec goes here>
- write an API for the backend that supports three methods: retrieve the list, add a new item, and remove an existing item.
<detailed spec of the API goes here>

The context for the frontend sub-problem will be:
- write a Web app frontend for a to-do list app
- the fronted will support viewing to-do list on mobile devices, and specifically on Safari for iPhone 
<detailed frontend spec goes here>
- the frontend will integrate with the following API
<detailed spec of the API goes here> 

🌀 Framing unsolvable problems

When the LLM guesses the problem as unsolvable, the reasoning structure guides the process toward discerning the tangle of forces that creates unsolvability. This part of the process is less about finding solutions and more about building a reasonable dynamic model of the problem.

I am not yet sure how such a model could be represented efficiently as text, but it will likely start with a list of statements that form into a causal graph. For example, here’s how a model may respond to a complex question:

Question: Should I stay in college or start my own AI company?
Response: making a decision on whether to stay in college or start an AI company appears to be an unsolvable problem. There does not seem to be one right answer and it is hard to check the correctness of the answer.

To better understand the forces at play,  I will ask the following clarifying questions:
- What is difficult about making this decision?
- Why does it have to be an either-or decision?
- What holds you back from starting a company?
- What is so attractive about starting a company?
- Why is leaving college necessary?
- Why is staying in college important for you?

After a few rounds of questions and answers, the graph will begin to take shape, the LLM might identify a few leverage points: solvable problems within the larger unsolvable problem. 

Question: Should I stay in college or start my own AI company?
Context: This is a complex problem and we are currently in the process of building a model of the problem. Here are the edges of the model graph:
- AI is very exciting, and there are a lot of cool opportunities 
- If I wait, I will miss these opportunities
- To graduate successfully, I need to wait
- To build a company, I need a lot of time
- To build successful foundation for my life, I need to excel at school
- To excel at school, I need a lot of time
- Time is an extremely limited resource
<graph edges continue>

Response: Based on the graph I have so far, I believe that time might be a leverage point in this problem. I will recommend framing the original problem as that of optimizing for the most effective use of time and creating space for both college and company in the schedule.

Suggested problem statement: Optimize my schedule to enable both staying in college and starting my own AI company.

Once the framing emerges, the reasoning structure will follow the recipe for the solvable problem that popped out. While the original problem remains unsolved, there are now concrete steps that the user can take to navigate it.

The dynamic model that is created by the process of navigating an unsolvable problem is a valuable artifact. In fact, it probably needs to be retained. Depending on the success and failure of the framing that was produced, it will need to be updated and refined.

🩹 Detecting problem classification mistakes

Because the first step of the reasoning structure is just a guess, the structure will need to provide guidance to detect the mistake and shift the problem into a different class.

Some of these mistakes aren’t as bad as the others. For example, mistaking a solved problem as solvable or a solvable problem as unsolvable will both result in inefficient use of resources, but produce a satisfying answer.

There is another category of mistakes where the solution is produced, but isn’t valid, like when a solvable problem was diagnosed as solved, and the attempt to solve the problem results in the wrong answer. I wonder if some sort of “inner critic” reasoning might be necessary even for solved problems to mitigate that.

The mistake that is a bit more troublesome is when the unsolvable problem was guessed to be solvable. It will look like inability to pin down constraints, creating a sense of divergence, rather than convergence toward the objective. Detecting this might be challenging, because the reasoning will continue finding new solvable sub-problems, and appear to make the infinite-regress sort of progress. Here, the reasoning structure will need some sort of a trip wire: a way to ask the model to lift its pencil and say “buddy, you’re spiraling out of control, this might be an unsolvable problem”.

😎 Mission impossible

Part of this essay was an exercise of trying to write a sketch of a reasoning structure in its entirety. Now, you and I could look at it and wonder whether or not something like a reasoning structure might even be feasible to imbue into a model.

It definitely feels a bit daunting. However, if the concept seems intriguing to you, please give it a try. Maybe hack together something very simple to start small, and expand from there. Who knows, maybe through this inquiry, the full depth of wisdom of a large language model will be revealed to us.