Schemish

I’ve been spending most of my hobby time playing with reasoning boxes and it’s been crazy fun. I need to write properly about my explorations later, but so far, I’ve convinced an LLM to brainstorm ideas and then critique itself in a diverge-converge exercise, evaluate Cynefin space of a situation, and even get some Humean reasoning going – albeit with mixed results on the last one.

One pattern that I’ve leaned on consistently is asking an LLM to produce output in JSON. There are several advantages to that. First, I get the output that I can then process outside of an LLM. Second, JSON surprisingly acts as a useful set of reasoning rails for LLMs itself. It kind of makes sense – since LLMs are predictive devices, laying out a structure of prediction helps organize and guide the prediction itself. In other words, JSON is not just useful for processing the output. It also helps with framing the process of text completion.

Of course, now that we’re asking an LLM to return JSON, the question arises: how do we inform it of the structure in which this JSON needs to be?

The first obvious answer is that we already have a way to describe the structure of JSON output. It’s called JSON Schema and it’s in broad use across the industry. Turns out, modern LLMs will happily consume JSON Schema. All you have to do is, somewhere in your prompt, tell them something along the lines of (you can see an example here):

Respond in valid JSON that matches the following JSON schema:
<insert your JSON schema here>

The output will (well, with 95% probability, as it usually goes with LLMs) dutifully follow the specified schema.

Unfortunately for us, JSON Schema is a bit chunky. Designed for precision and processing, it’s verbose and can quickly blow through our token budget if we’re not careful. It is also not exactly a human-readable format, making prompt hacking a bit of a chore.

On the other hand, describing JSON in English is also a bit too weird: “the output must be valid JSON containing an object with foo and bar as strings, blah blah blah” is something that LLMs will tolerate, but who has the time for typing it all in prose?

So I ended up using this weird pidgin that I named Schemish. It’s not really a JSON Schema, but not plain English either. Instead, it’s a mock of JSON output, with hints for the output provided as values. Kind of like this:

Respond in valid JSON of the following structure:
{
  "context": "A few sentences describing the context in which the question was asked",
  "subjects": [
    {
      "name": "Name of an subject mentioned or implied in the question",
      "assumption": "An assumption that you made when identifying this subject",
      "critique": "How might this assumption be wrong?",
      "question": "A question that could be asked to test the assumption",
    }
  ],
  "objects": [
    {
      "name": "Name of an object mentioned or implied in the question",
      "assumption": "An assumption that you made when discerning this object",
      "critique": "How might this assumption be wrong?",
      "question": "A question that could be asked to test the assumption",
    }
  ],
}

As you can see, I am specifying the structure of the output, but instead of relying on JSON schema syntax, I am simply mocking it out – just as I would describe JSON output in a sketch for another person.

If we want to go even more compact, we could go for YAML:

Respond in valid YAML of the following structure:
---
context: "A few sentences describing the context in which the question was asked",
  subjects:
    - name: "Name of an subject mentioned or implied in the question"
      assumption: "An assumption that you made when identifying this subject"
      critique: "How might this assumption be wrong?"
      question": "A question that could be asked to test the assumption"
  objects: 
    - name: "Name of an object mentioned or implied in the question"
      assumption: "An assumption that you made when discerning this object"
      critique: "How might this assumption be wrong?"
      question": "A question that could be asked to test the assumption"

Now, for an even cooler trick. You might have noticed that the structure has somewhat repeating elements in it. JSON Schema has references and other tools to deal with redundant declarations. Turns out, Schemish can do the same thing. Just use what you would usually do when sketching out a structure for yourself or your colleagues – leave a hand-wavy comment:

Respond in valid YAML of the following structure:
---
context: "A few sentences describing the context in which the question was asked",
  subjects:
    - name: "Name of an subject mentioned or implied in the question"
      assumption: "An assumption that you made when identifying this subject"
      critique: "How might this assumption be wrong?"
      question": "A question that could be asked to test the assumption"
  objects: 
    # same structure as `subjects`

What’s neat about Schemish is that it’s clearly derivable from a JSON Schema. Though I haven’t actually written the code, it’s fairly obvious that JSON Schema description fields as well as type and optionality can be expressed as English (or some shorthand of it) in the value of the Schemish structure.

This means that I could potentially connect a piece of machinery that speaks JSON Schema to an LLM via Schemish, with Schemish translator shepherding the schema from precise, mechanical code-land to the squishy, funky LLM-land – and then picking up the output, produced by the LLM to usher it back to the code-land by validating it with JSON Schema.

This Schemish idea seems fairly useful and I’ve seen similar techniques pop up in the AI tooling sphere. If you like it, please do something with it. I’ll probably write a simple wrapper for Schemish, too.