Dimitri Glazkov

Four layersseven days beforeThere seems to be some layering rhythm to how software capabilities are…Development environment for rapid experimentation

DX Pattern · What Dimitri Learned
· 8 min read

Build a thing to build the thing2023When building new products, there’s always a weird tension between making something…I’ve been playing with various software development configurations that might enable rapid experimentation, and landed on this particular one. I am pretty sure there are even more effective ways, and I can’t wait to learn from you about them. This is what I have so far.

A quick disclaimer: this is not yet another “how to set up a repo” tutorial. It’s mostly a capture of my learnings. I will refer to a couple of such tutorials, though.

Stages of project forking2023I’ve been thinking recently about how to design an open source project,…To set things up. I was looking for a way to enable a small-ish team to enable development of rapid prototypes. That is, write something, see if it does what we need, test the waters, learn like crazy, and break no sweat if it doesn’t.

🧫 Ecosystem

No ship ums2023I’ve been thinking about the path that software dandelions take toward becoming…Doers, Thinkers, and Magicians2023I’ve been reflecting on my experiences of working with developers and developer…Chances to get it right2023This has been rolling around in my head for a while, and…The first question on my mind was that of the developer ecosystem. To unlock fertile learning through testing the waters, prototypes need to ship. They do not have to ship as polished products with firm SLAs, but they do need to reach some users who would be willing to mess with the prototypes, react to them, and provide feedback. To maximize the chances of serendipitous feedback, we must play in the most populous ecosystems of folks who like to mess with unpolished stuff. When choosing a place to learn, pick the rowdiest bazaar.

The developer funnel2021If I ever chatted with you about developer experience in person, I’ve…This requirement narrowed down the possible environments quite a bit. Looking at Stack Overflow survey results, the two ecosystems stood out as by far the most legible for the title: Javascript developers and Python developers. They form the fat head of the developer environment power curve. These are the places to play.

I then spent a bunch of time messing with both environments, and ended up deciding on the Javascript ecosystem. There were several reasons for that, not all of them entirely objective. Roughly, it all came down to two factors:

🧰 Toolchain

XPath, unleashed — coming to Internet Explorer 5+ HTML DOM near you2004While coding away in JavaScript, reshaping/augmenting your HTML code using DOM, have…GWT And Now Script#: The Agony of a Thousand Puppies2006Uh oh. Another reactive rant. Am I turning into a grumpy old…Layer gaps2023I’ve been writing a bit more code lately, and so you’ll notice…After picking the environment, I wasted a bunch of time resisting TypeScript. As a Javascript old-timer and a known build step grump, I really didn’t want to like it. But after getting over my hang ups, I must admit: TypeScript is basically the best thing that could ever happen to unlock rapid prototyping. As long as I know where the layer gaps are (hint: the missing runtime type support), it’s basically the perfect tool for the job. Especially with the way it is integrated into VSCode, TypeScript hovers at just the right altitude to help me write the code quickly and have high confidence in this code working on the first run.

Behavior over time graphing tool2022It was an interesting adventure. From the start, I wanted to go…Which brings me to the next increment in my journey. If we choose TypeScript, we must go with VSCode as the development surface. I am sure there are other cool editors out there (I hear you, vim/emacs fans!), but if we’re looking for something that fits TypeScript like a glove, there is simply no substitute. Combined with eslint and prettier, the VSCode support for TypeScript makes development an enjoyable experience.

So… Node, Web, TypeScript, VSCode. These are the choices that came out of my exploration. I briefly played with the various Node package managers, and concluded that npm is likely the thing to stick with. I love the idea behind pnpm and yarn is super-fun, but at least for me, I decided to go with what comes in the box with Node. Deno is cool, too – but as a newcomer, it simply doesn’t meet the “rowdiest bazaar” bar.

The choices made so far define the basic shape of the prototypes we will develop and the sketch of the development flow. The prototypes will be either shipped as Web apps or libraries/tools as npm packages. Every prototype will start as an npm package. It might have server-only code, client-only code, or a mix of both. Prototypes that look like tools and libraries will be published on npm.

#️⃣ Runtime versions and settings

I invested a bit of time deciding on versions and settings of TypeScript and Node. One key guiding principle I chose was “as close to the metal as possible”. TypeScript compiler is quite versatile and it can output to a variety of targets to satisfy the needs of even the most bizarre deployments. Given that we’re prototyping and writing new code, we don’t need to concern ourselves with the full breadth of deployment possibilities – and we certainly can be choosy about the version of the browser we expect to present our experiments.

With this leeway and the recognition that TypeScript is mostly an implementation of ECMAScript (the standard behind Javascript) plus type annotations, we can configure the TypeScript compiler to mostly remove type annotations.

For Node, I chose to go with v18.16, primarily because this is the version that introduced the real fetch implementation, which matches what modern Web browsers ship.

So, if we have Node 18 and the config of the TypeScript below, we should minimize the amount of new code introduced by the TypeScript compiler and maximize the client/server code compatibility.

{
    "lib": ["ES2022", "DOM"],
    "module": "NodeNext",
    "target": "ES2022"
}

Porcelains2023// my API which only supplies a well-formatted Request. const myCoolServiceRequest =…As an aside, there was a fun rabbit hole of a layer gap into which I fell while exploring this space. Turns out, Node TypeScript type annotations don’t have the declarations for the fetch implementation. So I ended up doing this funky thing with adding the “DOM” library to the TypeScript config. This worked better than I expected. As long as we remember that a) TypeScript types are not seen by the actual Javascript runtime and b) most of the actual DOM objects aren’t available in Node, one can get away with a lot of fun hacks. For example, we can run unit tests for client-side code on the server!

🏠 Repository configuration and layout

Composing graphs with Breadboard2023As you may remember, I am also very interested in the role…With versions and runtime configs squared away, I proceeded to fiddle with configuring the repository itself. I first started with the “let a thousand tiny repos bloom” idea, but then quickly shifted toward the Node monorepo. This choice might seem weird given the whole rapid prototyping emphasis. The big realization for me was that we want to encourage our prototypes to mingle: we want them to easily reuse each other’s bits. It is out of those dependencies that interesting insights emerge. We might spot a library or a tool in a chunk of code that every other prototype seems to rely on. We might recognize patterns that change how we think about the boundaries around prototypes and would need space to reshape them. With all prototypes being individual packages, the friction of dependency tracking will simply prevent that.

There are multitudes of ways in which one could bring up a TypeScript monorepo. I really liked this guide, or this setup that relies exclusively on the TypeScript compiler to track dependencies. Ultimately, I realized that I prefer to use separate build tools that track the dependency build graph, and invoke the compiler to do their bidding. This is the setup that Vercel’s Turborepo folks advocate, and this is the one I ended up choosing.

Any Node monorepo will loosely have this format: there will be a bunch of config files and other goop in the root of the repository, and then there will be a directory or two (usually called “packages” or “apps”) that contains directories for the individual packages.

Separating cargo from the train2023I’ve been puzzling over a problem that many engineering teams face and…My intuition is that to facilitate rapid prototyping, we need a convention that reflects the state of any package in the monorepo. For example, we could have two package-holding directories, one for “seeds” and one for “core”. In the “seeds” directory, we place packages that are early prototypes that we’re just playing around with. Once a package acquires dependencies and becomes useful for other prototypes, we graduate to the “core” directory.

Another useful convention when working with Node monorepos is that the npm package names are all scoped under the same npm organization and the name of that organization matches the name of the repo.

So for example, if our monorepo is named “awesome-crew-prototypes”, all packages are published under the “@awesome-crew-prototypes” npm organization. For example, a prototype for a library that does URL parsing will be published as “@awesome-crew-prototypes/url-parser”. This way, the fact that the “url-parser” is part of the “awesome-crew-prototypes” monorepo is reflected in its name.

🚀 Team practices

As the final challenge, I worked out the best practices for the team that might be working in this repository. This section is the least well-formed, since typically, the practices emerge organically from collaborating together and depend quite a bit on the mix of the people on the team.

Having said that, the following rules of thumb felt right right as the foundation for the practices:

Armed with all of these, a team that is eager to experiment should be able to run forward quickly and explore the problem space that they’ve chosen for themselves, and have fun along the way. Who knows, maybe I’ll actually set up one of these myself one day. And if I do, I’ll let you know how it goes.

Models and structured reasoningeight days afterOne of my colleagues quipped recently that instruction tuning might not be…I also quickly put together a template for the environment that I described in this post. It probably has bugs, but should give you a more concrete idea of the actual setup.