Boring Work; or, The Art Of Getting Things Done

There’s many (27) definitions of work. There’s only one real definition of busy work. I’m not certain how to define boring work. I can’t say I’d thought much about the concept at all, prior to reading Dan Luu’s recent article on metrics.

Side Note: Dan’s article is, as always, a fantastic read, and incredibly informative. I highly recommend you read it (if you end up skipping the more technical bits – at least read the final few paragraphs, they are highly insightful.)

The term boring work stuck with me. It’s “plugging boring technologies together and doing the obvious thing”. If you add the implied outcome, the definition becomes:

boring work : plugging boring technologies together, and doing the obvious thing, in order to rapidly produce results and create value

After all, producing results is the motivation of all work!

So this post will take a look at some of the boring work I’ve done in the past, and you can be the judge of just how boring it may be.

Test Automation

I was working on an application server that served responses as JSON (exciting, I know). The web framework we used had a community-built library for generating swagger 2.0 docs from the application code. This all sounds boring, I’m sure.

Now, swagger codegen often takes the form of “compile time” translation, either from swagger docs into client code or server code into swagger docs. In our case, we were happy to have the API routes generated from the application, but the response objects were complex enough and dynamic enough that generating them from compile-time class definitions was harder than manually specifying the swagger schemas.

The key point here is that swagger docs are only as reliable insofar as they match the server running in production. If you inadvertently change the application code and the response no longer matches the swagger, your client will be upset: we needed a way to verify that our routes were respecting the swagger specification – at runtime.

Of course, there’s already libraries for that! But there was no library to assert: “this controller method shouldn’t violate the spec”.

So I took the responsibility for pulling in a swagger parser, parsing our swagger file during unit testing, and providing a method for developers to create test cases like: “this json came from this route, make sure it satisfies swagger”.

response = get("/path/to/route")
assertSwagger(response.json(), SwaggerSpec("GET", "/path/to/route")

There’s no “high-availability lambda-based bitcoin network” involved here. There’s just some libraries, and a bit of wiring.

I never published the code as a library, because I figured it was boring enough that anyone who wanted the same thing probably already did what I did. But if everyone thinks that way, no one would ever publish their work, and we’re all duplicating this boring work. Something to consider.

Jira Housekeeping

Jira, every engineer’s favorite website /s

Ok, this one actually does use lambda. But the explanation could probably fit into one sentence. I’ll use a couple sentences instead…

Our team used Jira for tracking tasks. We also used github for source code management. We found it helpful to keep track of which tasks were in which release, as you can then go to a specific release and see what changed or you can search for a specific task and find out when it was released.

We want to make sure tasks are added to the next release when they are marked “DONE”. This might be easily captured by just setting the release when starting a task, but then tasks that aren’t done might be assigned to a release they aren’t actually a part of.

For this, we resorted to mapping the task “component” field to the “release” field. When a task is moved to “Done”, Jira is configured to send a webhook to our lambda. The lambda inspects the task and based on the component, it adds it to the appropriate release.

The component and release mappings were coarse-grained enough, at the “application” level, so that “my-app” component mapped to the “my-app-next” release (which is renamed to the version i.e. v1.0 when actually released, and a new “my-app-next” is created).

The time spent setting up a trivial housekeeping tool paid for itself. Whenever we had an issue with a release, engineers and project managers alike could readily inspect the changeset. And the automation in place helped us feel compfortable that changeset was actually reliable.

Kubernets Deployments – pre-Helm

Before Helm was widely adopted, Kubernetes deployments were a wild mess of yaml. Every resource you want on the cluster needs its own yaml configuration. You then apply it to the cluster (usually with kubectl apply) and it lives forever (or at least until someone removes it).

Now, it’s easy enough to imagine that each application is responsible for it’s own deployment (CI/CD) and can manage its own yaml configuration.

But this didn’t quite work for us at the time. We wanted a centralized configuration store, to give a clear picture of the desired state of the cluster. That way we could revive a dead cluster from a centralized location, or create a clone of the cluster, etc.

For this, we used a git repo, and every application developer was responsible for making sure the source-of-truth config for their application was kept up-to-date on the trunk branch.

But the application still needed to kubetctl apply the config during its CI/CD process. For this, I devised a simple scheme where each application kept its configs in a subdirectory of the shared k8s-config repo. Then, I built a minimal command line script that could be invoked with that subdirectory name, and it would git clone the k8s-config, checkout the config repo HEAD, and apply the configs in the desired directory.

It wasn’t elegant, but it was effective. It was basically Helm-lite.

Boring Work In The Wild

Boring work doesn’t win awards and it doesn’t necessarily make for a compelling blog post (I’ll let you be the judge). But I find that I always enjoy it: the problem is clear, the solution is obvious – there’s a great deal of satisfaction to be had rapidly stitching existing tools together.

If you're working on projects like this, I'd love to hear from you! If you have any questions or thoughts about this article, please reach out! Or, if you have a topic you'd like me to write about next, let me know!
[email protected]