There are any number of software development best practices you can adopt that will decrease your incidence of creating bugs. But there is only one sure-fire way of ensuring that you don’t release a broken app to prod: functional testing.

Functional testing is like manually testing your entire app, except the process is automated. Also known as behavioral testing, you’re testing the observable behavior of the system rather than its internal mechanics. Every feature has a test, or set of tests, to ensure that feature is behaving as expected. Let’s look at a basic concrete example: logging in. We might want to check that:

• When the user is not logged in, and navigates to the login page, they see a form with username and password fields.

• They are able to type into those fields.

• There is a login button on the page as well, and after the user has typed something into the username and password fields, the button is enabled.

• When the user enters valid credentials, and clicks the login button, they are redirected to the lobby.

• At this point, they are able to navigate to the user account page.

• On the other hand, when the user enters invalid credentials, and clicks the login button, they remain on the login page.

• A message appears that says, “login failed.“

• They are not able to navigate to the user account page.

Expected behavior like this can be extracted directly from a functional specification, if you have one. The next step is to write a little bot that walks through the steps of interacting with your website, and checks that expectations are met.

Implementing Functional Tests with Cypress

For a web application, there are a wide variety of tools you can use to run your app through a series of scripted steps. I like Cypress. Here is what my test for logging in looks like:

There are three tests here: invalid email; invalid password; and successful login. The beforeEach has each test start by navigating to the login page: cy.visit(helpers.url(”login”)). This is the equivalent of the user pasting the login URL into the address bar and pressing enter. Most of the work in these tests has been pulled out into helper methods, since they are used in more than one place. Here is testLoginFails:

Without going into details, you can see how the script attempts to log in with the provided credentials; asserts that we remain on the login page; and asserts that a “Login attempt failed” message appears on the page.

You can see how we use CSS selectors to pull up specific elements on the web page. This means these tests aren’t purely functional — element IDs and class names are implementation details, not really user-facing behavior. But it’s a fair tradeoff: these selectors are technically exposed to users (in the DOM, dev tools, etc.), and honestly, selector-based testing just makes life so much easier.

This is what running the login tests looks like:

Here’s what the entire Badux Go functional test suite looks like:

It takes over five minutes to run on my laptop, but I rarely run it myself these days. More on that later.

Test Suite Growth Trajectory

I started working on my functional test suite very early on in the Badux Go development process. I was implementing features like account creation; logging in and out; game invite creation; and accepting invites. I was pretty happy because I had full coverage for all these features. But I wasn’t able to get it working in continuous integration. Every time I push a code change up to GitHub, my continuous integration suite runs: it compiles all the back end code, and makes sure all the back end tests pass. But I hadn’t figured out yet how to get my Cypress tests running in GitHub.

The next phase of development was implementing the actual game mechanics. I was using a tool called PixiJS to do my interactive game board at the time, which uses WebGL under the hood. Testing WebGL with Cypress turned out to be extremely difficult, so I sort of set aside the Cypress test suite for a while. I had a framework for thoroughly testing game mechanics on the back end, and resorted to manually testing for front end integration. My functional suite was dormant, but my app was still small, so that was alright. When you are agile, you don’t follow rules for rules’ sake.

As coding continued, a number of important developments transpired. I started using Claude AI more and more for supervised coding. Claude AI was maturing to the point where I could start to trust it with larger and more involved tasks. And WebGL was proving unworkable. It was just killing browser performance. After many iterations of trying to tune my PixiJS setup so it wouldn’t lock up the browser, I finally admitted I made a mistake adopting PixiJS, and switched to Konva, which is built on HTML5 Canvas 2D, and not WebGL.

I asked Claude to rewrite the game board for me. With a little help from me, we got it migrated in less than a day. With Konva, all the performance issues were gone. And as an added bonus, Cypress has much better support for testing Canvas than WebGL. I haven’t gotten around to trying that yet, but when I do, I’ll let you know how it goes.

More recently, I was warming up to implement my guest account feature, and I knew I could no longer get away without functional testing. I felt fairly confident with the game mechanics themselves, but other aspects of the app were getting more complicated, and testing all the ins and outs of guest accounts manually was beyond what I was ready to take on.

I embarked on the following steps:

• Fix the suite. A few months of neglect had broken it in various places. It didn’t take that long to fix.

• Get it running in CI. This was super important. It’s annoying to run a big functional test suite by hand, and when you’re coding fast, you’re apt to want to skip this step. No worries! Just check it in and push it up; GitHub will rerun the suite for you. If there are any problems, you’ll be getting an email.

• Require coverage for new features. Your app is growing all the time. You do not have the resources to manually test the entire app every time you complete a new feature, or every time you want to release to prod.

  

At this point, I probably had functional test coverage for maybe 40% of my app. That’s not great, but even low coverage pays off quickly — the worst bugs, the ones you definitely don’t want in prod, tend to break something. With my low-coverage suite, I could at least ensure users could start and complete a game, and a lot of things have to be working right for that.

The coverage ratio improves naturally over time. Every new feature ships with tests. Any time I touch old code, I add the coverage that was missing. And the features that still aren’t covered? They were released a long time ago and have already run the gauntlet.

After adding tests for guest accounts and a couple more features, I’ve probably raised coverage to 60%. I really should add @cypress/code-coverage to my CI build so I know more precisely. It’s on my list.

AI Code Generation

But let me tell you my biggest win in this story: I’ve been betting on the continual improvement of AI codegen for a while now. I don’t know if there are going to be any massive breakthroughs, but I know it will keep getting better. And this investment has paid off, because I no longer need to write my own functional tests. I have Claude write them for me. Of course, I review all the code, and make sure everything I want to cover is covered. But the latest Claude model — Opus 4.5 — writes the tests, runs them, and fixes them, all on its own. This is super important, because writing functional tests can be extremely time consuming. Earlier Claude models were simply unable to take on writing these tests for me. It was just too complex for them.

It probably helps that Claude has been helping me write functional specs and technical specs for all of these features — it seems to give Claude better context when it comes time to write the code. Or maybe that’s just my imagination. Either way, that’s a story for another time.

Want to stay agile? Write functional tests. Better yet, have your AI write them for you. You’ll be a lot less likely to release broken stuff to prod — and you’ll ship faster doing it.

Subscribe now

Share

I knew this was not going to be a small undertaking. I needed so many things, including:

• Rework my user domain model in order to accommodate guest accounts.

• Allow logins by username and not just email, as we would not have emails for guests.

• Create a secure, non-invasive method for authenticating guest users.

• Modify game invitations so that members can choose to only play against other members.

• Allow for upgrading a guest account into a member account, if the user chooses to sign up later.

• Clean up guest accounts that are abandoned.

I wanted to put as much of the actual coding into the hands of my AI, Claude, but from past experience, I knew I would end up with a total mess if I just let Claude start in on it. I’ve witnessed Claude thrash many times. Humans do it too: in trying to get the code to work, they keep making fidgety changes they think might work, trying again, piling on, until there is a whole lot of confused and useless code at cross purposes, and zero increased understanding of the problem at hand. As an engineer, it is essential to recognize when you start thrashing as soon as possible, break out of the cycle, take a step back, and plan a better approach. My reaction so far when Claude gets into one of these cycles is to take back the reins, and get things back on track myself. This works, but wanting more out of my AI, I wondered if there was a better approach.

It didn’t take long for me to come up with the answer: Write a technical specification, or tech spec, and get Claude to implement the spec. A tech spec is an implementation plan that describes how to implement a new feature or feature set. It goes into sufficient detail that it can be implemented by engineers working independently. Specifications help lead engineers disseminate a plan of attack to a team of engineers with different levels of experience. They provide a checklist of what needs to be done, and help define work boundaries and integration points when the work is going to be split across the team. They also force the lead engineer to think through the problem in sufficient detail to catch any obvious oversights before work starts.

When I am coding on my own, I normally don’t bother to write tech specs. I have a pretty clear idea in my head, and if something comes up in the implementation process that I hadn’t considered up front, I just adjust to that realization on the fly. I rarely have to back-track more than a few minutes work. Of course, the smaller the feature being implemented, the better this on-the-fly approach can work. And it breaks down pretty much immediately when I try to bring other engineers into the effort.

Technical specifications have to be tailored to a specific audience. If you go into mind-numbing detail, a senior engineer might feel like you are talking down to them. They are quite capable of handling the details, thank you very much. Your efforts at over-specifying the technical solution have gone to waste, and you are squandering resources by putting a talented engineer on a mindless task. But that level of detail might be essential if you are handing over the work to a junior engineer. Otherwise, they might come back with so many constant questions that you might as well be doing the work yourself. Or they might just give it their best shot, and hand you something that needs extensive revisions before it can merge. I had worked with Claude enough1 to know the rough level of detail to provide: enough to satisfy just about the most junior engineer.

After I wrote up the spec (the first version clocked in at 930 words), I did the most natural thing in the world: I handed it to Claude, asking for feedback. My exact prompt was:

I have written up a tech spec for anonymous accounts. I want you to look it over and give feedback: 1) Is everything clear? Do you have any questions? 2) Are there any aspects of the proposed change that are vague? Are there any unconsidered factors? 3) Are there any potential improvements to the spec that you see?

Whenever possible, tech specs should be reviewed by other engineers to look for errors, oversights, and potential problem spots. Experienced engineers are prized here, because they can recognize patterns that have caused them problems in the past. But perhaps the most important feedback comes from the intended implementers. I want to make sure that Claude knows what is expected here, and that the document is at the right level of detail for them to write the code.

To my surprise, Claude gave me probably the best, most thorough feedback I have ever gotten on a tech spec in my life. And I’ve written a lot of tech specs. Their initial feedback was about 1100 words, and about two thirds of their suggestions were actually relevant. Claude pointed out a couple of major flaws and oversights. They also made clear that I had not gone into sufficient technical detail in a number of places. I also learned that, despite Claude being an automaton, they were still desperate for not just the technical details, but the motivations behind my choices. Just like a human, they were quite prepared to veer off spec and do things “the right way” if they didn’t understand my reasons.

Now, let’s be clear: AIs are not better at reviewing tech specs than humans. It’s mostly that the AI has a number of critical advantages over a human at this juncture, including:

• Carefully reviewing a thousand word tech spec would take an engineer hours; it takes an AI a couple minutes.

• While taking the time to review the document, the AI is not worrying about all the other responsibilities that they have at their job. They do not have to consider that, more likely than not, many of the people expecting other things from the engineer do not really understand the value of carefully reviewing the spec.

• The AI does not have to worry about the original author getting defensive, or prickly, about any of the suggestions they might make. And they do not have to worry about the interpersonal repercussions.

I fixed the major oversights in my spec, and I went into more technical detail where necessary, and I sprinkled a number of "Motivation” sidebars throughout. I gave it back to Claude, and once again got a mountain of feedback. This time, less of what they came back with was actually relevant. One problem with AIs, I have found, is they don’t know when to just be quiet. I did one more round, and asked if Claude was ready to go. It is critical, at this point, to get the buy-in of the people who are actually going to be writing the code. They need to feel like they own their work, and they certainly won’t if they feel pushed into it in any way. This might not be a concern for an AI, and it might have been somewhat of a formality on my part, but you do want to give your engineer one more chance to express any misgivings they might have.

The implementation process itself went about as well as I could have expected. Roughly speaking, I worked like so:

• Ask the AI to implement the next section of the spec.

• Review all the code.

• For every hundred lines or so that Claude writes, there are going to be one or two things that I am going to ask Claude to go back and do differently.

• Make sure everything compiles and all the tests pass.

• Make sure that tests were added to cover all the new functionality.

• Commit the changes.

Following this process, we successfully completed this major feature set in about two weeks. The diff comes in at around 5K lines. The AI more or less stayed on track the whole time. They certainly never started thrashing. There are some things that I already understood are beyond Claude’s capabilities to work out for themselves. One example: they are particularly bad at setup in integration tests. So we’re testing the API endpoint, and the endpoint needs an authentication token to run properly. How do we get it? Well, in the setup portion of the test, we hit the login API endpoint, and extract the authentication token from the positive login response. My experience is so far, Claude has a lot of difficulties with this kind of thing. They are not terribly great with functional effects, and they are pretty bad at generating code that works with external APIs that have undergone recent rapid change. Obviously, AIs are only going to be getting better at these kind of things. I am patient, and I’m not complaining. I’m getting FTE weeks for $100 a month.

Recently, the buzz on places like LinkedIn about “vibe coding” is that the AI cannot write maintainable code, and things quickly spin out of control, so that making any change risks breaking a half dozen other things. My response is: well, yeah. The same thing happens with a Junior engineer, if they are not given the guidance and tools they need. The same thing happens at all different scales of software development. I worked for a company once where the entire code base was like that, and they had hundreds of engineers working on it every day. If you do not adopt sound and rigorous software engineering practices and processes, then your project is going to decay over time. The rate of decay matches the rate of neglect.

This is why I treat Claude like I would treat a very junior engineer. If the project is small and isolated, I will let them take ownership. I will avoid even looking at the code. It’s theirs, and they don’t need my help maintaining a 500 line code base. If it’s the main code base, I will be reviewing every last line. That code base is critical, complex, and large, and I plan on it lasting a long time. So best practices are in full effect.

Prompting the AI via technical specification was highly successful for me in this use-case, and I plan on using this same process going forward for any substantial coding effort.

Share

Subscribe now

This past week or two I sprinted to get my app up on Kubernetes and up on the web at a real domain name. I already had the app fully configured and packaged into a jar, and at this point, all I need is a JDK and a Postgres server to run. Claude helped a lot as I built out k8s yaml and a deploy.sh, first targeting minikube running on my developer box. We sorted a bunch of secrets, and designed the deploy script to build to local (minikube), test, stage and prod (all on the cloud). Postgres would be running external to k8s up on Digital Ocean, so I targeted my local Postgres server for my local deploy. I stuck Prometheus and Grafana in the cluster, and set up some alerting via email and Discord for obvious problem situations, such as extended 75% CPU usage on the web server.

Once everything was working locally, I built out some databases (test, stage, and prod, all up on a single server at DO), generalized those configs, and started using Linux utility pass to store secrets locally and safely. There’s a little script to eval that sets a bunch of env vars for running the deploy, and it pulls out secrets from pass according to deployment target. I jacked the deployment to push Docker images to GitHub’s ghcr.io (free), and Claude and I hacked the deploy script to use sed to make alternate versions of certain k8s yaml, depending on the deployment target. (It would be better to use something like Helm, but I felt it was an unnecessary complication at this point.) Got the whole cluster up on test, stage and prod up on DO.

By this point, I needed to make my prod k8s cluster face the real world. (As we all know, this is not always easy! 😉) I took a couple of wrong paths before I got this all straight. At first, I thought I needed to get a “reserved IP” up on Digital Ocean. Claude and I tried this out, and after a few hours of thrashing, I knew this wasn’t the right way to do it. I figured out that I needed a load balancer, and proceeded to set one up manually on DO, hoping to coax my Kubernetes cluster to use it. Bashed at that for a while before Claude and I realized we would need to create the load balancer through k8s for it to work. Getting the load balancer up was easy enough, but configuring it right took a lot of pain. Trying different ways to get different configs to take, forcing pods to apply, tearing down clusters and rebuilding them, tailing logs, describing pods, etc. Once it finally started working, I could have sworn that the solution was one of the first things I tried. But who knows. I needed to set up the LB to take http and https, and to route the former to the latter. I needed to set keepalive, and to make sure it is up on IPv4 and IPv6 both. (Of course, kubectl describe service makes it look like it’s only running IPv4, which threw me for a loop.) And I needed to map my LB to my domain name, and set up a TLS certificate. And I have to flush DNS caches all over my computer to see it.

It was Friday evening 6:30 or so by the time the app was live. I was so excited, I shared the link with a couple of friends. One of them got back to me with a screenshot from his iPhone. The layout was messed up, and he could only see three quarters of the game board. I was like, fuck. Yes, I develop on Ubuntu, and my phone is a GrapheneOS. I’ve tested my web app on Brave, Firefox and Chromium on Ubuntu, and on Brave and Vanadium on Graphene. Not exactly the most popular browsing platform choices! I realized I was going to have to figure out how to test on Windows, MacOS, iPhone, and probably Android and iPad as well. And it just tore into everything I had just accomplished. A little bummed, I put my work down, and decided to enjoy the weekend. It didn’t take me too long to work out a basic strategy in my head for testing all these browser options, and to figure out I have all the hardware in house to pull it off. Sure, it’s more work. But building an app and a company on my own from the ground up, it’s just one more item on the work list. A tiny fraction of what I’ve already achieved. It will probably delay an official release a while longer, and I should have anticipated this happening, and both of these facts are annoying. But it’s alright.

I often times don’t think people realize how hard some seemingly basic things are, like getting a complete application deployment process working targeting multiple environments. I suppose I could just rent a VM, check out my project there, and tweak the thing until it’s working. But then three of four releases down the line, something goes wrong, and it’s impossible to recover, because who knows what’s actually on the VM image by this point. The way I have it, I can launch to multiple test environments however I want, including copying everything over from prod to stage, and testing any upgrades or migrations there. But yeah, when the layman asks, “What did you do this week?” I deployed my app up to my domain. And if I don’t reflect on the details, it doesn’t really seem like much. I guess that’s part of why I had to write this up.

When I was younger and learning a lot about agile, I really dug it a lot, but I didn’t like the term “sprint”, because it felt like manager-speak for working extra hours. But it makes more sense to me now. You get passionate about something, you just want it done. So you pick it up again after dinner, and when you’re not working, you’re thinking about next steps. And when you reach that goal, it really means something. It’s weekend and it’s time to chill with family. That’s what it’s all about.

Share

Subscribe now