Three bug reports in three days, and what they changed in our pipeline

Milo is a young product. We do not have a QA department, we have early users, and last week three of them took the time to write to us. Two reports came from the same person on two consecutive days, the third came from a developer on the WordPress.org forums.

All three were interface bugs. No data was ever at risk, no payment was affected, nothing touched the parts of the plugin that handle money. Each one was fixed and released within a day. But taken together they exposed something more interesting than the bugs themselves, so we are writing it down.

Report one: the checkbox that was not there

Rodolfo Melogli was creating a variable subscription product. He added an attribute, and then he could not generate any variations, because the "Used for variations" checkbox never appeared.

The cause turned out to be a detail of how WooCommerce handles custom product types. WooCommerce hides the elements belonging to every registered product type and only reveals the ones matching the current type. That checkbox is tagged for WooCommerce's own variable type, so for our variable-subscription type it stayed hidden. We already handled that on page load and when the type changes, but the attributes panel is also re-rendered over AJAX when you add or save an attribute, and those renders never fire the event we were listening to. Exactly the flow Rodolfo described.

The fix is small. What matters more is that the flow he used had no end to end test, so nothing would have caught it. It has one now: a test that opens the product editor, adds an attribute, saves it, and generates the variations, the same way a human does.

Report two: our own fix, one day later

The next day, same person, second report. On plain variable products, the ones that have nothing to do with subscriptions, the Attributes tab had disappeared entirely. He was not even sure it came from Milo.

It did. Our fix from the day before was built on a wrong assumption. We had tagged WooCommerce's Attributes tab with a visibility class for our own product type, believing that tab was conditional. It is not, it has no visibility class at all and is meant to always be there. Adding one turned it into a tab WooCommerce hides for every other product type.

That is the honest lesson of the week: a fix built on a wrong mental model does not just fail, it creates a second bug somewhere you were not looking. This one also became a test, a matrix that checks which tabs are visible for simple, variable, subscription and variable subscription products.

The part where we look at ourselves

While shipping those two fixes, we found something less comfortable.

Our end to end suite runs on every merge to master. Except that it had not completed for two of our recent releases. Not because the tests did not exist, they did, but because the runs never finished: some were cancelled by hand out of impatience while waiting for a release, and the concurrency rule that was supposed to protect the master branch turned out not to protect it at all, so successive merges were quietly cancelling each other.

Nothing broken reached users through that gap, and the versions in question have since been through the full suite. But a safety net you skip when you are in a hurry is not a safety net, it is a decoration. So we fixed the process rather than promising to be more disciplined:

• The release script now refuses to publish unless a successful test run exists for the exact commit being released, and unless the end to end job inside that run actually ran and passed. There is an emergency override, and using it is a deliberate act rather than an accident.
• Master runs get their own concurrency group, so nothing can cancel them, including us.

Report three: the bug that only existed on someone else's install

The third report came from the WordPress.org forums. A developer got a "Page not found" error when opening our Payments screen, on his local environment and in production. We could not reproduce it anywhere.

The clue was in the URL he pasted. He runs the roots.io stack, where WordPress core lives in a /wp/ subdirectory, so the admin is served from /wp/wp-admin/. His URL had no /wp/ in it. We had six links in the admin interface written as root relative paths, /wp-admin/something. On a standard install those work. On his install they point at the front end, where his theme correctly answers with a 404.

Six links, invisible on every machine we own, fatal on his. "It works on my machine" is a statement about your machine, not about your code.

Every one of those links now comes from the URL WordPress itself reports, which is correct on any install, and a test fails the build if a hardcoded admin path ever comes back.

What is permanently different now

None of these three reports cost us more than a day of work. What they changed is worth much more than the fixes:

• Three new end to end tests, covering the exact flows the reports described.
• A job that boots WordPress without WooCommerce on every push, because a fatal error there was our previous lesson.
• A smoke test that fails on any JavaScript error across the main admin screens.
• An end to end test for the migration from other subscription plugins, which is the single feature we most want to never break.
• A release process that cannot ship without proof that all of it ran.

The suite went from a comfortable number of tests to a slightly less comfortable one, and the release takes fifteen minutes longer. Both are a bargain.

Why we are writing this down

We could have shipped these fixes quietly. Most products do. But the whole bet behind Milo is that the engine stays free and open, and open means you get to see how the thing is actually maintained, not just the changelog.

There is also something worth saying out loud to anyone building a product with a handful of early users: those users are doing your quality assurance for free, on their own time, often on a Sunday. The only honest way to repay that is to make sure their bug is the last of its kind. That is what a test is, a promise that a specific person's afternoon will not be wasted twice.

So, thank you Rodolfo, twice. And thank you to the developer on the forums whose stack broke our assumptions in the most useful way.

If you are running Milo and something feels wrong, tell us. It is a good week to file a bug.