Short logs. Post-mortems, one-paragraph wins, ADRs and bench notes from client work and the home cluster.
62 entries
23 Sep 2026
Why the rewrite trades row-level security for one data-access layer and a test suite.
In 2025 the inventory moved its authorisation into the database ([[let-rls-do-the-filtering|Let RLS do the filtering]]). The rewrite moves it back, on purpose.
23 Sep 2026
The Next.js rewrite of the parts inventory moves to plain Postgres, Drizzle and Better Auth.
The first plan for the rewrite migrated the live Supabase database in place. Reviewing that plan ended it. New uuid ids would have broken the Flutter app in daily use, a self-hoster could never build the schema from an empty database, and the migration SQL had a recursive row-level security policy and a path from admin to owner.
15 Sep 2026
Quota as a scheduling signal rather than an error.
Porting [[fleet-that-launches-itself|the benchmark fleet]] to a third cloud, the launchers kept failing shapes on quota errors. The fix was to treat quota as a scheduling signal: wait and retry when CPUs or disk are exhausted, wait on expired tokens, keep polling when a describe call fails.
23 Aug 2026
A weekly refresh that trusted an upstream feed, and the abort gate it earned.
A weekly multi-cloud price refresh had run without a miss for the better part of a year. Then one upstream feed returned nothing: not an error, an empty result, and the pipeline was happy to publish a catalog with a whole cloud missing. Nothing failed, which was the problem.
23 Jun 2026
A race between chart animation and a screenshot, diagnosed from both sides in one day.
Charts loaded through the incremental path rendered blank in the print PDF. On the page side, the placeholder chart had no type and the update path only updated, so a chart whose type changed was never recreated. On the print side, the screenshot fired while the animation was still drawing.
9 Jun 2026
A clean decommission checklist, including what lives outside the repository.
The CV builder that ran on the cluster was replaced by a static site, so its namespace, volume, backups, ingresses and certificates were removed. The documentation was not: the pages were marked decommissioned and kept, with the removal commit linked.
26 May 2026
Two CDK stacks configuring one bucket overwrite each other silently.
Two stacks each declared a notification on the same bucket. Each deploy overwrote the other's configuration, and nothing failed; the loader simply stopped being told about uploads.
11 May 2026
A restore without ownership, and the confusing pair of errors it produces.
A cutover restore used the no-owner option to avoid role errors. It worked, and the next start of the auth service failed on a missing schema, and logins failed on a missing table. Both existed. They belonged to the wrong role.
7 May 2026
The components inventory live on k3s, with the exact list of what went wrong.
The components inventory went live on the cluster against a self-hosted Supabase. The runbook keeps the list: the registry token that had to exist before the first pull, two configuration values to change, a race between namespace creation and the first apply, the rollout pattern for an image tagged latest, and an API gateway that did not reload its routes until restarted.
6 May 2026
A backup job that exited 0 and restored zero rows, and the drill that caught it.
The nightly backup piped a full dump straight into a restore and exited 0 every night. The first real drill restored zero rows. Three causes stacked: a client-server version mismatch in the dump format, a schema shape pre-created by the chart bootstrap, and a parser error that cascaded silently.
6 May 2026
The second of two Hetzner and Ubuntu 24.04 traps; short and easy to miss.
Pods could resolve nothing. CoreDNS was forwarding to the node's resolver, and on Ubuntu 24.04 that file points at the systemd-resolved stub on a loopback address the pods cannot reach.
4 May 2026
Bundled Postgres per project for now, with written triggers to revisit.
An operator-managed Postgres was evaluated, an image was even built for it, and the decision was to defer: each project keeps the Postgres bundled with its Supabase release, backed up nightly.
26 Apr 2026
The start of the self-hosted platform every later entry runs on.
A k3s cluster on four Hetzner Cloud machines, one control plane and three workers, declared in a new infrastructure repository: Traefik for ingress, cert-manager for certificates, ArgoCD for GitOps, kube-prometheus-stack and Loki for observability.
26 Apr 2026
Coolify deployed the app fast; backups, logs and alerts were what it could not give me.
Coolify made the first deploy of the inventory quick: [[fifty-minutes-after-coolify|five fixes in fifty minutes]]. Running the app was another matter. Backups could not be managed the way I needed, the containers left little room for fine-tuning, there were no reports or alerts, and the logs were close to useless when something broke.
13 Feb 2026
Replacing a per-instance script matrix with detection at boot.
The [[fleet-that-launches-itself|first fleet]] had a script per instance type. It worked until the matrix had to change, and then every change was a dozen edits. The replacement is one script that decides at boot: swap sized from the memory it finds, CPU generation detected from the processor, compiler flags chosen per generation.
6 Feb 2026
What has to be in user-data for a machine to finish a job with nobody watching.
A benchmarking campaign across hundreds of instance types cannot have a person watching each one. Each instance boots from a user-data script that installs the toolchain, runs the benchmark, uploads the results and terminates itself. The first run where every report came back clean was the milestone.
19 Jan 2026
How a hard validation gate turned into a warning, and why that was right.
A validation-only state machine with detailed notifications, so the checks could be run without moving data. One check, a name mismatch between two records, kept blocking runs for cases that were legitimate. For months it grew exceptions, one per case.
8 Jan 2026
Why a rollback that runs on every failure can be worse than none.
An Iceberg insert stage went into the analytics pipeline in the morning. By half past four it had a rollback on failure; by five to five the rollback was muted.
27 Nov 2025
Five fixes for Flutter web behind nginx, each a reusable gotcha.
First self-hosted deploy on Coolify, then five fixes in fifty minutes: a deprecated renderer flag in the Dockerfile, stale HTML served from cache after a new build, nginx location blocks in the wrong order, row-level-security errors leaking raw into the UI.
27 Nov 2025
Moving authorisation from the client into the database, and the code that disappeared.
Every query in the app filtered by the current user's id on the client. Supabase row-level security already did that on the server, so the client filters were both redundant and a false sense of safety.
7 Oct 2025
A model call is too slow for a request-response API behind a gateway timeout.
The first version called the model synchronously inside the API handler. It worked in testing and timed out in production, because a gateway waits thirty seconds and a model does not promise to answer in thirty seconds.
18 Sep 2025
Archive, verify, delete, keep a way back.
Retiring functions is easy; retiring them safely is a sequence. Archive the code and configuration first, verify that the archive restores, delete, and keep the restore script next to the archive.
31 Jul 2025
An honest ADR on removing abstraction from a small app.
The components inventory was rewritten from Next.js with PostgreSQL and Docker to Flutter web on Supabase, and its layered architecture collapsed into a flat tree of services, providers and pages.
28 Jul 2025
One hub on Wi-Fi, battery nodes talking to it point to point, for places Wi-Fi does not reach.
The goal: see what happens where Wi-Fi does not reach, in a garage one or two floors down, at the mailbox, on the veranda. The design is a star. One hub indoors stays on Wi-Fi; battery nodes sleep almost all the time, wake to send one frame with all their readings, listen two seconds for commands, and send an alarm at once when a door or motion pin fires.
28 Jul 2025
A hub key that never leaves the chip, a household key handed out at pairing, revocation by deleting a row.
Security was designed before the firmware. The hub creates its private key on first boot, keeps it in the chip, and gets a signed certificate from a certificate request; it talks to the backend over mutual TLS. Nodes ship with nothing secret, only their unique ID.
27 Jul 2025
From Good Display's Arduino sample to a PlatformIO project, and an adapter board of my own.
Good Display's sample drove the panel from an Arduino board. Moving it to an ESP32 meant a PlatformIO project, the driver as a library, and a new pin map: busy, reset, data or command, and chip select on ESP32 pins, with SPI on its defaults.
21 Jul 2025
A streaming parser rolled back and re-landed in one day.
Five iterations of a streaming parser before eleven in the morning, the parser removed, a rollback to the stable build, then a fresh series of fixes until late evening. The eighth one held.
30 Jun 2025
What has to change for an analyst's notebook to run as a production handler.
The analysts had the algorithm and they lived in notebooks. Instead of translating each change into application code, a converter turns the notebook itself into the Lambda handler: first a local script, then a deployed stage, then a rewrite of the converter once the pattern proved itself.
24 Jun 2025
The e-ink frame renderer, built twice in one day on two backends.
The frame renderer was built twice in one day. In the morning it was an Appwrite function in Node: fill an SVG template with the word and picture, rasterise it with resvg, store the PNG. By the afternoon it was a Supabase Edge Function in Deno that also picks a random word from the database and returns the frame with the device's sleep time in a header.
23 Jun 2025
The e-ink device fetches a finished picture; the server does the layout.
In the design, the device lays nothing out. It wakes, asks the server for its next frame, and gets back a finished PNG in the panel's six colours, a pronunciation clip and how long to sleep. Themes, fonts and the word itself are the server's problem, so a new layout never needs a firmware update.
18 Jun 2025
A six-colour panel needs PSRAM, a dual-core ESP32-S3 and patience on every refresh.
The desk device shows a word, a phrase and a picture on e-ink. A comparison of two 800 by 480 panels, 2-bit grayscale and six-colour Spectra 6, decided most of the hardware before any board was drawn.
18 Jun 2025
Two Hetzner servers, keys-only SSH and an admin console behind Zero Trust, ten months before the cluster.
The backend for the e-ink desk device started on two Hetzner Cloud servers: one ran Appwrite in Docker, the other served the Flutter web app from nginx, with Cloudflare in front for DNS.
24 Jan 2025
Environment promotion for a serverless app done by branch merges.
Two Amplify environments mapped to two branches: main deploys dev, prod deploys prod, and promotion is a merge. A short helper document lists the order of operations and the one command that must not run against the wrong environment.
4 Oct 2024
A drag-and-drop layout editor in Flutter web, with Java Lambdas doing the heavy work.
A Flutter web app for building report pages from blocks and widgets. The user drags blocks into a tree and sets margins, borders and colors in side panels; the app saves the layout as JSON.
7 Sep 2024
A second-generation rewrite with a collaborator on the backend.
The sensor app came back as a Flutter web application on Amplify, with a collaborator owning the backend. The Lambdas moved to Python 3.12 and were split by responsibility, one function per device-management concern instead of one handler for everything.
13 Aug 2024
A new union approach, reverted, then re-applied the same evening.
A new approach to the union action went out in the afternoon, was reverted in the evening, and was re-applied ninety minutes later. The first version was right about the algorithm and wrong about one edge: columns present in one input and not the other.
24 May 2024
How a workbook engine eats memory, and where the real limit was.
The memory ladder over fifteen months: more heap in March and a switch to ZGC, an out-of-memory handler in May, the heap raised again, and a year later the cap on byte arrays raised again.
26 Feb 2024
Screenshot diffs before every release, and why partial results mattered more than a perfect diff.
Four days from an empty repository to a screenshot-based QA service: a Lambda driving a headless browser, state in DynamoDB, a two-stage error handler.
9 Jan 2024
A four-inch six-colour e-paper panel on the bench, eighteen months before the device had a name.
The desk device started with a panel, not a plan: a Good Display GDEP040E01, a four-inch six-colour Spectra 6 e-paper module, driven from an Arduino board by Good Display's sample code. The first frame was a hard-coded image, 400 by 600 pixels at 4 bits each, 240,000 bytes compiled into the firmware.
20 Nov 2023
From Node Lambdas to a Spring Boot container on Fargate, and why.
Two weeks after the first commit of the Java rewrite I joined it and shipped the substitute, dedup, join and graph actions in the first week. The platform decision behind it: the workbook engine had hit [[lambda-outgrows-itself|the Lambda time limit and the memory ceiling]], and there was nothing left to tune.
13 Jun 2023
How a function grows to thousands of lines, and what that costs.
A streaming variant of the ETL landed as one commit with a processor Lambda of several thousand lines. It worked, and it was the moment [[excel-engine-moves-to-java|the Java rewrite five months later]] became inevitable.
7 Jun 2023
A dead-letter handler is only tested when things fail.
The dead-letter-queue handler had been in place for six months and had never seen a real failure. The first poison message needed a "major fix" twice in one day.
29 Jul 2022
Validate, clean, process: the pipeline shape that every later version kept.
The intake pipeline got its own repository and its own shape: a preprocessor, an input handler, a validator and a cleaner as separate Lambdas, with the processor split out six weeks later. That shape survived three rewrites and a change of language, and the first version that used it is still in production.
23 Mar 2022
The check a bench cannot do: a real GPS fix on a city street, seen on Android and iOS at once.
The third generation went into a field check with two trackers and two phones, Android and iOS. Each tracker owns one Firestore document, where the first generation searched a shared Realtime Database list for its own entry. It listens there for commands (light, voice, reboot) and writes battery, temperature and a timestamp back every five seconds; position comes from the modem's GPS.
18 Feb 2022
A one-week, two-person Flutter experiment with on-device text recognition.
A travel helper meant to read a price tag through the camera and show it in another currency. In one week with a collaborator, one branch each, merged through pull requests, it reached the first half: a live camera view with on-device text recognition drawing what it read over the preview.
13 Dec 2021
Thirteen months and ninety versioned builds for a solo mobile release.
Version 1.0.0 of the sensor companion app, thirteen months after the prototype and ninety versioned builds later, with an iOS archive for testers roughly every fourth build.
9 Dec 2021
Numbers that are really strings, met in the first month and never forgotten.
Three fixes in one day, all the same bug wearing different clothes. An identifier came out in exponent form. Another lost precision past fifteen digits. A third lost its leading zero. The spreadsheet had decided these were numbers; they were codes.
24 Nov 2021
The week a home-grown formula evaluator stopped paying for itself.
The first serverless Excel ETL evaluated formulas with code I had written myself. In three days it needed six fixes: references that shifted when rows moved, division by zero, a lookup that missed. Each fix was a wrong assumption about how references behave when cells move.
3 Nov 2021
The first microcontroller project, and the device side of a problem the sensor app solved another way.
A temperature and humidity sensor on an ESP32 that takes Wi-Fi credentials over Bluetooth LE, stores them in flash and posts readings to a cloud function. One commit, one weekend.
31 Jul 2021
Let the camera's encoder do the video; the stream that stayed ran at a sixth of the first bitrate.
A pet tracker on a Raspberry Pi 4 with an LTE modem, a camera, an audio board and a UPS board. The app asks for video by setting a flag in Firebase; the Pi starts a stream and pushes it over RTMP to a cloud transcoder that serves HLS back to the phone.
8 Jul 2021
Prove the transport in a throw-away app, then wire the UI.
A control app for a fleet of cleaning robots: sign-in, live robot state, commands, plans and reports. Before any of the screens, a throw-away demo app proved the MQTT path to the broker in isolation. Two days later the credentials and the robot-state card landed in the real app. The demo video went out a week after that.
10 May 2021
A two-cloud setup chosen for one feature, and what it costs in configuration.
Auth and the API lived on AWS. Push notifications went to Firebase Cloud Messaging, because that was the path with a working client library on both mobile platforms the week it was needed.
21 Jan 2021
Copy-per-model was the fast choice for a four-month job; here is what it cost.
The second and third vehicle models were delivered as full copies of the first catalogue, plus a landing page to choose between them, instead of one application with a model parameter.
7 Nov 2020
The first managed-auth decision, and the CloudFormation that came with it.
The mobile app's first commit already had AWS Amplify with Cognito for sign-in, with the CloudFormation generated by the Amplify CLI committed next to the Dart code.
19 Oct 2020
An eight-day onboarding prototype that decided the next five years of mobile work.
After a Java app, a Kotlin app and a React Native app in one year, the next project started in Flutter: a sensor-onboarding prototype, connect, instructions, credentials, control page, in eight days.
17 Sep 2020
How a catalogue with hundreds of drawings was made navigable without a CMS.
A spare-parts catalogue for an all-terrain vehicle: hundreds of exploded-view drawings, each part clickable, each click landing in a revision-aware parts table with a basket that mails the order.
28 Jul 2020
Eight builds to stabilise a USB camera on a tablet; the final fix was one line.
Adding a USB camera feed inside a lesson slide took a JNI camera library and one afternoon. Making it survive on the actual tablet took eight builds in six days, each one a different combination of manifest permissions and foreground-service flags.
14 Jul 2020
Second store app, two months after the first, and the only React Native project.
A rhyme finder that shows a definition for any word you tap, built in React Native and listed on Google Play two months after the first Kotlin app. One external rhyme API, one dictionary lookup, a navigation stack and a handful of empty states.
11 May 2020
A Kotlin age calculator: five days of code, a public listing eight days after the first commit.
The first Kotlin app took five days to write and was on Google Play four days later: an age calculator with a custom numeric keyboard and a shake on invalid input. Small on purpose, so that the store process itself was the thing being learned.
11 Feb 2020
Branching by hardware target: what it cost in merges and why alpha suffixes replaced versions.
When a second tablet model arrived, the app got a build line per hardware generation instead of one binary with runtime detection. Each line carried its own version string with an alpha suffix, because the two lines never released in step.
27 Nov 2019
Localising a content-heavy app early is cheap; retrofitting is not.
The first three interactive lessons shipped complete in all three languages on the same day, with strings split into per-language resource files from the first commit.
12 Oct 2019
The first hardware-in-the-loop code I shipped, nine days after the first commit, on a science tablet with its own sensor port.
Nine days after the first commit of the Sensorium tablet app, a lab sensor on the tablet's own sensor port delivered its first reading into an Android activity. The vendor SDK arrived as a binary, so the integration was a foreground service that owned the connection and a callback that every lesson screen subscribed to.
Subscribe
Pulls new entries via RSS, no email, no tracking. Items carry the kind as a category and the event date in the description.
/rss.xml