Last week’s post ended with two fixes to kivuli that weren’t yet in a release: the audit chain race and the substitution meter. Both have shipped now, in v0.2.4, which went out on Friday night. On the way there kivuli also put out v0.2.3, raised its Go floor twice and moved the tools its release pipeline downloads, and its own release verification found that every SBOM I’d ever published listed 195 components that weren’t in either binary. None of that week was feature work. All of it was the plumbing around a release, which is the part of a project that only gets attention when a gate goes red, and this week three different gates did.

A floating tag doesn’t float

The first red build was on 17 October. govulncheck failed on the Linux agent with eight standard-library vulnerabilities, all fixed in Go 1.27.2, and the obvious question was why the build was still on 1.27.1 at all.

The Jenkinsfile asked for golang:1.27-bookworm. That reads like “the latest 1.27”, and on a fresh machine it is, but a Docker agent only pulls an image when it doesn’t already have one by that name. The ci-NUC had pulled 1.27-bookworm weeks earlier, when it meant 1.27.1, and it went on meaning 1.27.1 for as long as the cached copy sat there. Nothing about the tag told the agent it had moved upstream.

kivuli has an ADR for this, number 0009, written in September after the same scanner found 26 standard-library vulnerabilities in Go 1.25.0. It says the floor in go.mod is a patch release rather than a minor one, because a minor version alone permits a toolchain with known fixed holes, and that the answer to govulncheck failing is to raise the floor rather than weaken the gate. So d47fd24 raised go.mod to 1.27.2, and the Jenkinsfile now names golang:1.27.2-bookworm exactly, so moving the floor changes the tag and a changed tag forces a pull.

That turned the build red in a new place. ADR 0009 also says that a floor bump moves both of the analysis tools that read compiler output, and I’d moved only the toolchain. Go 1.27.2 writes export data version 5, and staticcheck v0.8.1 reads up to version 4, so the staticcheck gate failed on every package in the module until a2858d0 took it to v0.8.2. That’s a patch release of Go changing a format that a linter depends on, which I wouldn’t have guessed, and which the Makefile now warns about in a comment next to the pin.

The same lesson went into the Jenkins shared library. goBuild, the step in JeakylJenkins/shared-library/vars/goBuild.groovy that kivuli’s whole pipeline runs through, had golang:1.27-bookworm as its default image, and 2934efd changed it to a patch release with a comment explaining the cached-tag trap. It made no difference to kivuli, which passes its own image and ignores the default. The default exists for the next Go project, and I’d rather that project inherit a pinned tag than rediscover this one.

What v0.2.3’s verification found

v0.2.3 was tagged the next morning, the first release built on 1.27.2. Every kivuli release gets a written verification: I download it from the Gitea release page the way a user would, run scripts/verify-release.sh over it, check what each binary reports about itself and compare the SBOM with the previous release’s. This one found two things, and both had been true of every release since v0.1.0.

The first was in the Linux binary’s build information. Go stamps vcs.modified=true into a binary when git status shows anything at build time, untracked files included, and every Linux release binary said true; the darwin ones said false. The SBOM carried the same flag through to a scanner as a second kivuli entry, versioned v0.2.3+dirty. The cause was the pipeline. The Linux release stage cross-compiles with zig, and goBuild unpacked zig into .zig/ inside the checkout; kivuli’s .gitignore covers /bin/ but not /.zig/, so the tree was dirty before the compiler started. I reproduced it locally: the same commit built in a clean tree records false, and with an untracked .zig/zig it records true. That explains the flag without any tracked file having changed. It doesn’t prove none did, because the provenance records the commit and not the state of the tree, and a byte-for-byte rebuild needs zig 0.14.1, which the pipeline pins; my machine has 0.17.0, whose C output differs.

The second finding was worse, and I didn’t find it until 23 October, a week after the release, when I went back over v0.2.3’s SBOM. It had 271 components, and 195 of them belonged to syft, the tool that generated it: a file entry for the syft binary, syft v1.20.0 itself and the 193 Go modules it’s built from, every one located at /home/jenkins/agent/workspace/JeaKylConsulting_kivuli_v0.2.3/bin/syft. goBuild fetched syft into bin/ in the checkout and then ran syft dir:. over the checkout, so syft read the Go build information in its own executable and catalogued it as part of what it was describing.

Only 76 of the components were kivuli’s: 46 modules read from the two release binaries, 21 from go.mod, four from the go.mod files of a test fixture and the patched Markdown grammar, and five file entries. Anyone feeding that SBOM to a vulnerability scanner would have got findings for syft’s dependencies, none of them linked into kivuli, with nothing but the file path to tell them apart.

Then I went back through the older releases. The SBOMs of v0.1.0 to v0.1.3 have 262 components and those of v0.2.0 to v0.2.2 have 271, and in every one of them 194 modules sit at bin/syft. Eight earlier verifications had read those files and none of them noticed. That isn’t because nobody looked; v0.2.0’s verification found and documented a real SBOM defect, a patched grammar that showed up only as a local path, and v0.2.2 fixed it. The verifications compared each SBOM with the one before it, and syft was the same version in all eight, so its 194 modules never appeared in a diff. Comparing releases finds what changed. It’s blind to anything that’s been wrong the same way since the start, and I’d been treating a clean diff as a clean SBOM.

Moving the tools out of the checkout

Both findings had one cause: a tool the pipeline brings with it was landing where the build and the scanner read. So the fix belonged in the shared library and not in kivuli, because a project shouldn’t have to .gitignore directories that its CI invents, and the next Go project on goBuild would have shipped the same SBOM.

b6b76b4 moves zig and syft under the workspace’s @tmp directory. Jenkins keeps that directory next to the checkout rather than inside it, exposes it as WORKSPACE_TMP and mounts it into the build container along with the workspace, so the tools are on hand to the build without being part of it. Each tool now unpacks into a directory named after its version, jeakyl-tools/zig-0.14.1 and so on, with a bin/ of symbolic links on PATH. The old code checked whether bin/zig existed and skipped the download if it did, so bumping the pinned version on an agent that still had the old one would quietly have kept the old one; a versioned directory makes a bump fetch the new tool. zig still gets a symlink rather than a wrapper script, because zig finds its lib/ directory from the real path of its executable, and a symlink resolves to that path where a shim would have had to hard-code it.

I could only test half of this before it shipped. Running the two install steps in golang:1.27.2-bookworm against a fresh repository left git status --porcelain --ignored empty, which was the point. But syft wouldn’t run there: under amd64 emulation on Apple Silicon it crashes at start-up inside the Go runtime, before it reads anything. So the SBOM change was unproven until the next real tag, and the v0.2.3 write-up says so in those words, along with a prediction: the next release should record vcs.modified=false on Linux and have an SBOM of about 76 components, with nothing at bin/syft.

Then 1.27.3

The next real tag came sooner than I’d planned. The build after those commits went red again on govulncheck, this time on GO-2026-6809, a crypto/x509 flaw in how intermediate certificates are trusted, present in 1.27.2 and fixed in 1.27.3. It had nothing to do with the week’s changes: build 39, on a documentation commit from before them, failed the same way, and build 40, which carried a fix for an unrelated flaky test, passed that test and then failed here.

kivuli makes no network connections, verifies no certificates and links no TLS of its own, and govulncheck still reported the vulnerable code as reachable, through a path from one of kivuli’s file readers into the standard library’s I/O. Reachability analysis is conservative, and I’m fairly sure no input to kivuli can reach certificate verification. The gate doesn’t take my word for that, though, which is the reason it’s a gate, and ADR 0009 is explicit that the response is to raise the floor and not to argue the finding down. A waiver I’d written myself, for a tool whose selling point is a security model, wouldn’t be worth much to anyone reading the security model.

So the floor moved again, six days after the last time, and this time across two repositories in the order the workspace’s rules require: the shared library first, then its consumer. b6179cf moved goBuild’s default image to golang:1.27.3-bookworm, and 0cfc7d7 moved kivuli’s go.mod, its Jenkinsfile’s image and the five documents that state the floor. The macOS build agent is my laptop, running Homebrew’s Go, and it had to be upgraded by hand before the push, or the darwin half of the build would have refused to start, because go.mod now asks for a toolchain the agent didn’t have and the pipeline sets GOTOOLCHAIN=local so Go won’t fetch one for itself. staticcheck v0.8.2 read 1.27.3’s output without complaint, so this time the second pin could stay where it was.

Count them: the image, the agent, the module and the linters. That’s four places a single version number has to be kept in step by hand, and ADR 0009 already says the floor “needs an owner and a cadence rather than a fix”. Two bumps in six days is what that sentence looks like in practice.

v0.2.4

v0.2.4 was tagged at 09:56:29Z on 23 October, the build started six seconds later, and it finished just under seven minutes after that. I verified it the same night.

The signatures, checksums and provenance all checked out, and both binaries report v0.2.4, go1.27.3 and network: none, the Linux one inside a debian:12-slim container with networking switched off. Both record vcs.modified=false, which no Linux release binary had done before. The SBOM has 76 components, the number the v0.2.3 write-up predicted: 71 modules and five file entries, nothing at bin/syft and no component that names syft or anchore. I also compared it with v0.2.3’s SBOM after stripping syft’s entries out by name, version and purl, and the only differences were kivuli’s own version, the standard library (pkg:golang/stdlib@1.27.2 before, @1.27.3 now) and the +dirty entry, which has gone.

The verification is careful about what that shows. It confirms the output and not the mechanism: I didn’t read the release stage’s log, so it doesn’t show where on the agent syft and zig actually ran, only that neither left a trace in what shipped. The provenance still doesn’t record which revision of the shared library built the release, either, and since the library now decides what goes into the SBOM, that’s the next gap worth closing.

What I’d take from the week

The two Go bumps were routine. Annoying, but the gate did its job twice, the ADR said what to do both times, and the only real lesson was that a Docker tag is a name for whatever you pulled first, not a promise about what upstream has now.

The SBOM is the part I keep thinking about. For eight releases kivuli shipped a signed bill of materials that a supply-chain section of its own documentation cited as evidence, and roughly 70% of it described a different program. Every check that touched it passed: the signature was valid, the file was well-formed CycloneDX, and each release’s diff against the last was small and explicable. What none of those checks asked was whether the components were actually in the binaries, and that’s the one question that matters to whoever reads an SBOM. A reconciliation against go version -m on the shipped binaries would have caught it on v0.1.0. I’d rather have written that check before the first release than after the eighth, but there’s an obvious place for it now: the same release stage that generates the SBOM, in the library that turned out to own the problem.