I needed Jenkins to push a staging stack to a sibling NUC, on every merge to main. No registry pushes, no Kubernetes, no Argo, no Helm. Just SSH, rsync, and docker compose up -d. The whole flow is one shared-library step now, and the calling Jenkinsfile is a handful of lines.

This post is the friction list from getting that step right. Six small surprises, none well-Googled, all costing twenty to forty minutes the first time. The step itself lives at shared-library/vars/deployComposeRemote.groovy in JeakylJenkins and is around 130 lines; the post is mostly about why each of those lines is the way it is.

The shape

The project lives in its own git repo with a docker-compose.staging.yml at the root that brings up an API container, a small web container, and a Postgres. Jenkins builds the project (lint, typecheck, test, esbuild bundle for the API, vite build for the SPA), then ships the compose file, the env file, and the source tree to a NUC that runs the actual containers.

flowchart TD
  Start([Jenkins build on main]):::start
  Start --> Build[pnpm build<br>esbuild api + vite web]
  Build --> Stage[Stage compose.yml + .env<br>under .deploy-stage-N/ in workspace]
  Stage --> RsyncStaged[rsync staged dir<br>to deployUser@host:/opt/myapp/]
  RsyncStaged --> RsyncSource[rsync source tree<br>to deployUser@host:/opt/myapp/repo/]
  RsyncSource --> SSHBuild[ssh: docker compose --env-file .env<br>-f repo/compose.yml build]
  SSHBuild --> SSHUp[ssh: docker compose up -d<br>--no-build --pull never --remove-orphans]
  SSHUp --> SSHPs[ssh: docker compose ps]
  SSHPs --> Cleanup[rm -rf .deploy-stage-N/]
  classDef start fill:#1b2740,color:#fff,stroke:#1b2740;

The decision I made early, and which most of this post is downstream of: build the images on the deploy host, not in CI. There are two ways to ship a compose stack from a CI box to a runtime box:

  • Build in CI, push to a registry, pull on the deploy host.
  • Rsync the source tree to the deploy host, build there.

The first is the conventional path. It costs me running a registry, or paying for one, and managing image cleanup on both sides. The second is what the Jenkins JCasC post implicitly assumed: the deploy host already has Docker, rsync, and an SSH key. No registry, no auth, no image bloat.

The trade-off is real. Every deploy ships the whole source tree, and the deploy host needs the build toolchain in its compose Dockerfiles. For a Node TypeScript project a few MB in size, sitting on a LAN, that’s a non-issue. For a 4GB Go monorepo on a slow link, you’d want a registry.

I went with rsync-build-on-target. A Jenkinsfile using the step looks like this:

stage('Deploy to nuc-staging') {
  when { branch 'main' }
  steps {
    withCredentials([
      string(credentialsId: 'postgres-staging-password', variable: 'POSTGRES_PASSWORD'),
      string(credentialsId: 'anthropic-api-key',        variable: 'ANTHROPIC_API_KEY'),
    ]) {
      deployComposeRemote(
        host: 'nuc-node2.jeakyl.com',
        deployUser: 'deploy',
        deployDir: '/opt/myapp',
        composeFile: 'docker-compose.staging.yml',
        sshCredId: 'nuc-node2-deploy-ssh',
        envVars: [
          IMAGE_TAG: env.GIT_COMMIT.take(7),
        ],
        envSecrets: ['POSTGRES_PASSWORD', 'ANTHROPIC_API_KEY'],
      )
    }
  }
}

What it does, in order:

  1. Stages the compose file plus a freshly rendered .env in a build-numbered scratch directory inside the workspace.
  2. Rsyncs the staged files to <deployDir>/.
  3. Rsyncs the source tree to <deployDir>/repo/ so the compose Dockerfiles can build there.
  4. SSHes in: docker compose ... build, then docker compose ... up -d --no-build --pull never --remove-orphans, then docker compose ... ps.
  5. Cleans up the scratch directory.

About 40 lines of happy path. The other 90 lines are where the surprises live.

Surprise 1: writeFile is workspace-relative, even when you hand it an absolute path

The first version of the step staged to /tmp/deploy-stage-${BUILD_NUMBER}:

String stageDir = "/tmp/deploy-stage-${env.BUILD_NUMBER}"
sh "mkdir -p '${stageDir}'"
writeFile file: "${stageDir}/.env", text: renderEnvFile(envVars)
sh "chmod 600 '${stageDir}/.env'"

That silently writes to ${WORKSPACE}/tmp/deploy-stage-42/.env, not to /tmp/deploy-stage-42/.env. The mkdir -p and the chmod operate on the real /tmp path; the Jenkins writeFile step interprets its file: argument as workspace-relative regardless, so a leading slash gets treated as “literally a directory named tmp under the workspace root”. They diverge.

The symptom is that chmod fails with chmod: cannot access '/tmp/deploy-stage-42/.env': No such file or directory, which is technically true: the file is in the workspace under a tmp/ directory you didn’t create deliberately, not in /tmp where everything else expects it.

The fix is to stage inside the workspace from the start:

String stageDir = ".deploy-stage-${env.BUILD_NUMBER}"
sh "rm -rf '${stageDir}' && mkdir -p '${stageDir}'"
writeFile file: "${stageDir}/.env", text: renderEnvFile(envVars)
sh "chmod 600 '${stageDir}/.env'"

Now writeFile and chmod agree on where the file is. The leading dot keeps it out of casual ls; the build number scopes it per build so concurrent runs don’t clobber each other.

The Jenkins docs do mention that writeFile is workspace-relative. They mention it once, in a sentence on the step reference page. Easy to skim past, and the consequence of skimming past is one of those subtle “the same path means two different things to two different lines of code” bugs.

Surprise 2: withCredentials values aren’t on env.X

This is the big one, and the reason the step has two separate parameters for env-var injection: envVars for plain values, and envSecrets for credential-bound values.

The first version had only envVars. The pattern looked clean:

// In a project Jenkinsfile, the obvious-looking thing:
withCredentials([string(credentialsId: 'db-pw', variable: 'DB_PASSWORD')]) {
  deployComposeRemote(
    envVars: [
      IMAGE_TAG:   env.GIT_COMMIT.take(7),
      DB_PASSWORD: env.DB_PASSWORD,    // <-- this looks fine, this is wrong
    ],
  )
}

It is not fine. withCredentials binds the value into the shell environment of any sh step that runs inside its block, but it does not write the value onto Groovy’s env map. env.DB_PASSWORD from Groovy is null. The map being passed into deployComposeRemote therefore has DB_PASSWORD: null. The renderer turns null values into the literal four-character string null, and that’s what ends up in the .env file shipped over:

IMAGE_TAG=4f70d44
DB_PASSWORD=null

The containers come up. The app tries to connect to Postgres with the password being the literal characters “null”, and you get a Postgres log entry about authentication failure that takes a second look to decode – authentication failed for user "app" is exactly what you’d see if the password were wrong, which is also exactly what’s happening. The bug is upstream of where it manifests.

The fix is to give the step a separate parameter that takes a list of variable names and reads them from inside a sh step:

envSecrets.each { name ->
  if (!(name ==~ /[A-Za-z_][A-Za-z0-9_]*/)) {
    error("invalid envSecrets entry '${name}'")
  }
}

if (envSecrets) {
  String appendLines = envSecrets.collect { name ->
    "printf '%s\\n' \"${name}=\${${name}-}\" >> '${stageDir}/.env'"
  }.join('\n        ')
  sh """
    set -eu
    { set +x; } 2>/dev/null
    ${appendLines}
  """
}

Three details on that block:

  • printf '%s\n', not echo. echo misbehaves on values containing backslashes, percent signs, or -e as the first character. printf keeps the value literal.
  • set +x keeps the value out of the xtrace log. Jenkins enables set -x on sh steps by default, so without the suppression the value would land in the build log. That defeats the entire point of using a credential.
  • ${NAME-} not ${NAME}. Under set -u, an unset variable terminates the script. ${NAME-} substitutes empty if the variable isn’t set, which matters because withCredentials doesn’t bind a Secret Text whose value is empty – an unprovisioned secret degrades to “” rather than failing the deploy. The consuming app has to treat empty as “absent”, which is the right default behaviour anyway.

The regex on envSecrets keeps a Jenkinsfile author from passing a name with shell metacharacters in it and accidentally opening an injection vector.

If you do nothing else from this post, take this one. The “credentials bound by withCredentials are shell env vars, not Groovy env vars” thing has bitten me three times in different shared-library steps and at least one Stack Overflow thread has the same confusion. It is the kind of bug where the wrong value is the literal string null, which makes the failure look like a configuration mistake rather than a binding mistake.

Surprise 3: –no-build –pull never on up, after a separate build

Compose’s default behaviour on up is: if an image has a build: block, attempt to pull it from a registry first, then fall back to building. That’s a reasonable default in a build-and-push-and-pull world. In build-on-target it’s actively broken:

app-api Pulling
manifest unknown

The image tag app-api:abc1234 exists locally on the deploy host after the build step, but no registry knows about it. docker compose up tries to fetch it, fails with manifest unknown, and gives up before it would have fallen back to a local build.

The fix is a two-phase invocation:

docker compose -f repo/compose.yml --env-file .env build
docker compose -f repo/compose.yml --env-file .env up -d \
  --no-build --pull never --remove-orphans

--no-build skips the build step (we just did it). --pull never skips the registry pull entirely. --remove-orphans cleans up containers from services that have been removed across deploys, which would otherwise pile up. Without all three flags, the deploy is flaky in ways that depend on the order of compose’s internal checks, the cache state, and what compose decided to call “missing” today.

Compose v2 changed the default pull behaviour from missing to always somewhere around v2.20, so if you copy-pasted a working snippet from a year ago it might have worked then and stopped working now. Pin the flags rather than rely on the default that’s right today.

Surprise 4: rsync excludes and the missing dist/

When I first ran the deploy end-to-end, the build on the deploy host succeeded but the API container immediately crashed with:

Error: Cannot find module '/app/dist/handlers/intake.js'

The Dockerfile’s COPY dist/handlers/ /app/dist/handlers/ was looking for files that didn’t exist in the build context. The reason was my rsync exclude list, lifted from a generic “common rsync excludes” snippet:

--exclude='.git'
--exclude='node_modules'
--exclude='dist'            # <-- this one
--exclude='.terraform'

For a Python project that ships source and builds inside the container, excluding dist/ is right. For a TypeScript project where Jenkins runs pnpm build to produce dist/ and the Dockerfile COPYs from dist/, it’s exactly the opposite of what I want: the build artefacts have to ride along.

Three characters of fix; the exclude list now is:

--exclude='.git'
--exclude='node_modules'
--exclude='_refs'              # large reference data, deploy doesn't need
--exclude='.terraform'
--exclude='*.tfstate*'
--exclude='.deploy-stage-*'    # the staging dir we just created

This is one of those bugs where the error message is exactly right and the cause is nowhere near the error. If you do build-on-target, your CI-produced artefacts have to either be shipped or rebuilt on target. Decide which, then check your rsync flags agree with the decision.

Surprise 5: SSH -e quoting

This cost me 45 minutes of “the deploy refuses to authenticate, but I can SSH manually with the same key”. The original step ran rsync over SSH with the key file bound via withCredentials:

withCredentials([sshUserPrivateKey(credentialsId: sshCredId, keyFileVariable: 'SSH_KEY_FILE')]) {
  String sshOpts = '-o StrictHostKeyChecking=accept-new -i $SSH_KEY_FILE'
  sh "rsync -az -e 'ssh ${sshOpts}' ./ ${remote}:${deployDir}/"
}

Watch the single quotes around 'ssh ${sshOpts}'. Bash does not expand $SSH_KEY_FILE inside single quotes. The string ssh receives as its -e command is the literal ssh -o StrictHostKeyChecking=accept-new -i $SSH_KEY_FILE, with the dollar-string intact. SSH treats $SSH_KEY_FILE as a filename, fails to find it, then falls back to password auth, which the deploy account refuses (correctly).

The error in rsync’s output mentions password auth being refused, which sent me staring at ~/.ssh/authorized_keys on the deploy host for a misconfiguration that wasn’t there. The actual problem was four characters of quoting two levels up.

The fix is double quotes around the inner ssh ... string:

String sshOpts = "-o StrictHostKeyChecking=accept-new -i \$SSH_KEY_FILE"
sh """
  rsync -az -e "ssh ${sshOpts}" ./ ${remote}:${deployDir}/
"""

Now bash expands $SSH_KEY_FILE before invoking ssh; ssh gets a real path to the private key; auth succeeds.

The general rule, which I should have internalised by now and somehow keep needing to relearn: when you hand a command string to a tool that runs it through a subshell, work out which layer expands what, then write the quotes accordingly. The triple-quoted Groovy string makes it look like one big block of text, but the inner "ssh ..." is a separate shell-string that has to expand at shell time, not at Groovy time.

Surprise 6: Docker IPs on a shared external network drift across reboots

The most recent one, and the most insidious. The compose stack attaches its API and web containers to two networks: a private app network for internal traffic, and the pre-existing proxy-net so NPM (Nginx Proxy Manager, in its own compose stack on the same NUC) can reach them by container name. That worked for weeks.

Then the host rebooted – kernel update, nothing to do with any of this – and the public staging hostnames started returning 502. NPM was up. The staging containers were up. docker network inspect proxy-net showed everyone attached. Container-name DNS resolved correctly from inside the NPM container. The IPs were just not the IPs the system had been talking to before the reboot, and one of them was an IP NPM itself used to own.

When you attach a container to a shared external bridge network without pinning its IP, Docker allocates the next free address in the subnet at attach time. The order in which containers attach depends on the order their compose stacks come up, which (across a host reboot, with several stacks restarting in parallel) is not deterministic. NPM had been on 172.18.0.2 before the reboot; afterwards, the staging API came up first, was handed .2 because it was free, and NPM landed somewhere else. A leftover binding inside NPM that referred to its old self-address broke; everything downstream broke with it.

The fix is to pin the IP explicitly on the shared network, on every service that participates in it:

services:
  api:
    networks:
      app:
      proxy-net:
        ipv4_address: 172.18.0.15
  web:
    networks:
      app:
      proxy-net:
        ipv4_address: 172.18.0.14

Plus a small convention written into both compose files as a comment: .2-.9 are reserved for NPM-class services other things depend on; residents (anything NPM proxies to) pin from .10 up. Docker doesn’t enforce the reservation; the comment does. Two stacks on one host is the limit of what that scales to, and that’s the situation, so it’s enough.

Boot-order races are invisible until the first reboot, which can be months after the system was last touched. Pin IPs on any container that’s a stable upstream for somebody else, even when the subnet has headroom. The cost of pinning is one line per service; the cost of not pinning is a half-hour outage with no obvious cause.

What’s left

A few things I haven’t done yet:

  • A dryRun: true flag that prints the rsync and ssh commands without executing. The project has a scripts/deploy-staging.sh --dry-run for this on the local side, but a CI-side equivalent would save a build cycle when iterating on the deploy itself.
  • Per-service health waits after up. The compose file has healthchecks, but the step returns as soon as up -d returns, which is before any healthcheck has run. A “wait until all services Healthy” loop would tighten the failure feedback by a couple of minutes.
  • A rollback-on-failure path. Right now a failed deploy leaves a broken stack until the next merge. Worth doing once I trust the rest of the pipeline to be boringly stable.

None of these are needed yet. The step has done about 30 deploys across two projects without manual intervention. That’s good enough for now.

The full step

The active code is maybe 60 lines. The other 70 are comments explaining the envSecrets pattern, the writeFile workspace-relative behaviour, and the two-phase build/up split, because those are the points that bit and I want the next person (including me, six months from now) to land on the comment instead of repeating the discovery.

The pitch for putting it in the shared library is the same as for claudeReview: fix the bug once and every project that calls the step picks it up on its next build. The two projects using it now share the same envSecrets fix, the same SSH quoting fix, the same exclude list. If I add a third tomorrow, it inherits all of them by default.

Worth the half-day it took to get right.