From two hours to ten minutes: automating deployment safely

Map decisions before automating commands

A manual runbook mixes repeatable commands with human judgment. Compiling the application and running the same test suite should produce the same result every time, so the pipeline should own that work. Deciding whether a database change remains backward compatible requires context and an accountable human.

For each stage, I first wrote what it accepts. I then wrote the result it must produce and the condition that should stop it. Finally, I named the person who can approve an exception. Automating an undocumented sequence would have made it faster without making it safer.

I rejected the idea of hiding the whole runbook inside one large script. That design would make a partial failure difficult to locate and would leave a reviewer unsure about what had already changed. Named stages preserve the decision points. They also let the pipeline stop before production access is granted.

Make the delivery contract visible in stages

Stages are useful because they create named evidence boundaries. A production deployment should never rebuild an artifact that passed tests earlier; it should promote the identified artifact. Environment-specific configuration is injected at deployment time from controlled resources, not copied into the repository or typed into a terminal.

The skeleton below communicates the dependency graph. It is intentionally incomplete: production approvals and checks belong to protected Azure DevOps environment resources, where editing pipeline YAML alone cannot silently remove them.

Automexia</>YAML⧉Copy

Simplified multi-stage delivery shape

stages:
  - stage: Validate
    jobs: [unit_tests, integration_tests, security_checks]

  - stage: Package
    dependsOn: Validate
    jobs: [build_once, publish_artifact, render_helm]

  - stage: Deploy_Staging
    dependsOn: Package
    jobs: [deploy_exact_artifact, smoke_test]

  - stage: Deploy_Production
    dependsOn: Deploy_Staging
    environment: production
    jobs: [deploy_bounded_change, verify_rollout]

  - stage: Verify_Outcome
    dependsOn: Deploy_Production
    jobs: [journey_check, publish_release_evidence]

SourcesMicrosoft Learn

Use gates that answer different failure questions

More tests do not automatically create a safer release. Each gate should rule out a different class of failure. Unit tests protect local decision logic. WireMock and Testcontainers exercise integration behavior with realistic dependencies. Contract tests protect service boundaries. Playwright protects a small number of critical journeys. Re-running identical tests at every stage only increases time without adding evidence.

A gate must also fail clearly. The release record should name the failed check and the exact release that was running. It must also say whether production changed before the failure. A red stage with thousands of ungrouped log lines recreates the same cognitive load as the manual process.

• Fast gate: rejects code that cannot compile or pass its unit tests.
• Dependency gate: starts isolated dependencies and proves that the service can communicate with them.
• Contract gate: proves that a new build still speaks the interface expected by existing callers.
• Journey gate: runs a small synthetic payment through the behavior users depend on.
• Deployment gate: shows the exact image and configuration change before promotion.

Keep production authority outside the pipeline file

A contributor who can edit YAML should not automatically gain the authority to deploy to production. I attach approval rules to a protected Azure DevOps environment. The environment is a separately controlled deployment target, so its policy does not live in the pipeline file. A pull request can change the workflow, but it cannot silently remove the policy that protects production.

The service connection is scoped to the environment it must change. Secrets stay in protected storage and appear in the pipeline only as references. An approval is useful when the reviewer can see the exact artifact and configuration difference. Without that evidence, the approval only adds waiting time.

SourcesMicrosoft LearnMicrosoft Learn

Verify the image before checking the rollout

An image tag such as release-latest can move to different content. An image digest is derived from the image content and identifies one exact image. The pipeline should therefore carry the approved digest into production and compare it with the image declared on the Deployment.

Both commands below are read-only and require namespace-scoped access. The image comparison happens before rollout validation because a healthy rollout of the wrong artifact is still a failed release.

Step 01

Read the exact image declared on the Deployment

Ask Kubernetes for the image assigned to the application container. In a multi-container pod, the container name must be explicit so a sidecar image is not compared by mistake.

AutomexiaUbuntu-24.04⧉Copy

λ

kubectl -n '<namespace>' get 'deployment/<service>' -o jsonpath='{.spec.template.spec.containers[?(@.name=="<container>")].image}'

Sanitized output✓  137ms

<registry>/<service>@sha256:<deployed-digest>

Step 02

Wait for Kubernetes to finish the rollout

Only after the digest matches should the pipeline wait for the Deployment rollout. The timeout must reflect how long this service normally needs to become ready.

AutomexiaUbuntu-24.04⧉Copy

λ

kubectl -n '<namespace>' rollout status 'deployment/<service>' --timeout='<agreed-timeout>'

Sanitized output✓  46.2s

deployment "<service>" successfully rolled out

SourcesKubernetesKubernetes

Design rollback and partial failure before promotion

Retries are safe for reads and for steps that are genuinely idempotent. An idempotent operation reaches the same state when it is repeated, so a lost response does not turn the retry into a second change. Retries are dangerous when a write may have succeeded but its response was lost. Before repeating a deployment task, the pipeline reads the current external state and checks whether the declared artifact is already active.

Helm keeps release revisions, but rollback is not a universal undo. A database migration may have changed data that the old application cannot read. I therefore expand the schema before using new fields and postpone removal until the old release is outside the rollback window. The same compatibility question applies to messages and external side effects.

• Retry: a read-only step is usually safe to repeat. A write must be idempotent or protected by an idempotency key before the pipeline retries it.
• Rollback: when the previous application and configuration remain compatible with current state.
• Compensate: when an external side effect cannot be erased but an explicit counter-action exists.
• Stop for a human: when the pipeline cannot prove what state production is in.

SourcesHelmKubernetes

Measure the result without overclaiming it

The result I can substantiate is deployment duration: approximately two hours manually and approximately ten minutes through the automated path, a reduction of roughly 92 percent. That improvement matters because feedback arrives in the same working session and the release no longer depends on reconstructing a long sequence from memory.

I do not have a publishable baseline for change-failure rate, so I will not claim a reliability improvement by percentage. The next version should measure how long each release waits and how long it executes. It should also record when a person intervenes or a post-deployment incident follows.

Automation earned trust because it made the release state easier to inspect. The speed was the consequence.