Daily releases without lost jobs: maintenance mode in CI

One release a week can still be rolled out at three in the morning, when hardly any job is running. One release a day cannot. Shipping happens during the day, and it meets an application that is busy synchronising, reconciling and calculating. Every deploy restarts services while background jobs are running or just coming due.

We operate a .NET application with many recurring jobs that ships almost every day. The jobs run on Croniq, our open-source job scheduler written in Rust. This article describes how the deploy window became an ordinary, predictable step of the pipeline there.

The deploy window is an operating state of its own

During a deploy, three things happen that a scheduler has to keep apart:

1. Running executions are hit by the restart. If a runner is stopped mid-run, a half-finished run is left behind, and a later retry starts from scratch.
2. Due executions are handed to a stack that is still starting up. At best they fail and go into retry, at worst they meet a mix of old and new version.
3. Changed jobs no longer match the state. If a release changes what a reconciliation job has to do, the old behaviour stays in place until that job's next scheduled run.

A scheduler that simply ignores the deploy treats all of this as random failures. It is better to tell it explicitly: we are shipping now.

Maintenance mode: freeze dispatch, do not stop the scheduler

Croniq has a global maintenance mode for this (PUT /v1/maintenance, admins only). While it is active, the scheduler creates no new work and runners get nothing when they poll. The server itself keeps running, accepts triggers and answers requests.

In the pipeline this becomes a fixed sequence of four steps:

1. Switch on. Before the deploy, CI switches maintenance on, with a note containing the version being shipped and a link to the pipeline run. Anyone looking at the dashboard later sees right away why nothing is running.
2. Wait. CI polls GET /health until no execution is running any more (running is zero), with a fixed upper limit on the waiting time. Running jobs get to finish their work cleanly.
3. Ship. Only now are the containers restarted, and the pipeline waits for the application's health check.
4. Switch off and verify. Once everything is healthy, CI switches maintenance off and then reads the state once more. The deploy only counts as complete when Croniq confirms that maintenance is inactive.

What matters is what happens to work during maintenance. Croniq deliberately distinguishes:

• Running executions finish normally. A retry they schedule is queued and held.
• Work already queued, manual and event-driven triggers stay in the queue and are handed out as soon as maintenance ends.
• Scheduled fires are skipped. The schedule moves on, nothing is caught up.

Skipping is intentional. A maintenance window of several hours must not end in an avalanche of every missed run. The flip side: a daily job at 02:45 does not run that day if maintenance stays on overnight. That is why Croniq logs every skipped fire and counts it in a Prometheus metric per job. After the window it is clear which jobs should be triggered by hand if needed. For a deploy of a few minutes, this usually only affects high-frequency jobs whose next run is coming shortly anyway.

When the deploy fails

If a deploy fails, maintenance deliberately stays on. This is fail-closed: as long as nobody knows what state the stack is in, no jobs should run against it. To keep this from becoming a silent permanent state, there are three signals:

• GET /health reports maintenance without authentication, next to the status. The status itself stays ok, so no health check restarts the server just for doing what it was told.
• An alert rule maintenance_active fires when maintenance has been on longer than a set time. For us that is 30 minutes, reported through the same channels as failed jobs.
• The pipeline's failure path checks at the end whether maintenance is still on and writes that prominently into the run summary.

The next successful deploy ends maintenance again, including a rollback to the previous version. Anyone who does not want to wait switches it off in the dashboard.

A detail that is easy to miss: during maintenance, Croniq suspends the liveness alerts, meaning the rules for missed, late and blocked runs. Late fires are the whole point during a deploy, not a scheduler fault. Without this rule, every slightly longer deploy would raise a false alarm for every active job.

Errors before the server: the configuration gate

All jobs, schedules and alert rules live in the Croniqfile, a dedicated configuration language. Croniq treats unknown directives as errors, not warnings. That is right, but it has a consequence: if the Croniqfile uses a directive that the shipped server does not know yet, the server rejects the whole file and does not start. If the application depends on the scheduler via health check, it does not start either.

That is why CI runs a dedicated gate. It starts exactly the Croniq image that will ship and runs croniq validate on the Croniqfile inside it. It also checks that every place where the Croniq version is pinned points to the same tag. An error therefore surfaces in the pull request and not during the deploy on the server.

The gap after the deploy: run_on_register

That leaves the third point from the beginning: jobs whose definition changes with the release. A reconciliation job that runs daily at 01:30 only picks up changed logic the following night.

The run_on_register directive handles this. A job marked with it runs exactly once when Croniq adopts its definition: the first time, and whenever the job's compiled definition changes. An ordinary restart does not trigger it. If the adoption falls into maintenance, the run is held and starts as soon as maintenance ends, which is exactly when the new stack is healthy.

IaC first, but with an emergency exit

Behind all of this is one principle: the desired state lives in the Croniqfile, versioned in Git and reviewed like any other code. In the dashboard, jobs from the Croniqfile are marked and cannot be edited directly, because a change there would be silently overwritten on the next reload.

In operations you still need ways to intervene that cannot wait for the next deploy. Croniq separates these cleanly:

• Operational actions do not change the definition but the behaviour right now: trigger or cancel a job manually, replay a dead letter, toggle maintenance.
• Alert overrides snooze a rule for a set time, disable it or change its throttle window. Each requires a reason that is recorded in the audit log, and time-limited overrides expire on their own.
• Adoption is the deliberate exit from the Croniqfile for a single job. It has to be allowed server-wide, makes the job editable through API and dashboard, and can be reversed. After that, the definition from the Croniqfile applies again.

This way the Croniqfile stays the reliable source, without the admin having to wait for a pull request in an emergency.

What remains

Daily releases are not a scheduler problem, as long as the scheduler knows that a release is in progress. Maintenance mode turns the deploy window into an explicit state with clear rules, the gate moves configuration errors into the pull request, and run_on_register closes the gap afterwards. All three came out of production use and are available to everyone in Croniq.