Distributed job scheduling: the schedule is the easy part

Anyone building a job scheduler first misjudges which part of it is hard. Parsing cron expressions and computing the next execution time is a solved problem with libraries in every language. The hard part starts afterwards: when several machines take on work and one of them fails.

With Croniq we are building a distributed scheduler in Rust, currently at v0.31.0, Apache-2.0. The interesting problems in recent releases had nothing to do with schedules.

Pull instead of push

The first architectural decision was that runners fetch work rather than being assigned it. Runners ask the server over HTTP long-poll whether something is waiting for them. The server knows each runner's capabilities and only hands out matching jobs.

That costs a little latency and saves a lot of operations. Runners need no reachable address, no inbound firewall rules and no registration up front. You start another container and it picks up work. Scaling becomes a question of how many processes are running, not how distribution is configured.

What happens when a runner disappears

This is where the actual work sits. A runner claims a job, marks it as claimed, and then dies. The process is gone, the record says "running". Without a countermeasure that job stays in this state forever and, for singleton jobs, blocks every further execution.

The fix needs three parts that have to work together:

1. Reap orphaned executions. Claimed executions without a live runner have to be detected and released so another runner can take them over.
2. Take over identity on restart. When the same runner comes back it should reclaim its old identity instead of appearing as a newcomer. Otherwise the runner list grows with every restart.
3. Detect flapping. A runner that disappears and returns every minute is a symptom. These takeovers are audited and the counters for them sit on the metrics endpoint so an alert can react to them.

The third point is the one you forget in the first design. A system that repairs itself without reporting it hides exactly the information you need in production.

Calendar gating has a quiet trap

Jobs can be bound to a calendar: working days only, not on public holidays, only within certain periods. The naive implementation computes the next time from the schedule and discards it if the calendar forbids it.

The consequence is that across a longer blocked period a job exhausts its candidates and then stops firing altogether. The correct behaviour is to advance the trigger to the next allowed instant instead of discarding. The difference does not show up in tests, because test calendars have short blocks. It shows up over Christmas.

When something like this is worth it

A custom scheduler is rarely the right answer. Anyone with a handful of cron jobs on one server does not need one. It becomes sensible in a narrow band: roughly twenty to two hundred scheduled jobs spread over several machines, without a dedicated platform team that wants to run a large workflow engine.

Croniq covers exactly that band: one binary or one container, SQLite as the default backend, a dashboard for jobs and execution logs. The point is less the tool than the insight behind it. In distributed execution the effort does not sit in the schedule but in the question of what the system does when part of it stops answering.