Sleeping at Scale

One Buildkite Agent doing something once per second (like collecting new log output, splitting it into chunks and preparing them for upload) is not especially interesting. But there are hundreds of thousands of Buildkite Agents connected and running around the world talking to our backend. If enough of them happen to do the same thing at roughly the same time, a harmless little loop becomes a large spike in work for the servers on the other end.

We need the agents to keep a regular pace without marching in step and this has turned out to involve more thought and more kinds of loops than we expected.

To understand what we’ve done to achieve this, we need to take a step back and go back to basics: one program that needs to do a thing over and over again, forever.

Doing a thing repeatedly forever

The simplest way to write that (in Go, but this is hopefully readable to everyone) is:

for {
    doThing()
}

Immediately repeating the action without any sleep in between will probably cause some resource to be consumed as quickly as possible, such as CPU or network bandwidth.

So let’s add a sleep to slow things down:

interval := 1 * time.Second
for {
    doThing()
    time.Sleep(interval)
}

Do the thing, sleep 1 second, repeat.

Question: In the code above, is doThing called exactly once every second?

Answer: No. Aside from the system clock changing, drifting, or otherwise just being inaccurate, the loop does not account for the length of time it takes doThing to do its thing.

Waiting 1 second in between actions leads to the actions happening less frequently than once per second.

This approach was used in the Buildkite Agent (repo) for processing job logs not that long ago!

Let’s be a bit more sophisticated and use a time.Ticker, Go’s recurring timer. time.Tick gives us a stream of ticks, one per interval (e.g. 1 second). If our code falls behind (i.e. doThing takes longer than the tick interval), Go may skip ticks rather than queueing every tick we missed.

interval := 1 * time.Second
for range time.Tick(interval) {
    doThing()
}

The compact for range syntax above waits for each tick and then runs the body of the loop. We can write those same steps explicitly: first store the stream of ticks in tick, then use <-tick to wait for the next one:

interval := 1 * time.Second
tick := time.Tick(interval)
for {
    <-tick
    doThing()
}

This is the same as above. But both implementations have the disadvantage that on the first iteration, we wait for the interval up front before calling doThing. In the time.Sleep approach, we did the thing and then waited.

This is easy to remedy: we can reorder the operations inside the loop.

interval := 1 * time.Second
tick := time.Tick(interval)
for {
    doThing()
    <-tick
}

Now it will do the thing, wait until the next second, do the thing, wait until the next second… and the ticker can keep things as close to on-time as possible, regardless of how long doThing takes to run. That solves it, right? (Right?)

Lots of things doing the thing repeatedly forever

Let’s suppose doThing calls out to some network service, like requesting work, or uploading chunks of job logs. Let’s also suppose that copies of the loop are running in a large fleet around the world.

Ideally we want the load on the servers to be spread across time evenly. This is so that servers aren’t sitting idle (we have to pay for them whether or not they are used, so may as well use them) and to avoid overload (through surges of requests).

We might hope that the load will just naturally spread itself across time evenly. After all, not all copies of the loop will be started at the same time. If that’s true, then we would expect the load to be spread out a bit randomly within the first interval and then remain somewhat periodic (repeating in time), changing in shape as instances are added and removed.

Hope is not a strategy.

From time to time, though, large groups of agents will synchronise. Suppose there is a power outage at a data centre running lots of agents — we can expect they will all start up again at about the same time.

Or suppose (more likely) that there is a network blip that causes a large group of agents to reconnect to Buildkite at the same time, causing the loops to start at the same time as a knock-on effect.

Another reason is clock drift: some system clocks run slightly slower, some slightly faster, and over time they will drift in and out of synchronisation with one another.

Thirdly, a reason to avoid the time.Sleep approach in particular is that when lots of agents do the thing at the same time, the servers will be bogged down trying to handle them all, so they will tend to finish their requests around the same time, and when the interval between them is fixed (with time.Sleep) that means they will all start again at the same time!

(Footnote: Sometimes the opposite happens. Computers are hard!)

We must counteract the tendencies for loops to spontaneously synchronise.

“I know! Let’s wait a random amount of time!” This is called jitter. The way in which jitter is added is important!

Jittering a sleep loop

Suppose for a second we still used time.Sleep in the loop. The easiest way to add jitter (with some big drawbacks that we will get to shortly) is to change the sleep interval to some random amount of time:

interval := 1 * time.Second
for {
    doThing()
    time.Sleep(rand.N(interval))
}

rand.N chooses a uniformly distributed value from zero (inclusive) up to the provided limit (exclusive). Because Go durations are numeric values, passing interval gives us a random duration between 0 and 1 second.

Every duration in the range is equally likely. The midpoint between 0 and 1 second is 500ms, so over many iterations the sleeps average 500ms. A short run, like the one above, can average higher or lower. If we assume that doThing takes no time to run, the loop therefore averages one iteration every 500ms. To make the time between runs average 1 second, we need to double the upper limit:

interval := 1 * time.Second
for {
    doThing()
    time.Sleep(rand.N(2 * interval))
}

Sleeping a random amount between requests, but accounting for the fact that the expected sleep duration is half of the range of the random variable.

There are two things to understand about this approach.

The first is a fact every gambler knows: there are “hot runs” and “cold runs”. In this context, “cold runs” happen when the random number generator (RNG) sometimes generates several high numbers (longer intervals) consecutively, and “hot runs” happen when the RNG generates several low numbers (shorter intervals) consecutively. The behaviour of the random-sleep loop is reasonable on average, but it is inconsistent.

Hopefully, in aggregate, hot and cold runs aren’t a problem for the backend. But from a user experience perspective, while we may be willing to accept a longer interval from time to time, having long intervals repeatedly would suck. In the case of Buildkite agents, they would take longer to pick up new jobs and make new logs available, so we want to avoid that.

The second is a bit more statistical. A single random sleep could be anywhere between 0 and 2 seconds. But as each copy of the loop repeats, its short and long sleeps begin to balance out. The simulation below adds up the time each loop has spent sleeping, then divides it by the number of iterations. After the first iteration the results fill the entire range, but after many iterations they gather around 1 second.

Jitter + Ticker

We can effectively solve the “hot/cold runs” problem by combining a random sleep with the Ticker loop:

interval := 1 * time.Second
tick := time.Tick(interval)
for {
    time.Sleep(rand.N(interval))
    doThing()
    <-tick
}

First, we sleep a random amount of time. Then we do the thing. Then we wait for the ticker. And then return to the top of the loop.

The effect of this is: for each time interval, doThing will happen at a random time during that interval. This may be slightly easier to see if we rearrange the loop and introduce a little extra channel that exists solely to skip the ticker on the first interval:

interval := 1 * time.Second
tick := time.Tick(interval)
first := make(chan struct{}, 1)
first <- struct{}{}
for {
    // Wait to begin on each tick, except for the first iteration,
    // which begins immediately.
    select {
    case <-tick:
    case <-first:
    }

    // Wait a random amount of time.
    time.Sleep(rand.N(interval))

    // Now do the thing.
    doThing()
}

This example is very close to how the Buildkite Agent log processing loop was implemented for most of 2025, after a January change introduced jitter and ticker-based timing.

The “hot/cold runs” problem is harder to run into here. Suppose the RNG produces a string of larger intervals (close to 1 second, in this example). Because of the ticker, the Nth loop iteration always begins N seconds after the loop begins (and if not, the ticker internally adjusts). So while a “run” of long intervals might introduce a longer than average gap immediately before the run, within such a run the inter-event time is still around 1 second!

interval := 1 * time.Second
for {
  doThing()
  time.Sleep(rand.N(2 * interval))
}

interval := 1 * time.Second
tick := time.Tick(interval)
for {
  time.Sleep(rand.N(interval))
  doThing()
  <-tick
}

The worst-case performance is now an alternating long/short pattern. Suppose the RNG rolls 1s, 0s, 1s, 0s… In that case, there will be alternating gaps of ~2s and ~0s. Not ideal, but the average load in terms of backend requests is consistently 1 per second.

But why jitter the whole interval?

With the jitter + ticker loop, we’ve greatly mitigated the problems with the random-sleep loop, but we still have the worst-case of alternating long/short patterns, meaning we could potentially have as little as 0x or as much as 2x the chosen interval between requests.

Can we reduce this variability somehow? We certainly want to — consider the 0x case. If the loop exists to repeatedly request jobs to run, we could model the expected number of jobs available as being proportional to time spent waiting. The less time spent waiting, the fewer jobs available. If we generated a 0 interval, then waited 0 time and immediately tried to fetch a new job right away, then we expect 0 new jobs to have been made available in that interval. So such a request is expected to be a complete waste of effort. Similarly, if we model a job as producing a steady stream of log output, creating a new chunk right away would also probably waste all the overheads that go into making the request, because we expect there to be no new log output.

One possible and very tempting solution starts like this:

“Aha! Why not only jitter for half the interval?”

interval := 1 * time.Second
time.Sleep(rand.N(interval / 2))

“Or for some fixed duration that doesn’t vary with the interval?”

// This stays at 100ms even if interval changes.
jitterWindow := 100 * time.Millisecond
time.Sleep(rand.N(jitterWindow))

“That way we can bound the time between requests!”

The downside is, again, the tendency of physical effects such as clock drift or power outages to cause loops to synchronise. Throwing hope out the window again, let’s assume a bad case where the tickers for a large group of loops all fire at the same time. Jittering for half the interval would spread the requests out better than running the request right away… but would still tend to concentrate requests from the large group within one half of the interval, which could be a load problem. The rest of the interval would have fewer requests, leading to under-used server capacity.

Jittering sometimes

Can we do better than that? Perhaps we want the requests to be much more regular most of the time, and we only want to spice things up the minimum amount (let’s call this the “English cooking” of jitter).

At this scale, we can add a little jitter every so often to a fixed ticker loop to keep the agents spread out a bit. A tiny sprinkling.

The idea is, run a fixed ticker loop for some finite length of time, then “re-offset” that loop by a random amount every so often:

interval := 1 * time.Second

// Every 32 iterations, jitter a bit
for {
    // First, jitter to re-offset the inner loop
    time.Sleep(rand.N(interval))

    // Then, start the fixed ticker loop, but only run 32 iterations
    tick := time.Tick(interval)
    for range 32 {
        doThing()
        <-tick
    }
}

This should be pretty resilient to clock drift, as long as the clocks aren’t drifting too quickly. Its average request rate is slightly less than 1 per interval: each batch makes 32 requests over 32 ticker intervals plus a random sleep that averages half an interval.

You’ll notice, though, that this is a fixed-length ticker loop within a random sleep loop — so what about that Central Limit Theorem mumbo-jumbo that applies to random-sleep loops? That’s still valid, and if we wanted to, we could make mathematical claims about how, after a very long time, the timing of the Nth iteration of the inner loop is approximately normally distributed, etc., etc.

If that’s a problem, we can fix it by… using the fixed ticker loop inside another jitter + ticker loop!

interval := 1 * time.Second

// Every 33 _seconds_, jitter a bit
outerTick := time.Tick(33 * interval)
for {
    // First, jitter to re-offset the inner loop
    time.Sleep(rand.N(interval))

    // Then, start the fixed ticker loop, but only run 32 iterations (32 seconds)
    tick := time.Tick(interval)
    for range 32 {
        doThing()
        <-tick
    }

    // Wait for the next outer interval to begin.
    <-outerTick
}

For a real-world implementation of this, see the Buildkite Agent’s log-processing loop!

Sleeping at scale

Jitter is often described as adding randomness to a loop. But where the randomness goes, how much we add, and how often we add it all change the result.

For one loop, those differences might not matter much. But when you run thousands upon thousands of copies of that loop, they determine whether work arrives smoothly or in spikes, and whether each loop keeps a useful pace or sometimes runs twice in quick succession.

Randomness is not useful by itself. We could spread requests perfectly and still make each agent unpleasantly inconsistent. Or we could keep each agent perfectly regular and allow them all to synchronise. The trick is deciding where the irregularity is useful.

At sufficient scale, even sleeping requires some coordination.