Intro
We wished to run a pipeline every five minutes, but when you do the math, this will result in its running more than 1000 times per week, which according to the documentation is forbidden. On the other hand, we are using private agents – our own – so why should Micrsoft put limits on how often we run jobs on them??
The details
Given that there are 10080 minutes in a week, to arrive at fewer than 1000 pipelkine runs per week you’d need to pace out your jobs at no more than run once every 12 minutes. And that’s what I had been doing. So then I would create a second pipelikne running the same code, but running it an the inbetween times to end up with a logical job which runs every six minutes. But is this approach really required for our private agents?
We decided to put this to a real test. I created a Hello World yaml file and ran it every minute. The results are not at all what we expected!
The results
Essentially, the job runs 10 times out of every 15 minutes. This is another published limit. And you see this effect right away. So this is like some kind of burst rate limiting you might say, and it applies.
And during those times when it’s not being run, you don’t see it paused or anything. It simply isn’t run. But you can run it by hand (I think) and it will run.
So then you think, OK, limits apply, even to private agent pools. Then we left it running, and something funny happened.
After about 640 runs in the course of 24 hours, it simply stopped. Then about three days later it started up again, ran about 637 times, then stopped again.
So there seems to be an additional unpublished limit of something like 640 runs in a 72 hour interval.
But, we were able to exceed 1000 runs in a week, for what it’s worth.
Alternatives
I guess we were not using pipelines for what it was intended. It’s not really to be considered cron on steroids. We’ll be looking at Azure Functions to see if it’s a better fit for our requirements.
Conclusion
Even when you use your own agents in your pool, your Azure Pipeline job will be rate limited to about 10 runs per 15 minutes, about 640 runs per three day interval (unpublished limit), though you can exceed 1000 runs per week. These limits prevent you from executing a run every five minutes! If you need to execute a job so often, consider finding a different approach!