How to benchmark GitHub Actions runners fairly
Compare CI runners on the same YAML, warm caches and repeated runs, measure cost per job instead of rate per minute, and see what our benchmark shows.
Guzman Pintos
Product at Luxor Agent Cloud
Updated 9 min read

Every runner provider publishes a chart where its bar is shorter than GitHub's, and so do we. Those charts are hard to compare with each other, because a runner benchmark has a lot of knobs: which workflow, which machine size on each side, whether the cache was warm, how many runs, and whether "cheaper" means the per-minute rate or the bill for the job.
This post covers the choices that make a runner comparison fair, how we ran ours, what the numbers show, and a workflow you can copy to test your own jobs on both runners.
What a fair runner benchmark controls for
Run the same YAML on both sides and change only runs-on. If one side gets a custom cache action, a prebuilt image or a trimmed test suite, you're comparing configurations instead of runners.
Then warm the caches. A first run on a new runner downloads everything and fills an empty cache, and since your normal CI doesn't run like that, a cold-cache number overstates the cost of every job. Tenki hosts its own drop-in Actions cache (caching), so a cache warmed on GitHub is empty on Tenki and the reverse. Run each side at least once before you start recording.
Repeat the runs, too, because a single run tells you little. For short jobs the noise can be bigger than the difference you're measuring: a job that takes 5 seconds on one runner and 3 on another differs by two seconds, and a slow package download or a busy host can move it by that much.
State the machine size on each side plainly. Two vCPUs on one provider and four on another is a legitimate comparison as long as you say so, but it answers a different question than an equal-core test does (more on that below).
Finally, compare cost per job rather than rate per minute. The number that ends up on your bill is duration × per-minute rate, with each provider's rounding rules applied. A runner with a higher rate can still cost less per job if it finishes fast enough, and a lower rate doesn't help if the job runs longer.
How we ran ours
The benchmarks page has the full setup. In short:
- The Tenki side ran on
tenki-standard-large-8c-16g, 8 vCPU and 16 GB on bare-metal x64 Linux, with one fresh VM per job. - The GitHub side ran on GitHub's 8-core larger runner, 8 vCPU and 32 GB. Kubernetes ran on 16 vCPU on both sides.
- Each workload is the project's own CI job or standard build at a pinned commit, with identical steps on both sides and only
runs-onchanged. - Dependency caches were warm on both sides. Build caches (sccache, Docker layers, the Gradle build cache) were off, so every run compiled in full.
- Times are the whole job as GitHub reports it, including setup and checkout.
- The workloads are builds of well-known open-source projects: PostHog's production Docker image, Node.js, the Linux kernel, Apache Kafka, Spring Boot and Kubernetes.
- The suite ran several times on November 11, 2026, and each row is the project's fastest run relative to GitHub among runs where both sides succeeded.
If you reproduce this, decide up front which number you report: the best run, the last warm run or the median of the warm runs. They can differ a lot. Across our runs, the Node.js job took between 35m and 38m on Tenki, and between 1h 17m and 1h 50m on GitHub. The median is the more conservative choice. Whichever you pick, say which one you used.
Warm dependency caches don't mean Tenki ran without help. Tenki's dependency mirrors for npm, APT, Go modules, Cargo, Maven, Gradle and Docker Hub are on for every job with nothing to configure (caching). They're part of what you get on Tenki, so they're in the Tenki numbers.
Equal cores or equal price
This benchmark puts both sides on the same core count, so the speed difference comes from the machines, not from extra cores. Tenki charges $0.002 per core-minute (pricing), so its 8-core runner is $0.016/min. GitHub's 8-core Linux runner is $0.022/min, and its 16-core runner is $0.042/min (GitHub's runner pricing). At equal cores Tenki's rate is 27% lower, so it's cheaper even if a job runs at the same speed.
The other fair comparison is equal price: a larger Tenki runner for about what GitHub charges for a smaller one, which buys more cores for parallel work. That answers a different question, whether you can get a faster build for the same spend, and our published benchmark doesn't include it. If that matters for your decision, run both comparisons.
The results
Durations below are from the benchmarks page:
Speedups vary by workload, from 1.6x to 3.2x, so there is no single "Tenki is X× faster" number that applies to your pipeline.
What each run costs
The two providers round differently. GitHub rounds each job up to the next whole minute (pricing reference), so a 7m 1s job is billed as 8 minutes. Tenki bills per second of job runtime (limits and billing). The table shows both views: GitHub prorated to the second, which isolates the rate and speed difference, and GitHub as billed, which is what the invoice shows. Tenki's cost is the same in both.
Each workload here is a single job. GitHub rounds every job in a workflow separately, so a workflow with many jobs is billed more than one rounded total.
On these builds, which all run for minutes, rounding adds at most a few cents, and the billed and prorated columns are close. It matters most for short jobs: on a job under a minute, GitHub charges the full minute however quickly the job finishes, and those jobs add up, since lint, typecheck and each leg of a matrix are billed separately.
Per run, the long builds move the bill the most. The Node.js job saves about $1.85 per run and Spring Boot about $1.07, multiplied by every push that triggers them.
What drives the differences
With the core count equal, two differences remain: the hardware and the download path. This benchmark doesn't separate them.
On hardware, Tenki's x64 runners are bare-metal 5th Gen AMD Ryzen servers, where each pair of vCPUs maps to one physical core (x64 runners), with one VM per job.
Downloads are the other difference. Package installs and FROM pulls go through Tenki's dependency mirrors, so repeat downloads don't cross the public internet, and Docker pulls avoid Docker Hub's anonymous rate limits (caching). How much that contributes depends on how much of your job is downloading.
If you want to isolate one factor, change one thing at a time: switching off the Actions cache for the workspace in the dashboard shows how much your jobs lean on it, and a run on a larger Tenki runner shows how much more your job gains from extra cores.
Benchmark your own workflow
You can run this comparison on your own job in an afternoon. Copy the workflow you want to test into a new file on your default branch (workflow_dispatch only works from there, per GitHub's docs), and turn runs-on into a matrix so both runners run the same steps:
name: runner-benchmark
on: workflow_dispatch
jobs:
build:
strategy:
fail-fast: false
matrix:
runner: [ubuntu-latest, tenki-standard-medium-4c-8g]
runs-on: ${{ matrix.runner }}
steps:
- uses: actions/checkout@v4
# Your existing setup, cache and build steps, unchanged
The matrix value resolves to a single label, which is what Tenki needs: a Tenki label must be the whole runs-on value (sizes). Your repository has to be connected to Tenki first (quickstart).
Run it three times, one after the other, so the later runs find a warm cache on both sides:
for i in 1 2 3; do
gh workflow run runner-benchmark.yml
sleep 10
gh run watch "$(gh run list --workflow runner-benchmark.yml --limit 1 --json databaseId --jq '.[0].databaseId')" --exit-status
done
Then pull each job's duration from the GitHub API and apply each provider's billing rules:
gh run list --workflow runner-benchmark.yml --limit 3 --json databaseId --jq '.[].databaseId' |
while read -r id; do
gh api "repos/{owner}/{repo}/actions/runs/$id/jobs" --jq '
.jobs[]
| ((.completed_at | fromdate) - (.started_at | fromdate)) as $s
| if .labels[0] == "ubuntu-latest"
then [.labels[0], $s, (($s / 60 | ceil) * 0.006)]
else [.labels[0], $s, ($s / 60 * 4 * 0.002)]
end
| @tsv'
done
Each line prints the runner label, the job's duration in seconds, and its cost: GitHub's rounded up to whole minutes at $0.006/min, Tenki's per second at $0.002 per core-minute × 4 cores. Drop the oldest run, since it filled the caches, and compare the two remaining runs on each side.
Adjust the numbers to what you're testing:
- For other sizes, change the
4to the Tenki runner's core count, and use GitHub's rate for the size you're comparing against: $0.012/min for its 4-core Linux runner, $0.022/min for 8-core (pricing reference). - In a public repository, standard GitHub-hosted runners are free (Actions billing), and
ubuntu-latesthas 4 vCPU and 16 GB instead of 2 vCPU (runner specs). The cost comparison above doesn't apply there, and the fair speed comparison is against a 4-core Tenki runner. - For jobs under a minute, run more than three times before trusting a small difference.
Job duration excludes queue time, so these numbers measure the work itself. If time-to-first-feedback matters to you, also compare how long each run takes from dispatch to completion.
Once you have per-job costs, GitHub Actions cost optimization covers turning them into a monthly bill. For the full methodology and current results, see the benchmarks page.



