Using Turborepo in GitHub CI

[00:00 - 00:20] In this lesson, we're going to look into using Turborepo in our GitHub CI. I'm going to open .github/workflows/ci.yaml,And we can see that in our CI job we're using pnpm's recursive runner to run our "test", "lint" and "build" scripts.

[00:21 - 00:59] If we look at our package.json, we can see that in one of the previous lessons, we added the CI script that runs these in parallel with the concurrency of half of your CPU cores, and we're going to use the same approach for CI. Instead of running everything one by one with "if: always () ", which means run the next step in the CI if previous one fails. What we're going to do is run "pnpm turbo run test:types lint test --continue".

[01:00 - 01:12] So this means execute in parallel the "test:types", the "lint" and the "test" commands. And if any fails, do not stop but continue executing, so we can get rid of these three.

[01:13 - 01:34] And for build we're going to do again "pnpm turbo run build". Now this command no longer fails, and this command I also marked as "--continue", so if it fails on any step we will still try to build the next ones.

[01:35 - 02:01] And just a reminder, the reason for that is that in CI, we do want to try executing everything that we can try to execute and see all the failures at once rather than doing multiple commits, pushing to GitHub, waiting for the action to run again, only to see one additional failure that we weren't aware of. We're not going to use the "--concurrency" option in this case because we are running on GitHub's normal CI runners.

[02:02 - 02:13] These CI runners have two cores, and if we don't specify anything for turbo, it's going to run two tasks in parallel. For our case, this is fine.

[02:14 - 02:29] It will depend on your exact built environment, and if you're using large runners, if you should use a concurrency setting. This is something that you would need to test yourself and see which concurrency setting behaves the best for your particular setup.

[02:30 - 02:39] Trying to optimize that is worthwhile, but not in the case where we're using the normal two core base runner. So let's create a branch.

[02:40 - 02:46] I'm going to call it "add-turbo-to-ci". Let's add our changes which is just the change to our "ci.yaml".

[02:47 - 03:07] Commit this as "Update CI to use turbo" and push this "add-turbo-to-ci". I'm going to create a pull request, and we can go to our CI workflow and see it execute. And as you can see, it did execute both tasks in parallel.

[03:08 - 03:20] But because we are on CI, if you run this again, we're not going to have any cache available. Locally, when we execute turbo, it writes into a ".turbo" folder.

[03:21 - 04:19] If I do "tree .turbo | less", because there are lots of files in there, we're going to see that it is a file cache of manifests and cached outputs of tasks based on their hashes. This means that we can change our CI to actually restore this cache. To do this, we can add a turbo cache step that uses "actions/cache" and caches the ".turbo" path with a key that's "${{ runner.os }}-turbo-${{ github.sha }}", which is the commit hash, and it restores just based on "${{ runner.os }}-turbo-". What this means is every time we write to the cache, the key is going to be, let's say "linux-turbo-123", while we restore based on the most recent key that matches just the ["linux-turbo-" part].

[04:20 - 04:35] With this change in place. With this change in place, I'm going to commit this as "Add turbo cache using actions/cache". Then I'm going to push to my "add-turbo-to-ci" branch.

[04:36 - 04:49] And if we go to the pull request we now have a "Turborepo cache" action. Now the first time it runs it's not going to match any items.

[04:50 - 05:06] No cache has been found for input "linux-turbo- (the hash of this commit) ", neither for the prefix. We have both actions execute, so our tests execute in 13 seconds and then the build executes in 24 seconds.

[05:07 - 05:20] If I rerun this action, there is now going to be... There it is, so there is now going to be a cache hit for this commit.

[05:21 - 05:42] And it restores completely, so we get full turbo on both our builds and our tests. And that makes our execution time go from about a minute to 19 seconds. And you can imagine how this win will be larger the larger the project is.

[05:43 - 06:19] However, this is a very naive cache because it's only based on the prefix and changes with each commit, if you have two branches that are making different changes, your cache hits are going to be pretty bad because we're going to get the cash from the latest action with the "linux-turbo-" prefix, and it's going to be random if that is helpful for our particular changes or not. If we have a branch that changes any of the root inputs to turbo, which is stuff like your pnpm-lock.yaml file, you're always going to have cache mismatches.

[06:20 - 06:33] Still, this is better than nothing. If you don't want to use Vercel or you don't want to set up your own Turbo Cache server, doing this is usually going to be a minor speed improvement.

[06:34 - 06:46] However, if we want to do it correctly, we should do it with Vercel's remote caching. If I go to Vercel, I can go under settings, and here I can get my Team ID.

[06:47 - 06:59] Then I'll go back to GitHub actions and under Secrets and Variables for actions I'm going to add a new variable. I'm going to name that variable "TURBO_TEAM" and copy my team value.

[07:00 - 07:18] And then I need a secret token. To do that. I need to go under my account, settings, tokens. And I'm going to name this token "github-actions-token", with no expiration, that is scoped to this project.

[07:19 - 07:32] I'm going to create this. Copy the value, create a new repository secret and name this "TURBO_TOKEN", paste the secret, and then I'm going to return to our workflow.

[07:33 - 08:10] I'm going to comment out our actions/cache cache, and instead what I'm going to do is add "env" of TURBO_TOKEN that retrieves this from that retrieves from "secrets.TURBO_TOKEN", TURBO_TEAM that retrieves from "vars.TURBO_TEAM", and if I've done this correctly and I push this to GitHub, we should see our CI using Vercel's remote cache. Let me go back to my pull request.

[08:11 - 08:23] If I look at my CI action we have removed the actions/cache step. And now we're using remote caching.

[08:24 - 09:08] Because I run some things locally and I have remote caching enabled, turborepo and Vercel are actually going to share the same cache between my remote runs and the CI. Also, for each CI run, we're going to get unique caches generated and since it's not a single shared folder being restored, the way that GitHub actions were, the way that the GitHub actions cache step was doing. We're going to have a lot more cache hits in our CI. In this case, because I had run some of the tasks locally, they got cached and executed here.

[09:09 - 09:33] And as you can imagine, when I rerun the CI it is going to now be fully cached. And there it is: full turbo on both running the tests, lint and type-checking and on running the builds. I'm going to return to the pull request and quickly look at our change.

[09:34 - 09:51] We've added the "env" setup for our job that sets up "TURBO_TEAM" and "TURBO_TOKEN", and we are now running everything concurrently using turbo, except for builds, because those can break the other tasks. I'm going to merge this and then see you in the next lesson.