How to Run Snapshot Tests in CI/CD with Playwright

Posted on Mar 7 • Originally published at browsercat.com Visual tests are a powerful tool, but they're hell to work with in CI/CD. They're slow, flaky, a pain to debug, and difficult to maintain.Getting them going is no easy task. After all, you need some way to store the snapshots between runs, and you need some way to let your pipeline know when it's OK to update the snapshot cache.Thankfully, I took care of all this for you. :)This article solves all of these problems, and lays the foundation for you to take your visual tests to the next level.If you get value here, you'll love my Ultimate Guide to Visual Testing in Playwright. It covers advanced testing techniques, fine-tuning your snapshot configuration, and more.Let's get started...Your CI/CD environment is not like your local environment, and we should account for that in our Playwright configuration.Update your playwright.config.ts with the following options. Feel free to merge these with any other options you've set so far:Here's what's going on:Next, let's create a basic pipeline. It doesn't do everything (yet), but it's a solid foundation to work from.Create a file at .github/workflows/visual-tests.yml:Notice that we're running our tests with the --ignore-snapshots flag. Since we have no way to store snapshots between test runs, this is literally the only way to get our tests to pass.If you push your repo to Github now, this workflow will run, and it will pass, but it won't guarantee anything about your app.Let's remedy that...For our visual tests to work, we'll need to store snapshots between runs. For this, we'll cache them using Github's "artifacts" feature.Second, let's also write a step that generates new snapshots if they don't exist. This will solve the problem of our tests failing the first time they run on a branch.Third, let's stop ignoring our snapshot assertions.Here's the updates:Notice that our cache key includes the current branch as well as a fallback to the master branch. That way a new branch has some material to work from on the first push.If you trigger the pipeline now, on the first run, you'll see snapshots generated, the tests pass, and the cache updated. On subsequent runs, you'll see the cache hit, no new snapshots generated, and the tests still pass.Not bad, but what happens when the snapshots need to be updated?Now that we have visual tests that really work, let's change our workflow so that we can update snapshots on demand.First, let's parameterize our workflow...workflow_call allows us to manually trigger the workflow with or without updating snapshots. workflow_dispatch allows other Github workflows the same set of options.Here's what the manual widget looks like:Let's also update "Initialize snapshots" to trigger an update when this parameter is enabled:Lastly, let's upload our HTML report as an artifact, so that we can reference it before deciding if we should ask the workflow to update snapshots... or if we need to debug the problem:Notice that our "Test" step now enables continue-on-error, so that we can upload our report even if the tests fail.The new "Upload test report" step uploads the report as an artifact. In the workflow results, you'll see a direct link to a zip file containing our report.Things are going great, but there's one more thing we can do to make visual tests even better: make them blazingly fast!You've probably noticed two steps are slowing your pipeline down:Thankfully, we can fix both of these issues, and most-likely for free!BrowserCat hosts a fleet of Playwright browsers in the cloud that you can connect to with just a few lines of code.If you do so, you won't have to install browsers in your CI/CD environment, and you'll be able to run your tests entirely in parallel.And the best part? BrowserCat has an awesome forever-free plan. Unless you're running a huge team, you'll probably never have to pay a dime.So let's get started!First, sign up for a free account at BrowserCat.Second, create an API key and store it as a secret named BROWSERCAT_API_KEY in your Github repository. You can do this by navigating to your repository, clicking "Settings," then "Secrets," then "Actions," then "New repository secret."Third, let's update playwright.config.ts to use BrowserCat:In the updates above, you'll notice we've increased parallelization to 10 workers when using BrowserCat. This is a good starting point, but you can increase it significantly more without issue.Notice that this config connects to BrowserCat whenever BROWSERCAT_API_KEY is defined. This is useful for running tests locally, as well as in CI/CD.OK, now here's the last bit of magic... Let's stop installing Playwright's browsers every time we run our pipeline:If you run your pipeline now, you'll witness a dramatic speed-up, even with just a single test in your suite. And with 1000 free credits per month, you can run your pipeline for hours on end with plenty of time left over.You've got your CI/CD pipeline working. Now it's time to write some tests!Be sure to check out my Ultimate Guide to Visual Testing in Playwright for advanced snapshot testing strategies and ready-made solutions for running visual tests in CI/CD.In the meantime, happy testing!Templates let you quickly answer FAQs or store snippets for re-use. Are you sure you want to hide this comment? It will become hidden in your post, but will still be visible via the comment's permalink. Hide child comments as well Confirm For further actions, you may consider blocking this person and/or reporting abuse Karen Payne - Feb 24 Clifton Beale - Feb 1 Gregor Schafroth - Feb 1 Mohamed Jubair - Feb 29 DEV Community — A constructive and inclusive social network for software developers. With you every step of your journey. Built on Forem — the open source software that powers DEV and other inclusive communities.Made with love and Ruby on Rails. DEV Community © 2016 - 2024. We're a place where coders share, stay up-to-date and grow their careers.