Have you ever encountered the frustration of code that runs smoothly on your local machine but fails unexpectedly in the CI pipeline? The Automation pod found itself in precisely this predicament. New test script changes were not being ran against CI, leading to frequent issues during scheduled jobs.

Running our entire suite of tests locally was out of the question; the time and CPU resources required would impede progress on other tasks. CI does provide a manageable feedback loop of ~15 minutes, but, is it truly necessary to execute all tests?

The Solution

The current setup of the solution is in Github Actions and Playwright, your setup may look different but many CI providers should have identical support!

The Github Actions job that runs these tests is configured to run on pull requests as seen below.

on:
  pull_request:
    types:
      - opened
      - ready_for_review
      - reopened
      - synchronize

Let’s say we have the following changed files. Foo.ts Bar.spec.ts

Get the changed files.

FILES=$(git diff --name-only --diff-filter=d $(git merge-base HEAD origin/main))
echo $FILES
Foo.ts
Bar.spec.ts
  • --name-only gets the full filepath and name.
  • --diff-filter=d filters out deleted files.
  • git merge-base HEAD origin/main ensures we are comparing against the base branch.

Use grep to filter out files matching a pattern and use tr to replace newlines with a space, which is useful for multiple changed specs.

CHANGED=$(echo $FILES | grep ".*.spec.ts" | tr '\n' ' ')
echo $CHANGED
Bar.spec.ts

In Github Actions, this bash script is ran well before the tests, so we need to expose these changed specs to the step or job that needs it.

echo "CHANGED_SPECS=$CHANGED" >> $GITHUB_OUTPUT

OR

echo "CHANGED_SPECS=$CHANGED" >> $GITHUB_ENV

Running the tests

yarn playwright test $CHANGED_SPECS

yarn playwright test Bar.spec.ts

And there we’ve done it! We’re able to run just the changed specs in CI, providing a quicker feedback loop and increased confidence in our changes.