How to check agent skills in CI: validate, lock, sync

Skills are code now. A SKILL.md folder tells Claude Code or Codex how to run your migrations, write your release notes or review your pull requests. If someone breaks the frontmatter, the skill stops loading. If someone tweaks the committed copy by hand, the agent acts differently on the next run and nobody notices.

CI can catch both. This guide sets up three checks: every skill in the repo is valid, every installed skill matches a lockfile, and skill updates arrive as reviewable pull requests. Every command and exit code below was run with Skillpack CLI 0.3.5 on October 30, 2026, in a fresh git repo with one public skill installed for Claude Code and Codex.

What can go wrong with skills in a repo

Once a team commits skills to a repo, the same three failures keep coming back:

• A broken skill. Someone edits SKILL.md and deletes the YAML frontmatter, or renames the skill to something with spaces. The agent may simply stop picking it up. Nothing errors.
• Drift between copies. Claude Code reads .claude/skills/, Codex reads .codex/skills/, and a shared copy can live in .agents/skills/. Someone fixes a typo in one of them. Now two agents follow two different instructions.
• Unreviewed updates. A skill installed from GitHub gets refreshed on someone's laptop and the new version lands in a big commit. Nobody reads what the skill now tells the agent to do.

None of these break the build. That's exactly why you want a check.

Step 1: validate every skill you write

If your repo contains skills you author, validate each folder before it merges.

skillpack skills validate ./skills/release-notes

On a valid skill it prints the name, the file count and a checksum, then exits 0:

checksum: "sha256:38e3bae7..."
files: 1
name: "release-notes"
serverValidation: "performed during publication"
valid: true

We broke two folders on purpose. Here's what came back:

Add --json when you want a machine-readable result. The broken folder returned {"code":5,"error":"SKILL.md needs YAML frontmatter","ok":false}.

Note the serverValidation line. The local check covers the folder and its frontmatter. More checks run when you publish the skill to a Skillpack workspace, so passing locally doesn't mean you're done.

To validate every skill folder in the repo, loop over each SKILL.md:

set -e
for f in $(git ls-files 'skills/**/SKILL.md'); do
  skillpack skills validate "$(dirname "$f")"
done

Point the pattern at wherever you keep source skills. Skip the copies that the CLI installs into .claude/skills/ and .codex/skills/; those are covered by the next step. If you're writing your first one, how to create a Claude Code skill covers the frontmatter fields.

Step 2: install skills with a lockfile and commit it

For skills you install rather than write, use a project install:

skillpack install github:anthropics/skills/skills/webapp-testing \
  --scope project --tools claude-code,codex --yes

In our test repo this wrote:

• .agents/skills/webapp-testing/ (the shared copy)
• .claude/skills/webapp-testing/ for Claude Code
• .codex/skills/webapp-testing/ for Codex (more on paths in how to use skills in Codex)
• .companion/skills.lock.json

The lockfile records the source repository, path and commit, a package checksum, and one entry per target folder with its own checksum. For a GitHub skill it also lists the executable files it found, here scripts/with_server.py.

--yes accepts the "not verified by Skillpack" warning for third-party GitHub skills without prompting. Read that warning once before you add the flag. It names the commit and the executable files, and it reminds you that the skill's instructions can make your agent run commands. Skillpack only copies files; it never runs package scripts.

Commit the skill folders and .companion/skills.lock.json together. Leave out the temporary .skillpack-stage-* and .skillpack-backup-* files. From here on, the lockfile is the record of what your agents are allowed to load. For the wider team setup, see how to share Claude Code skills with your team.

Step 3: fail CI on drift with frozen sync

If you only add one check, make it this one:

skillpack sync --frozen

It compares every committed copy with the lockfile. Here's what we saw:

Two things to notice.

First, drift fails loudly and touches nothing. The edited file and the lockfile stay as they were, so you can see the change in the CI logs and fix it in a commit. Frozen sync never rewrites the lockfile.

Second, a missing copy doesn't count as a failure. Sync puts it back from another verified copy and exits 0. That's handy on a laptop. In CI it can hide a commit that forgot a folder. If you want CI to fail on that too, check the working tree right after the sync:

skillpack sync --frozen
test -z "$(git status --porcelain)"

git status --porcelain lists both edited and newly created files, so a restored folder shows up even though it was never committed.

Or read the JSON output and fail when anything was restored:

skillpack sync --frozen --json | jq -e '.restored == 0 and .status == "verified"'

On our clean repo the JSON was {"dryRun":false,"lockChanged":false,"restored":0,"runtimePrerequisites":"not_verified_offline","status":"verified","targets":3}.

runtimePrerequisites: "not_verified_offline" means sync checked the files, not whether the tools a skill needs (Python, a browser, an API key) exist on the runner. If a skill depends on them, test that separately.

Sync runs offline when at least one verified copy is in the repo. If every copy of a skill is missing, it has to download the locked version, which needs credentials for your Skillpack organization. Pass SKILLPACK_API_KEY through your CI provider's secret store, never in the workflow file.

A GitHub Actions example

Here's a minimal workflow with the two checks. The first step installs the Skillpack CLI with the official installer.

name: skills
on:
  pull_request:
    paths:
      - "skills/**"
      - ".agents/skills/**"
      - ".claude/skills/**"
      - ".codex/skills/**"
      - ".companion/skills.lock.json"

jobs:
  check-skills:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Install Skillpack CLI
        run: |
          cd "$RUNNER_TEMP"
          base=https://github.com/The-Vibe-Company/skillpack/releases/download/runtime-v0.3.5
          curl -fsSLO "$base/install.sh"
          curl -fsSLO "$base/SHA256SUMS"
          grep ' install.sh$' SHA256SUMS | sha256sum -c -
          sh install.sh
          echo "$HOME/.local/bin" >> "$GITHUB_PATH"
      - name: Validate authored skills
        run: |
          set -e
          for f in $(git ls-files 'skills/**/SKILL.md'); do
            skillpack skills validate "$(dirname "$f")"
          done
      - name: Frozen sync
        env:
          SKILLPACK_API_KEY: ${{ secrets.SKILLPACK_API_KEY }}
        run: |
          skillpack sync --frozen
          test -z "$(git status --porcelain)"

The paths filter keeps the job off pull requests that don't touch skills. Exit codes do the rest: any non-zero code from validate or sync fails the step.

Step 4: update skills through pull requests

Locked skills don't update themselves. To see what's behind, run a dry run:

skillpack update --all --dry-run

It lists each skill it would update and changes no files. A weekly scheduled job can run it and post the result, or a teammate can run it locally. Then apply the update, commit the skill folders and the lockfile in one pull request, and let the frozen sync job check it. The diff shows which instructions changed. That's the part worth reading: a new line in a skill is a new thing your agent will do. Pinned skills stay where they are unless you choose to move them.

Which check do you need?

Start with frozen sync. It's one line, it runs offline, and it catches the failure that costs the most time to debug: an agent following a skill nobody agreed on. When you want those skills versioned across several repos, a free Skillpack workspace keeps them in one place.