claude-mods/skills/testing-ops/references/ci-testing.md

CI/CD Testing Patterns

Testing strategies for continuous integration pipelines.

Test Pipeline Stages

┌─────────────────────────────────────────────────────────────────┐
│                        CI Pipeline                               │
│                                                                  │
│  ┌──────┐   ┌──────┐   ┌───────┐   ┌─────┐   ┌──────┐   ┌────┐│
│  │ Lint │ → │ Unit │ → │ Build │ → │Integ│ → │  E2E │ → │Dep.││
│  │      │   │Tests │   │       │   │Tests│   │Tests │   │    ││
│  └──────┘   └──────┘   └───────┘   └─────┘   └──────┘   └────┘│
│     1m        2-5m       1-3m       5-10m     10-30m      -   │
│                                                                  │
│  ◄─────── Fast Feedback ───────►  ◄─── Comprehensive ──────►   │
└─────────────────────────────────────────────────────────────────┘

GitHub Actions Example

name: CI

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  lint:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: '3.11'
      - name: Lint
        run: |
          pip install ruff
          ruff check .

  unit-tests:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: '3.11'
      - name: Install dependencies
        run: pip install -e .[test]
      - name: Run unit tests
        run: pytest tests/unit -v --cov=src --cov-report=xml
      - name: Upload coverage
        uses: codecov/codecov-action@v4

  integration-tests:
    needs: unit-tests
    runs-on: ubuntu-latest
    services:
      postgres:
        image: postgres:15
        env:
          POSTGRES_PASSWORD: postgres
        ports:
          - 5432:5432
        options: >-
          --health-cmd pg_isready
          --health-interval 10s
          --health-timeout 5s
          --health-retries 5
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: '3.11'
      - name: Run integration tests
        env:
          DATABASE_URL: postgres://postgres:postgres@localhost:5432/test
        run: pytest tests/integration -v

  e2e-tests:
    needs: integration-tests
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Run E2E tests
        run: |
          docker-compose up -d
          pytest tests/e2e -v
          docker-compose down

Test Parallelization

pytest-xdist

- name: Run tests in parallel
  run: pytest -n auto  # Use all available CPUs

- name: Run with specific workers
  run: pytest -n 4  # 4 parallel workers

Matrix Testing

jobs:
  test:
    strategy:
      matrix:
        python-version: ['3.9', '3.10', '3.11']
        os: [ubuntu-latest, macos-latest]
    runs-on: ${{ matrix.os }}
    steps:
      - uses: actions/setup-python@v5
        with:
          python-version: ${{ matrix.python-version }}
      - run: pytest

Sharded Tests

jobs:
  test:
    strategy:
      matrix:
        shard: [1, 2, 3, 4]
    steps:
      - name: Run test shard
        run: pytest --shard-id=${{ matrix.shard }} --num-shards=4

Caching for Speed

- name: Cache pip packages
  uses: actions/cache@v4
  with:
    path: ~/.cache/pip
    key: ${{ runner.os }}-pip-${{ hashFiles('**/requirements*.txt') }}
    restore-keys: |
      ${{ runner.os }}-pip-

- name: Cache pytest
  uses: actions/cache@v4
  with:
    path: .pytest_cache
    key: pytest-${{ github.sha }}
    restore-keys: pytest-

Flaky Test Handling

Retry Mechanism

- name: Run tests with retry
  uses: nick-fields/retry@v3
  with:
    timeout_minutes: 10
    max_attempts: 3
    command: pytest tests/e2e

pytest-rerunfailures

# Rerun failed tests up to 3 times
pytest --reruns 3 --reruns-delay 1

Quarantine Flaky Tests

@pytest.mark.flaky(reruns=3, reruns_delay=2)
def test_sometimes_fails():
    # This test is known to be flaky
    pass

@pytest.mark.skip(reason="Flaky - investigating")
def test_quarantined():
    pass

Test Reports

JUnit XML

- name: Run tests
  run: pytest --junitxml=results.xml

- name: Publish Test Results
  uses: dorny/test-reporter@v1
  if: always()
  with:
    name: Test Results
    path: results.xml
    reporter: java-junit

Coverage Reports

- name: Run with coverage
  run: pytest --cov=src --cov-report=xml --cov-report=html

- name: Upload coverage to Codecov
  uses: codecov/codecov-action@v4
  with:
    files: ./coverage.xml
    fail_ci_if_error: true

- name: Coverage comment on PR
  uses: py-cov-action/python-coverage-comment-action@v3

Branch Protection Rules

# Require tests to pass before merge
# Settings → Branches → Branch protection rules

Required status checks:
  - lint
  - unit-tests
  - integration-tests

Require branches to be up to date: Yes

Test Selection

Changed Files Only

- name: Get changed files
  id: changed
  uses: tj-actions/changed-files@v41
  with:
    files: |
      src/**
      tests/**

- name: Run affected tests
  if: steps.changed.outputs.any_changed == 'true'
  run: pytest tests/ -v

Skip Expensive Tests

- name: Quick tests on PR
  if: github.event_name == 'pull_request'
  run: pytest -m "not slow and not e2e"

- name: Full tests on main
  if: github.ref == 'refs/heads/main'
  run: pytest

Secrets in Tests

- name: Run tests with secrets
  env:
    API_KEY: ${{ secrets.TEST_API_KEY }}
    DATABASE_URL: ${{ secrets.TEST_DATABASE_URL }}
  run: pytest tests/integration

# Use environment for sensitive tests
jobs:
  integration:
    environment: testing  # Requires approval
    steps:
      - run: pytest tests/integration

Best Practices

1. Fast feedback first - Run linting and unit tests before slow tests
2. Fail fast - Stop pipeline on first failure (pytest -x)
3. Parallel when possible - Use matrix builds and xdist
4. Cache aggressively - Pip, node_modules, docker layers
5. Keep tests deterministic - No reliance on external state
6. Isolate flaky tests - Quarantine or fix, don't ignore
7. Report clearly - Use test reporters and coverage comments
8. Secure secrets - Never log, use GitHub secrets