Jobs, Parallelism & Dependencies — CI/CD Pipelines with GitHub Actions

A real pipeline isn't a straight line — it's a graph. Some work can happen simultaneously; other work must wait. This lesson teaches you to orchestrate work for maximum speed and correctness.

The Key Insight: Jobs Are Isolated

Each job runs on a separate, fresh VM. They do NOT share filesystems, memory, or state. This has massive implications:

Jobs are COMPLETELY isolated. No shared state. Each starts from zero.

This means: if job A installs dependencies, job B does NOT have those dependencies. Each job must check out code and install deps independently (or use artifacts to pass data — covered later).

Parallel vs Sequential Execution

By default, all jobs run simultaneously. You control ordering with needs:.

Parallel saves time, but sequential ensures ordering. The art is knowing which to use where.

The `needs:` Keyword

needs: creates a dependency edge in your pipeline graph:

jobs:
  lint:                    # No needs → starts immediately
    runs-on: ubuntu-latest
    steps: [...]

  test:                    # No needs → starts immediately (parallel with lint)
    runs-on: ubuntu-latest
    steps: [...]

  build:
    needs: [lint, test]   # Waits for BOTH lint AND test to pass
    runs-on: ubuntu-latest
    steps: [...]

  deploy:
    needs: build          # Waits for build to pass
    runs-on: ubuntu-latest
    steps: [...]

This produces a Directed Acyclic Graph (DAG):

The pipeline DAG. build waits for the slowest predecessor (test at 45s), not both individually.

Total pipeline time = longest path through the graph, not sum of all jobs. The art of pipeline design is minimising the critical path while keeping dependencies correct.

Common Pipeline Patterns

Four common pipeline topologies. As an architect, you'll combine these.

Passing Data Between Jobs

Since jobs don't share filesystems, you need explicit mechanisms to pass data:

Mechanism	Use Case	Size Limit
Job outputs	Small values (version number, image tag, URL)	~1 MB total
Artifacts	Files (build output, test reports, binaries)	500 MB (free tier)

Job Outputs (Small Data)

jobs:
  build:
    runs-on: ubuntu-latest
    outputs:
      version: ${{ steps.ver.outputs.value }}   # Expose to other jobs
    steps:
      - id: ver
        run: echo "value=1.2.3" >> $GITHUB_OUTPUT

  deploy:
    needs: build
    runs-on: ubuntu-latest
    steps:
      - run: echo "Deploying ${{ needs.build.outputs.version }}"

Artifacts (Files)

jobs:
  build:
    steps:
      - run: npm run build
      - uses: actions/upload-artifact@v4
        with:
          name: dist
          path: dist/

  deploy:
    needs: build
    steps:
      - uses: actions/download-artifact@v4
        with:
          name: dist
      - run: ls dist/  # Your build files are here!

🏋️ Exercise: Build a Multi-Job Pipeline

Create a pipeline with parallel validation and a sequential build:

# .github/workflows/ci.yml
name: CI

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  lint:
    name: 🔍 Lint
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with: { node-version: '20' }
      - run: npm ci
      - run: echo "✅ Lint passed (add real linter later)"

  test:
    name: 🧪 Test
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with: { node-version: '20' }
      - run: npm ci
      - run: npm test

  build:
    name: 📦 Build
    needs: [lint, test]       # Only if both pass!
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with: { node-version: '20' }
      - run: npm ci
      - name: Build
        run: |
          mkdir dist
          cp src/* dist/
          echo '{"built":"'$(date -u)'"}' > dist/build-info.json
      - uses: actions/upload-artifact@v4
        with:
          name: build-output
          path: dist/

After pushing:

Go to Actions → click the run
Notice lint and test start simultaneously (parallel)
Notice build waits until both complete
Click "build-output" artifact at the bottom to download your built files

Failure Behaviour

🧠 Recall Check

Do jobs share a filesystem? What does that mean for dependencies?
How do you make job B wait for job A to finish?
If lint takes 20s and test takes 45s, and build needs both — what's the total time?
Name two ways to pass data between jobs.
What happens to a dependent job if one of its needs: fails?

Reveal answers

No. Each job gets a fresh VM. Every job must checkout code and install dependencies independently.
needs: A on job B. Or needs: [A, C] to wait for multiple.
45 + 30 = 75 seconds. lint (20s) runs parallel with test (45s). Build starts at 45s (when test finishes). Build takes 30s → 75s total.
Job outputs (small values via $GITHUB_OUTPUT) and artifacts (files via upload/download-artifact).
It's skipped (not run at all). Override with if: always().

You now understand pipeline topology. You can design a DAG that maximises parallelism (speed) while enforcing correctness (build only runs if tests pass). Next: building a real-world CI pipeline with lint, test, and build using actual tools.