Dagger CI/CD: The Future of Pipeline Development (Hands-On Tutorial)

Introduction

CI/CD pipelines are notoriously hard to debug. You push to GitHub, wait 3 minutes, get a cryptic error, fix one line, push again, wait 3 minutes. The "10-second feedback loop" is a myth.

Dagger solves this by making pipelines containerized, portable, and locally executable. Write your pipeline once—run it on your laptop, in GitHub Actions, or anywhere that runs Docker.

No more vendor-specific YAML. No more "it works in CI but not locally." Just code.

What is Dagger?

Dagger runs your CI/CD pipeline inside containers. Every step—build, test, lint, deploy—executes in its own cached container. The pipeline is defined in code (Go, Python, TypeScript, or the CUE language), not YAML.

# Install Dagger
curl -L https://dl.dagger.io/dagger/install.sh | sh

# Run any pipeline locally
dagger run go run ci/main.go

The same command works on your machine and in CI. The pipeline code is the same too.

Pipeline as Code

Here's a complete Dagger pipeline in Go:

package main

import (
    "context"
    "dagger.io/dagger"
)

func main() {
    ctx := context.Background()
    client, _ := dagger.Connect(ctx)
    defer client.Close()

    // Build environment
    src := client.Host().Directory("./src")
    golang := client.Container().
        From("golang:1.22-alpine").
        WithDirectory("/app", src).
        WithWorkdir("/app")

    // Run tests
    test := golang.WithExec([]string{"go", "test", "./..."})

    // Build binary
    build := golang.
        WithExec([]string{"go", "build", "-o", "/app/server"}).
        File("/app/server")

    // Build Docker image
    image := client.Container().
        From("alpine:3.19").
        WithFile("/usr/bin/server", build).
        WithEntrypoint([]string{"/usr/bin/server"})

    // Publish
    addr, _ := image.Publish(ctx, "registry.example.com/myapp:latest")
    println("Published:", addr)
}

Run it:

dagger run go run main.go

Why Dagger Beats Traditional CI

Traditional CI systems have fundamental architectural problems that Dagger eliminates:

1. Local Debugging

Pipeline fails? Run it locally with the exact same container:

1. Local Debugging

Pipeline fails? Run it locally with the same exact container:

dagger run go run main.go  # Same as CI

2. Automatic Caching

Dagger caches every step automatically using content-addressed storage. Changed only one test file? Only that test step re-executes. The Go module cache, npm node_modules, compiled binaries — everything that hasn't changed is reused from cache:

# First run: builds everything
dagger run go run main.go

# Second run: near-instant (only changed steps execute)
dagger run go run main.go

This is not Docker layer caching — it is smarter. Dagger's cache is based on content hashes, so even if the order of steps changes, cached outputs from identical inputs are reused.

3. Vendor Agnostic

Switch CI providers without rewriting your pipeline code. The Dagger pipeline is a regular program that runs anywhere:

# .github/workflows/ci.yml — GitHub Actions
jobs:
  dagger:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: dagger/dagger-for-github@v6
        with:
          args: run go run ci/main.go

# .gitlab-ci.yml — GitLab CI
dagger:
  image: golang:1.22
  script:
    - dagger run go run ci/main.go

Same pipeline code. Two different CI systems. Zero YAML translation.

4. Multi-Language Support

Dagger SDKs are available for Go, Python, and TypeScript, with a GraphQL API underneath. Your team writes pipelines in the language they already know, not yet another YAML dialect:

# ci/main.py
import dagger

async def main():
    async with dagger.Connection() as client:
        src = client.host().directory("./src")
        node = (client.container()
            .from_("node:20-alpine")
            .with_directory("/app", src)
            .with_workdir("/app")
            .with_exec(["npm", "ci"])
            .with_exec(["npm", "test"]))
        await node.sync()

import asyncio
asyncio.run(main())

Real-World Pattern: Multi-Stage Pipeline

Here is a realistic multi-stage pipeline for a Go microservice that runs linting, tests, and a production build:

func main() {
    ctx := context.Background()
    client, _ := dagger.Connect(ctx)
    defer client.Close()

    src := client.Host().Directory(".")

    // Stage 1: Lint
    lint := client.Container().
        From("golangci/golangci-lint:v1.56-alpine").
        WithDirectory("/app", src).
        WithWorkdir("/app").
        WithExec([]string{"golangci-lint", "run"})

    // Stage 2: Test with race detection
    test := client.Container().
        From("golang:1.22-alpine").
        WithDirectory("/app", src).
        WithWorkdir("/app").
        WithExec([]string{"go", "test", "-race", "-cover", "./..."})

    // Stage 3: Optimized production build
    build := client.Container().
        From("golang:1.22-alpine").
        WithDirectory("/app", src).
        WithWorkdir("/app").
        WithEnvVariable("CGO_ENABLED", "0").
        WithExec([]string{"go", "build", "-ldflags", "-s -w", "-o", "server"})

    // Export binary for Docker packaging
    build.File("server").Export(ctx, "./bin/server")
}

Each stage is an independent, cached DAG node. If only source code changes, the lint and test stages re-run but cached Go module downloads are reused. Stages can also execute in parallel when there are no dependencies between them.

Dagger Modules (The Daggerverse)

Dagger v0.12+ introduced modules — reusable, versioned components that encapsulate common pipeline logic. The Daggerverse is the module registry, similar to npm for CI/CD pipelines.

Installing a Module

# Search available modules
dagger mod list

# Install a module
dagger mod install github.com/dagger/daggerverse/aws@v0.3
dagger mod install github.com/dagger/daggerverse/github-actions@v1

Writing Your Own Module

Create a reusable deployment module that any team can import:

// deploy/deploy.go
package main

import "dagger.io/dagger"

type Deploy struct {
    Registry string
    Tag      string
}

func (d *Deploy) PushImage(ctx context.Context, src *dagger.Directory) (string, error) {
    image := dag.Container().
        From("alpine:latest").
        WithDirectory("/app", src).
        WithEntrypoint([]string{"/app/server"})
    addr, err := image.Publish(ctx, d.Registry+":"+d.Tag)
    return addr, err
}

Publish for your organization:

dagger mod publish ghcr.io/myorg/deploy-module:v1

Now any team deploys with a single function call — no copy-paste.

Testing Dagger Pipelines

Because Dagger pipelines are regular Go or Python code, you can unit test them directly:

// ci/main_test.go
func TestBuildPipeline(t *testing.T) {
    ctx := context.Background()
    client, _ := dagger.Connect(ctx)
    defer client.Close()
    src := client.Host().Directory("./testdata")
    build := client.Container().
        From("golang:1.22-alpine").
        WithDirectory("/app", src).
        WithWorkdir("/app").
        WithExec([]string{"go", "build", "-o", "/dev/null", "."})
    _, err := build.ExitCode(ctx)
    if err != nil {
        t.Fatalf("build failed: %v", err)
    }
}

Run like any Go test:

go test ./ci/... -v

No more pushing to GitHub to find pipeline bugs — test them in seconds locally.

Monorepo Pipeline Pattern

For monorepos with multiple services, use Dagger's directory filtering to parallelize:

func main() {
    ctx := context.Background()
    client, _ := dagger.Connect(ctx)
    defer client.Close()
    root := client.Host().Directory(".")
    services := []string{"api", "worker", "web"}
    for _, svc := range services {
        svc := svc
        go func() {
            src := root.Directory(svc)
            build := client.Container().
                From("golang:1.22-alpine").
                WithDirectory("/app", src).
                WithWorkdir("/app").
                WithExec([]string{"go", "build", "-o", "/dev/null", "."})
            _, err := build.Sync(ctx)
            if err != nil {
                log.Fatalf("%s build failed: %v", svc, err)
            }
        }()
    }
}

Dagger runs all service builds in parallel, cached independently. A monorepo CI run that took 15 minutes in GitHub Actions drops to under 3 minutes.

Comparison: Dagger vs Traditional CI

| Feature | GitHub Actions | GitLab CI | Dagger | |---------|---------------|-----------|--------| | Local execution | No | Partial | Full | | Cache management | Manual | Manual | Automatic | | Pipeline language | YAML | YAML | Go/Python/TS | | Vendor lock-in | Full | Full | None | | Debug cycle | Push-wait | Push-wait | Local dagger run | | Module ecosystem | Marketplace | None | Daggerverse |

The table highlights the key differentiator: Dagger treats pipelines as software rather than configuration. When your pipeline is code, you get all the software engineering benefits — versioning, code review, unit testing, modular design, and reuse through modules.

This is why many organizations are adopting Dagger as a layer on top of existing CI systems. You keep your current GitHub Actions or GitLab runners but replace the YAML with testable, portable code.

Dagger vs Makefile vs Scripts

Many teams start with a Makefile or shell scripts for local builds. Here's how Dagger compares:

| Approach | Reproducibility | Caching | Language | CI Portability | |----------|----------------|---------|----------|----------------| | Makefile | Depends on local tools | None | Shell | Must be adapted | | Shell scripts | Environment-dependent | None | Bash | Must be adapted | | Dagger | Exact container match | Automatic | Go/Python/TS | Same code everywhere |

Makefiles work fine for simple tasks, but as your pipeline grows, the "works on my machine" problem reappears. Dagger's containerized execution guarantees that every run uses the exact same tools and versions, regardless of where it executes.

Getting Started: Your First Dagger Pipeline

The fastest way to adopt Dagger is to convert your most painful pipeline:

1. Install Dagger with the official install script
2. Copy your build steps into a Go main.go using dagger.Container().From().WithExec()
3. Replace file operations with dagger.Host().Directory() mounts
4. Test locally: dagger run go run ci/main.go
5. Wrap in a thin CI YAML (3 lines for GitHub Actions)

Most teams convert their first pipeline in under an hour. The second pipeline takes 15 minutes because you reuse the same patterns.

Production Best Practices

• Pin base image versions: Use golang:1.22-alpine@sha256:abc... instead of golang:1.22-alpine for reproducible builds
• Use .daggerignore: Exclude node_modules, .git, and build artifacts from directory mounts to speed up caching
• Set resource limits: Dagger containers inherit host resources by default; use WithContainerResources() for CI runners
• Export test reports: Use .WithDirectory("/app/coverage") to persist test coverage data outside the container
• Monitor Dagger Cloud: Dagger Cloud provides pipeline insights, cache hit rates, and execution history

Conclusion

Dagger is CI/CD's "Docker moment"—taking something that was ad-hoc and environment-specific and making it portable, reproducible, and developer-friendly.

Start by converting one pipeline (your most painful one) to Dagger. Run it locally, watch it succeed on the first CI push, and experience the joy of never debugging CI in YAML again.