Woodpecker

Overview

Woodpecker CI is a lightweight, cloud-native continuous integration and deployment system built on Docker. This document covers the deployment and configuration of Woodpecker CI using Docker Compose with Traefik as a reverse proxy.

Service Details

Component	Value
URL	https://ci.trymondo.com
Server Image	woodpeckerci/woodpecker-server:v3.5.2
Agent Image	woodpeckerci/woodpecker-agent:v3.5.2
Database	PostgreSQL 17
Services	woodpecker-server, woodpecker-agent, woodpecker-db
Networks	woodpecker-net, traefik-net
Authentication	GitHub OAuth

Architecture

The deployment consists of three main components:

Woodpecker Server: The core service that manages the UI, API, and orchestrates CI jobs
Woodpecker Agent: Executes CI jobs in containers
PostgreSQL Database: Stores all Woodpecker CI data

                 ┌─────────────┐
                 │   Traefik   │
                 └─────────────┘
                       ↑
                       │ HTTP/HTTPS
                       ↓
              ┌─────────────────────┐
              │  Woodpecker Server  │
              └─────────────────────┘
                ↑               ↑
    gRPC (9000) │               │ Database
                ↓               ↓
      ┌───────────────┐    ┌────────────┐
      │ Woodpecker    │    │ PostgreSQL │
      │ Agent         │    │            │
      └───────────────┘    └────────────┘
             ↑
             │
      ┌──────────────┐
      │ Docker API   │
      └──────────────┘

Prerequisites

Docker Engine (24.0+)
Docker Compose v2
Traefik reverse proxy configured and running
External networks: woodpecker-net and traefik-net
GitHub OAuth application
DNS configured for ci.trymondo.com

Configuration Preparation

GitHub OAuth Setup

Create a GitHub OAuth application at https://github.com/settings/applications/new
Set the callback URL to https://ci.trymondo.com/login
Note the Client ID and Client Secret for your .env file

Environment Variables

Create a .env file with the following variables:

# Woodpecker Configuration
WOODPECKER_HOST=https://ci.trymondo.com
WOODPECKER_OPEN=true
WOODPECKER_GITHUB_CLIENT=your_github_client_id
WOODPECKER_GITHUB_SECRET=your_github_client_secret
WOODPECKER_AGENT_SECRET=generate_a_secure_random_string
WOODPECKER_ADMIN=github_username

# Database Configuration
POSTGRES_PASSWORD=secure_postgres_password

Generate a secure, random string for WOODPECKER_AGENT_SECRET. This is used to authenticate agents with the server.

You can generate a secure random string with:

openssl rand -hex 32

Deployment Configuration

Docker Compose File

services:
  woodpecker-server:
    image: woodpeckerci/woodpecker-server:v3.5.2
    container_name: woodpecker-server
    restart: unless-stopped
    expose:
      - '8000'
      - '9000'
    volumes:
      - woodpecker-server-data:/var/lib/woodpecker
    environment:
      - WOODPECKER_OPEN=${WOODPECKER_OPEN:-true}
      - WOODPECKER_HOST=${WOODPECKER_HOST?Variable WOODPECKER_HOST is not set}
      - WOODPECKER_GITHUB=true
      - WOODPECKER_GITHUB_CLIENT=${WOODPECKER_GITHUB_CLIENT?Variable WOODPECKER_GITHUB_CLIENT is not set}
      - WOODPECKER_GITHUB_SECRET=${WOODPECKER_GITHUB_SECRET?Variable WOODPECKER_GITHUB_SECRET is not set}
      - WOODPECKER_AGENT_SECRET=${WOODPECKER_AGENT_SECRET?Variable WOODPECKER_AGENT_SECRET is not set}
      - WOODPECKER_DATABASE_DRIVER=postgres
      - WOODPECKER_DATABASE_DATASOURCE=postgres://woodpecker:${POSTGRES_PASSWORD?Variable POSTGRES_PASSWORD is not set}@woodpecker-db:5432/woodpecker?sslmode=disable
      - WOODPECKER_GITHUB_URL=https://github.com
      - WOODPECKER_ADMIN=${WOODPECKER_ADMIN?Variable WOODPECKER_ADMIN is not set}
    networks:
      - woodpecker-net
      - traefik-net
    labels:
      - traefik.enable=true
      - traefik.docker.network=traefik-net
      - traefik.http.services.woodpecker.loadbalancer.server.port=8000
      - traefik.http.routers.woodpecker-secure.rule=Host(`ci.trymondo.com`)
      - traefik.http.routers.woodpecker-secure.entrypoints=websecure
      - traefik.http.routers.woodpecker-secure.service=woodpecker
      - traefik.http.routers.woodpecker-secure.tls=true
      - traefik.http.routers.woodpecker-secure.tls.certresolver=production
      - traefik.http.middlewares.woodpecker-redirect.redirectscheme.scheme=https
      - traefik.http.middlewares.woodpecker-redirect.redirectscheme.permanent=true
      - traefik.http.routers.woodpecker-http.rule=Host(`ci.trymondo.com`)
      - traefik.http.routers.woodpecker-http.entrypoints=web
      - traefik.http.routers.woodpecker-http.service=woodpecker
      - traefik.http.routers.woodpecker-http.middlewares=woodpecker-redirect@docker

  woodpecker-agent:
    image: woodpeckerci/woodpecker-agent:v3.5.2
    container_name: woodpecker-agent
    restart: unless-stopped
    depends_on:
      - woodpecker-server
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:rw
    environment:
      - WOODPECKER_SERVER=woodpecker-server:9000
      - WOODPECKER_GRPC_SECURE=false
      - WOODPECKER_AGENT_SECRET=${WOODPECKER_AGENT_SECRET?Variable WOODPECKER_AGENT_SECRET is not set}
    networks:
      - woodpecker-net

  woodpecker-db:
    image: postgres:17
    container_name: woodpecker-db
    restart: unless-stopped
    environment:
      - POSTGRES_USER=woodpecker
      - POSTGRES_PASSWORD=${POSTGRES_PASSWORD}
      - POSTGRES_DB=woodpecker
    volumes:
      - woodpecker-db-data:/var/lib/postgresql/data
    networks:
      - woodpecker-net

volumes:
  woodpecker-server-data:
  woodpecker-db-data:

networks:
  woodpecker-net:
    external: true
  traefik-net:
    external: true

Network Configuration

Before deployment, ensure the required networks exist:

docker network create woodpecker-net
docker network create traefik-net

Deployment Instructions

Create the deployment directory:
```
mkdir -p /opt/apps/woodpecker
```

Create the docker-compose.yml and .env files:

nano /opt/apps/woodpecker/docker-compose.yml
nano /opt/apps/woodpecker/.env

Deploy the service:

cd /opt/apps/woodpecker
docker compose up -d

Verify all services are running:
```
docker compose ps
```

Traefik Integration Details

The Woodpecker server is configured with the following Traefik settings:

Host Rule: ci.trymondo.com
Entrypoints: websecure (HTTPS on port 443), web (HTTP on port 80 with redirect to HTTPS)
TLS: Enabled with production certificate resolver
Backend Port: 8000 (Woodpecker Web UI)
Middleware: HTTP to HTTPS redirect for enhanced security

Pipeline Configuration

Basic .woodpecker.yml Example

Create a .woodpecker.yml file in the root of your GitHub repository:

pipeline:
  build:
    image: node:20
    commands:
      - npm install
      - npm run build
      - npm test

  docker:
    image: plugins/docker
    settings:
      registry: registry.example.com
      repo: registry.example.com/myapp
      tags: latest
      username:
        from_secret: docker_username
      password:
        from_secret: docker_password
    when:
      branch: main

Secrets Management

To add secrets to your Woodpecker CI projects:

Navigate to your repository in the Woodpecker UI
Go to Settings > Secrets
Add required secrets (e.g., docker_username, docker_password)

Scaling and Performance

Additional Agents

To scale out CI capacity, you can add more agents:

woodpecker-agent-2:
  image: woodpeckerci/woodpecker-agent:v3.5.2
  container_name: woodpecker-agent-2
  restart: unless-stopped
  volumes:
    - /var/run/docker.sock:/var/run/docker.sock:rw
  environment:
    - WOODPECKER_SERVER=woodpecker-server:9000
    - WOODPECKER_GRPC_SECURE=false
    - WOODPECKER_AGENT_SECRET=${WOODPECKER_AGENT_SECRET}
  networks:
    - woodpecker-net

Resource Constraints

For improved stability, consider adding resource constraints:

deploy:
  resources:
    limits:
      cpus: '2.0'
      memory: 2G

Maintenance

Backup Strategy

Back up Woodpecker CI regularly:

Database backup:

docker compose exec woodpecker-db pg_dump -U woodpecker woodpecker > woodpecker_backup_$(date +%Y%m%d).sql

Volume backup using Restic/Backrest:
- woodpecker-server-data (contains server state)
- woodpecker-db-data (contains database files)

Updates

To update Woodpecker CI:

Update the image tags in docker-compose.yml

Apply the update:

docker compose pull
docker compose up -d

Check logs for any issues:
```
docker compose logs
```

Troubleshooting

Server Startup Issues

If the server fails to start:

Check environment variables:

docker compose logs woodpecker-server | grep "required key"

Verify database connection:

docker compose logs woodpecker-server | grep "database"

Agent Connection Problems

If agents can’t connect to the server:

Check agent logs:
```
docker compose logs woodpecker-agent
```
Verify agent secret is consistent between server and agent

Check network connectivity between containers:

docker compose exec woodpecker-agent ping woodpecker-server

GitHub Integration Issues

If GitHub integration is not working:

Verify OAuth callback URL is correctly set to https://ci.trymondo.com/login
Check GitHub client ID and secret in environment variables

Examine server logs for OAuth-related errors:

docker compose logs woodpecker-server | grep -i "github"

Security Considerations

Agent Secret: Use a strong, randomly generated secret for agent authentication
Admin Access: Limit admin access to specific GitHub usernames
Docker Socket: The agent has access to the Docker socket, which grants significant privileges
HTTPS: Enforce HTTPS-only access through Traefik
Network Isolation: Use separate networks for communication between services

Additional Resources

Pipeline Examples

Node.js Application

pipeline:
  test:
    image: node:20
    commands:
      - npm install
      - npm test

Docker Build and Push

pipeline:
  build:
    image: plugins/docker
    settings:
      registry: docker.io
      repo: myuser/myapp
      tags: latest
      username:
        from_secret: docker_username
      password:
        from_secret: docker_password

Multi-stage Pipeline with Approval

pipeline:
  test:
    image: golang:1.21
    commands:
      - go test ./...

  build:
    image: golang:1.21
    commands:
      - go build -o myapp

  deploy:
    image: plugins/ssh
    settings:
      host: production.example.com
      user: deploy
      key:
        from_secret: ssh_key
      script:
        - ./deploy.sh
    when:
      status: success
      branch: main
    approval:
      approved_by:
        - admin_user

Introduction

Applications

Infrastructure

Overview

Service Details

Architecture

Prerequisites

Configuration Preparation

GitHub OAuth Setup

Environment Variables

Deployment Configuration

Docker Compose File

Network Configuration

Deployment Instructions

Traefik Integration Details

Pipeline Configuration

Basic .woodpecker.yml Example

Secrets Management

Scaling and Performance

Additional Agents

Resource Constraints

Maintenance

Backup Strategy

Updates

Troubleshooting

Server Startup Issues

Agent Connection Problems

GitHub Integration Issues

Security Considerations

Additional Resources

Pipeline Examples

Node.js Application

Docker Build and Push

Multi-stage Pipeline with Approval

Introduction

Applications

Infrastructure

​Overview

​Service Details

​Architecture

​Prerequisites

​Configuration Preparation

​GitHub OAuth Setup

​Environment Variables

​Deployment Configuration

​Docker Compose File

​Network Configuration

​Deployment Instructions

​Traefik Integration Details

​Pipeline Configuration

​Basic .woodpecker.yml Example

​Secrets Management

​Scaling and Performance

​Additional Agents

​Resource Constraints

​Maintenance

​Backup Strategy

​Updates

​Troubleshooting

​Server Startup Issues

​Agent Connection Problems

​GitHub Integration Issues

​Security Considerations

​Additional Resources

​Pipeline Examples

​Node.js Application

​Docker Build and Push

​Multi-stage Pipeline with Approval

Overview

Service Details

Architecture

Prerequisites

Configuration Preparation

GitHub OAuth Setup

Environment Variables

Deployment Configuration

Docker Compose File

Network Configuration

Deployment Instructions

Traefik Integration Details

Pipeline Configuration

Basic .woodpecker.yml Example

Secrets Management

Scaling and Performance

Additional Agents

Resource Constraints

Maintenance

Backup Strategy

Updates

Troubleshooting

Server Startup Issues

Agent Connection Problems

GitHub Integration Issues

Security Considerations

Additional Resources

Pipeline Examples

Node.js Application

Docker Build and Push

Multi-stage Pipeline with Approval