DevOps & Deployment

CI/CD Pipelines: Complete Setup

Build robust CI/CD pipelines using GitHub Actions, Jenkins, and GitLab CI for automated deployment. Master continuous integration and deployment strategies.

TechDevDex Team

12/1/2024

22 min

#CI/CD#GitHub Actions#Jenkins#GitLab CI#DevOps#Automation#Deployment

CI/CD Pipelines: Complete Setup

Continuous Integration and Continuous Deployment (CI/CD) pipelines automate the process of building, testing, and deploying applications. This comprehensive guide covers setting up robust CI/CD pipelines using popular tools and platforms.

CI/CD Fundamentals

Core Concepts

Continuous Integration (CI)

Definition: Automated building and testing of code changes
Benefits: Early bug detection, faster feedback, reduced integration issues
Process: Code commit → Build → Test → Package
Tools: Jenkins, GitHub Actions, GitLab CI, CircleCI

Continuous Deployment (CD)

Definition: Automated deployment of tested code to production
Benefits: Faster releases, reduced deployment risk, consistent environments
Process: Package → Deploy → Monitor → Rollback (if needed)
Tools: Kubernetes, Docker, AWS, Azure, GCP

Pipeline Stages

Source Stage

Code Repository: Git-based version control
Branch Strategy: Feature branches, main branch protection
Trigger Events: Push, pull request, scheduled builds
Code Quality: Static analysis, security scanning

Build Stage

Environment Setup: Dependencies, runtime environments
Compilation: Source code compilation and packaging
Artifact Creation: Build outputs, container images
Quality Gates: Code coverage, test results

Test Stage

Unit Tests: Individual component testing
Integration Tests: Component interaction testing
End-to-End Tests: Full application testing
Performance Tests: Load and stress testing

Deploy Stage

Environment Promotion: Dev → Staging → Production
Configuration Management: Environment-specific settings
Database Migrations: Schema and data updates
Health Checks: Application and service validation

GitHub Actions

Basic Workflow

Workflow Configuration

          
yaml

name: CI/CD Pipeline

on:
  push:
    branches: [ main, develop ]
  pull_request:
    branches: [ main ]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v3
    
    - name: Setup Node.js
      uses: actions/setup-node@v3
      with:
        node-version: '18'
        cache: 'npm'
    
    - name: Install dependencies
      run: npm ci
    
    - name: Run tests
      run: npm test
    
    - name: Run linting
      run: npm run lint
    
    - name: Build application
      run: npm run build

Multi-Environment Deployment

          
yaml

name: Deploy to Environments

on:
  push:
    branches: [ main, develop ]

jobs:
  deploy-staging:
    if: github.ref == 'refs/heads/develop'
    runs-on: ubuntu-latest
    environment: staging
    steps:
    - uses: actions/checkout@v3
    
    - name: Deploy to Staging
      run: |
        echo "Deploying to staging environment"
        # Add your deployment commands here
    
  deploy-production:
    if: github.ref == 'refs/heads/main'
    runs-on: ubuntu-latest
    environment: production
    needs: deploy-staging
    steps:
    - uses: actions/checkout@v3
    
    - name: Deploy to Production
      run: |
        echo "Deploying to production environment"
        # Add your deployment commands here

Advanced GitHub Actions

Matrix Builds

          
yaml

name: Matrix Build

on: [push, pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        node-version: [16, 18, 20]
        os: [ubuntu-latest, windows-latest, macos-latest]
    
    steps:
    - uses: actions/checkout@v3
    
    - name: Setup Node.js ${{ matrix.node-version }}
      uses: actions/setup-node@v3
      with:
        node-version: ${{ matrix.node-version }}
        cache: 'npm'
    
    - name: Install dependencies
      run: npm ci
    
    - name: Run tests
      run: npm test

Docker Build and Push

          
yaml

name: Build and Push Docker Image

on:
  push:
    branches: [ main ]

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v3
    
    - name: Set up Docker Buildx
      uses: docker/setup-buildx-action@v2
    
    - name: Login to Container Registry
      uses: docker/login-action@v2
      with:
        registry: ghcr.io
        username: ${{ github.actor }}
        password: ${{ secrets.GITHUB_TOKEN }}
    
    - name: Build and push
      uses: docker/build-push-action@v4
      with:
        context: .
        push: true
        tags: |
          ghcr.io/${{ github.repository }}:latest
          ghcr.io/${{ github.repository }}:${{ github.sha }}

Jenkins Pipeline

Jenkinsfile Setup

Declarative Pipeline

          
groovy

pipeline {
    agent any
    
    environment {
        NODE_VERSION = '18'
        DOCKER_REGISTRY = 'your-registry.com'
        IMAGE_NAME = 'my-app'
    }
    
    stages {
        stage('Checkout') {
            steps {
                checkout scm
            }
        }
        
        stage('Build') {
            steps {
                sh 'npm ci'
                sh 'npm run build'
            }
        }
        
        stage('Test') {
            steps {
                sh 'npm test'
                sh 'npm run lint'
            }
            post {
                always {
                    publishTestResults testResultsPattern: 'test-results.xml'
                }
            }
        }
        
        stage('Docker Build') {
            steps {
                script {
                    def image = docker.build("${DOCKER_REGISTRY}/${IMAGE_NAME}:${BUILD_NUMBER}")
                    docker.withRegistry("https://${DOCKER_REGISTRY}", 'docker-registry-credentials') {
                        image.push()
                        image.push('latest')
                    }
                }
            }
        }
        
        stage('Deploy') {
            when {
                branch 'main'
            }
            steps {
                sh 'kubectl apply -f k8s/'
                sh 'kubectl rollout status deployment/my-app'
            }
        }
    }
    
    post {
        always {
            cleanWs()
        }
        failure {
            mail to: 'team@company.com',
                 subject: "Build Failed: ${env.JOB_NAME} - ${env.BUILD_NUMBER}",
                 body: "Build failed. Check console output for details."
        }
    }
}

Scripted Pipeline

          
groovy

node {
    stage('Checkout') {
        checkout scm
    }
    
    stage('Build') {
        def buildTool = tool name: 'NodeJS-18', type: 'nodejs'
        env.PATH = "${buildTool}/bin:${env.PATH}"
        
        sh 'npm ci'
        sh 'npm run build'
    }
    
    stage('Test') {
        sh 'npm test'
        sh 'npm run lint'
        
        publishTestResults testResultsPattern: 'test-results.xml'
        publishHTML([
            allowMissing: false,
            alwaysLinkToLastBuild: true,
            keepAll: true,
            reportDir: 'coverage',
            reportFiles: 'index.html',
            reportName: 'Coverage Report'
        ])
    }
    
    stage('Docker Build') {
        def image = docker.build("my-app:${BUILD_NUMBER}")
        docker.withRegistry('https://your-registry.com', 'docker-registry-credentials') {
            image.push()
            image.push('latest')
        }
    }
    
    stage('Deploy') {
        if (env.BRANCH_NAME == 'main') {
            sh 'kubectl apply -f k8s/'
            sh 'kubectl rollout status deployment/my-app'
        }
    }
}

Jenkins Configuration

Global Tool Configuration

          
groovy

// Global Tools Configuration
tools {
    nodejs 'NodeJS-18'
    dockerTool 'Docker'
    git 'Git'
}

// Global Pipeline Libraries
libraries {
    lib('shared-library@main')
}

Credentials Management

          
groovy

pipeline {
    agent any
    
    stages {
        stage('Deploy') {
            steps {
                script {
                    withCredentials([
                        string(credentialsId: 'kubeconfig', variable: 'KUBECONFIG'),
                        string(credentialsId: 'docker-registry', variable: 'DOCKER_REGISTRY')
                    ]) {
                        sh 'kubectl apply -f k8s/'
                    }
                }
            }
        }
    }
}

GitLab CI/CD

GitLab CI Configuration

Basic Pipeline

          
yaml

# .gitlab-ci.yml
stages:
  - build
  - test
  - deploy

variables:
  NODE_VERSION: "18"
  DOCKER_IMAGE: "$CI_REGISTRY_IMAGE:$CI_COMMIT_SHA"

build:
  stage: build
  image: node:18
  script:
    - npm ci
    - npm run build
  artifacts:
    paths:
      - dist/
    expire_in: 1 hour

test:
  stage: test
  image: node:18
  script:
    - npm ci
    - npm test
    - npm run lint
  coverage: '/Lines\s*:\s*(\d+\.\d+)%/'
  artifacts:
    reports:
      junit: test-results.xml
    paths:
      - coverage/
    expire_in: 1 week

docker-build:
  stage: build
  image: docker:latest
  services:
    - docker:dind
  script:
    - docker build -t $DOCKER_IMAGE .
    - docker push $DOCKER_IMAGE
  only:
    - main
    - develop

deploy-staging:
  stage: deploy
  image: bitnami/kubectl:latest
  script:
    - kubectl apply -f k8s/
    - kubectl rollout status deployment/my-app
  environment:
    name: staging
    url: https://staging.myapp.com
  only:
    - develop

deploy-production:
  stage: deploy
  image: bitnami/kubectl:latest
  script:
    - kubectl apply -f k8s/
    - kubectl rollout status deployment/my-app
  environment:
    name: production
    url: https://myapp.com
  when: manual
  only:
    - main

Advanced GitLab CI

Multi-Project Pipeline

          
yaml

# .gitlab-ci.yml
stages:
  - build
  - test
  - deploy

build-frontend:
  stage: build
  script:
    - cd frontend
    - npm ci
    - npm run build
  artifacts:
    paths:
      - frontend/dist/

build-backend:
  stage: build
  script:
    - cd backend
    - npm ci
    - npm run build
  artifacts:
    paths:
      - backend/dist/

test-frontend:
  stage: test
  script:
    - cd frontend
    - npm ci
    - npm test
  dependencies:
    - build-frontend

test-backend:
  stage: test
  script:
    - cd backend
    - npm ci
    - npm test
  dependencies:
    - build-backend

deploy:
  stage: deploy
  script:
    - echo "Deploying application"
  dependencies:
    - test-frontend
    - test-backend

Docker Integration

Multi-Stage Dockerfile

          
dockerfile

# Build stage
FROM node:18-alpine AS builder

WORKDIR /app
COPY package*.json ./
RUN npm ci --only=production

# Development stage
FROM node:18-alpine AS development
WORKDIR /app
COPY package*.json ./
RUN npm ci
COPY . .
EXPOSE 3000
CMD ["npm", "run", "dev"]

# Production stage
FROM node:18-alpine AS production
WORKDIR /app

# Create non-root user
RUN addgroup -g 1001 -S nodejs
RUN adduser -S nextjs -u 1001

COPY --from=builder /app/node_modules ./node_modules
COPY --chown=nextjs:nodejs . .

USER nextjs
EXPOSE 3000
CMD ["npm", "start"]

Docker Compose for CI

          
yaml

# docker-compose.ci.yml
version: '3.8'

services:
  app:
    build:
      context: .
      target: development
    ports:
      - "3000:3000"
    environment:
      - NODE_ENV=test
      - DATABASE_URL=postgresql://test:test@db:5432/testdb
    depends_on:
      - db
      - redis

  db:
    image: postgres:15
    environment:
      - POSTGRES_DB=testdb
      - POSTGRES_USER=test
      - POSTGRES_PASSWORD=test
    ports:
      - "5432:5432"

  redis:
    image: redis:7
    ports:
      - "6379:6379"

  test:
    build:
      context: .
      target: development
    command: npm test
    environment:
      - NODE_ENV=test
      - DATABASE_URL=postgresql://test:test@db:5432/testdb
    depends_on:
      - db
      - redis

Kubernetes Deployment

Deployment Manifests

          
yaml

# k8s/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
  labels:
    app: my-app
spec:
  replicas: 3
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      containers:
      - name: my-app
        image: my-registry.com/my-app:latest
        ports:
        - containerPort: 3000
        env:
        - name: NODE_ENV
          value: "production"
        - name: DATABASE_URL
          valueFrom:
            secretKeyRef:
              name: my-app-secrets
              key: database-url
        resources:
          requests:
            memory: "256Mi"
            cpu: "250m"
          limits:
            memory: "512Mi"
            cpu: "500m"
        livenessProbe:
          httpGet:
            path: /health
            port: 3000
          initialDelaySeconds: 30
          periodSeconds: 10
        readinessProbe:
          httpGet:
            path: /ready
            port: 3000
          initialDelaySeconds: 5
          periodSeconds: 5

Service and Ingress

          
yaml

# k8s/service.yaml
apiVersion: v1
kind: Service
metadata:
  name: my-app-service
spec:
  selector:
    app: my-app
  ports:
  - protocol: TCP
    port: 80
    targetPort: 3000
  type: ClusterIP

---
# k8s/ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: my-app-ingress
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  rules:
  - host: myapp.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: my-app-service
            port:
              number: 80

Monitoring and Alerting

Health Checks

          
yaml

# health-check.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: health-check-script
data:
  health-check.sh: |
    #!/bin/bash
    if curl -f http://localhost:3000/health; then
      echo "Health check passed"
      exit 0
    else
      echo "Health check failed"
      exit 1
    fi

Prometheus Monitoring

          
yaml

# monitoring.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-config
data:
  prometheus.yml: |
    global:
      scrape_interval: 15s
    
    scrape_configs:
    - job_name: 'my-app'
      static_configs:
      - targets: ['my-app-service:80']
      metrics_path: /metrics
      scrape_interval: 30s

Security Best Practices

Secrets Management

          
yaml

# secrets.yaml
apiVersion: v1
kind: Secret
metadata:
  name: my-app-secrets
type: Opaque
data:
  database-url: <base64-encoded-url>
  api-key: <base64-encoded-key>

Security Scanning

          
yaml

# security-scan.yml
name: Security Scan

on: [push, pull_request]

jobs:
  security-scan:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v3
    
    - name: Run Trivy vulnerability scanner
      uses: aquasecurity/trivy-action@master
      with:
        scan-type: 'fs'
        scan-ref: '.'
        format: 'sarif'
        output: 'trivy-results.sarif'
    
    - name: Upload Trivy scan results
      uses: github/codeql-action/upload-sarif@v2
      with:
        sarif_file: 'trivy-results.sarif'

Conclusion

CI/CD pipelines are essential for modern software development, enabling teams to deliver high-quality software faster and more reliably. By implementing the strategies and tools covered in this guide, you can build robust, automated pipelines that improve your development workflow and reduce deployment risks.

The key to successful CI/CD implementation is starting simple and gradually adding complexity as your team becomes more comfortable with the tools and processes. With proper planning and execution, CI/CD pipelines can significantly improve your software delivery capabilities.

Popular Topics

Stay Updated

CI/CD Pipelines: Complete Setup

CI/CD Fundamentals

Core Concepts

Continuous Integration (CI)

Continuous Deployment (CD)

Pipeline Stages

Source Stage

Build Stage

Test Stage

Deploy Stage

GitHub Actions

Basic Workflow

Workflow Configuration

Multi-Environment Deployment

Advanced GitHub Actions

Matrix Builds

Docker Build and Push

Jenkins Pipeline

Jenkinsfile Setup

Declarative Pipeline

Scripted Pipeline

Jenkins Configuration

Global Tool Configuration

Credentials Management

GitLab CI/CD

GitLab CI Configuration

Basic Pipeline

Advanced GitLab CI

Multi-Project Pipeline

Docker Integration

Multi-Stage Dockerfile

Docker Compose for CI

Kubernetes Deployment

Deployment Manifests

Service and Ingress

Monitoring and Alerting

Health Checks

Prometheus Monitoring

Security Best Practices

Secrets Management

Security Scanning

Conclusion