Jenkins: Build Automation & CI/CD (Part 1)

What is Build Automation?

Think about deploying code manually - you’d need to pull the latest changes, run tests, build the application, create Docker images, and push everything to repositories. Doing this every single time would be tedious and error-prone. That’s where build automation comes in.

Build automation automates the entire software delivery process:

• Retrieving source code from repositories
• Running automated tests
• Compiling code into binaries or building Docker images
• Deploying artifacts to various environments

Instead of developers handling these tasks manually on their machines, a dedicated server takes care of everything. This is the foundation of CI/CD and modern DevOps practices.

Understanding CI/CD

Continuous Integration (CI): New code changes are continuously built, tested, and merged into a shared repository. This ensures that code is always in a deployable state.

Continuous Deployment (CD): Takes automation further by automatically deploying applications through various stages (dev, staging, production) without manual intervention.

The goal? “Release early and often” by automating the entire software release cycle.

Introduction to Jenkins

Jenkins is the most widely used build automation and CI/CD tool in the industry. It’s open-source software that you install on a dedicated server, providing a UI to configure and manage your builds.

What makes Jenkins powerful?

1. Extensibility through Plugins: Jenkins has thousands of plugins for integrating with Docker, build tools (Maven, Gradle, npm), Git repositories, deployment servers, notification systems, and more.

2. Flexibility: Can handle simple tasks or complex multi-stage pipelines.

3. Community Support: Massive community with extensive documentation and plugin ecosystem.

Jenkins Roles

Jenkins Administrator (Operations/DevOps teams):

• Sets up and manages Jenkins infrastructure
• Installs and configures plugins
• Manages Jenkins clusters
• Handles backups and security
• Configures credentials and permissions

Jenkins User (Developers/DevOps teams):

• Creates and manages build jobs
• Configures pipelines for projects
• Monitors build status and logs

Setting Up Jenkins

Installation Options

You have two main approaches:

1. Run as Docker Container (Recommended for getting started):

docker run -p 8080:8080 -p 50000:50000 -d \
  -v jenkins_home:/var/jenkins_home \
  jenkins/jenkins:lts

• Port 8080: Jenkins web UI
• Port 50000: Agent communication
• Volume mount persists Jenkins data

2. Direct OS Installation: Install Jenkins directly on the server’s operating system.

Initial Setup

1. Access Jenkins at http://your-server-ip:8080

2. Retrieve the initial admin password:

   docker exec <container-id> cat /var/jenkins_home/secrets/initialAdminPassword

3. Install suggested plugins

4. Create your admin user

Configuring Build Tools

Jenkins needs access to build tools to compile and package your applications. Different programming languages require different tools:

• Java applications: Maven or Gradle
• JavaScript/Node.js applications: npm
• Python applications: pip

Method 1: Using Plugins (Recommended)

1. Navigate to Manage Jenkins → Tools
2. Find your build tool section (e.g., Maven installations)
3. Add a new installation and select the version
4. Jenkins will automatically download and configure it

Example: Setting up Maven

• Go to Tools → Maven installations
• Click “Add Maven”
• Name it (e.g., “maven-3.9.11”)
• Select “Install automatically” and choose version

Method 2: Installing Directly on the Server

For tools not available as plugins (like Node.js), install them directly in the Jenkins container:

# Enter the container as root
docker exec -it -u 0 <container-id> bash

# Check the OS distribution
cat /etc/issue

# Install Node.js (example for Debian/Ubuntu)
curl -fsSL https://deb.nodesource.com/setup_lts.x | bash -
apt-get install -y nodejs

# Verify installation
node --version
npm --version

Working with Docker in Jenkins

Many modern applications are containerized, so Jenkins needs Docker access. But there’s a challenge: Jenkins runs in a container, so how do we use Docker?

Docker-out-of-Docker (DooD) Approach

Instead of running Docker inside Docker (which is complex and has issues), we mount the host’s Docker socket into the Jenkins container:

docker run -p 8080:8080 -p 50000:50000 -d \
  -v jenkins_home:/var/jenkins_home \
  -v /var/run/docker.sock:/var/run/docker.sock \
  jenkins/jenkins:lts

Why this works: The Docker socket (/var/run/docker.sock) is the interface to communicate with the Docker daemon. By mounting it, Jenkins can send Docker commands to the host’s Docker engine.

Installing Docker CLI in Jenkins

The container has access to the Docker socket, but it still needs the Docker CLI:

# Enter as root
docker exec -it -u 0 <container-id> bash

# Install Docker CLI
curl https://get.docker.com/ > dockerinstall
chmod 777 dockerinstall
./dockerinstall

# Fix permissions so Jenkins user can access Docker
chmod 666 /var/run/docker.sock

Note: The socket permissions may reset after restarting containers, so you might need to reapply chmod 666 /var/run/docker.sock.

Managing Credentials

Jenkins needs credentials to access various services:

• Git repositories (especially private ones)
• Docker registries (Docker Hub, Nexus, ECR)
• Deployment servers (SSH keys)
• Cloud providers (AWS, Azure, GCP)

Credential Types

• Username with password: Standard login credentials
• Secret text: API tokens, access tokens
• SSH Username with private key: For SSH-based operations
• Secret file: Kubeconfig files, certificate files

Credential Scopes

Global: Available to all jobs across Jenkins - use this for shared resources.

System: Only available to the Jenkins server itself, not accessible in jobs - for system-level configurations.

Multibranch Pipeline Scoped: Only available within a specific multibranch pipeline project.

Creating Credentials

1. Navigate to Manage Jenkins → Credentials
2. Select appropriate domain (usually “Global”)
3. Click “Add Credentials”
4. Choose credential type and fill in details
5. Give it a meaningful ID (you’ll reference this in pipelines)

Example: Docker Hub Credentials

Kind: Username with password
Username: your-dockerhub-username
Password: your-dockerhub-password
ID: docker-hub
Description: Docker Hub Registry Credentials

Jenkins Job Types

1. Freestyle Jobs

Freestyle jobs are simple, UI-based configurations suitable for straightforward tasks.

Use cases:

• Single, simple tasks
• Quick prototyping
• Legacy projects

Creating a Freestyle Job:

1. Click “New Item” → Enter name → Select “Freestyle project”
2. Source Code Management: Add your Git repository URL
   • If private, add credentials
3. Build Environment: Configure as needed
4. Build Steps: Add steps like:
   • Execute shell commands
   • Invoke Maven/Gradle
   • Run Docker commands

Example Build Steps:

# Shell command
node --version
npm install
npm test

# Or use Maven (if configured as plugin)
# Goals: clean package

Chaining Jobs: You can chain freestyle jobs using “Build other projects” in post-build actions, but this gets messy quickly.

2. Pipeline Jobs

Pipeline jobs are designed for complex CI/CD workflows. They use code (Groovy scripts) to define the entire pipeline.

Advantages:

• Version controlled: Pipeline code lives in your repository
• Parallel execution: Run stages simultaneously
• Conditional logic: Execute stages based on conditions
• Better visualization: See the entire pipeline flow
• Reusability: Share code across projects

3. Multibranch Pipeline

Automatically discovers and manages pipelines for multiple branches in a Git repository.

How it works:

• Jenkins scans your repository
• Creates a pipeline for each branch containing a Jenkinsfile
• Automatically builds when branches are updated
• Cleans up pipelines when branches are deleted

Use case: You have a main branch and multiple feature branches. Each needs testing, but only main gets deployed to production.

Jenkins Directory Structure

Understanding where Jenkins stores data helps with troubleshooting:

• /var/jenkins_home/jobs/: Contains all job configurations and build history
• /var/jenkins_home/workspace/: Working directory where Git repositories are cloned during builds
• /var/jenkins_home/plugins/: Installed plugins
• /var/jenkins_home/secrets/: Security-related files

Pipeline as Code: Jenkinsfile

A Jenkinsfile is a text file containing your pipeline definition, stored in your project’s Git repository. This is the industry best practice: “Everything as Code.”

Jenkinsfile Syntax Formats

Declarative Pipeline (Recommended for beginners):

• Structured, predefined format
• Easier to learn and read
• Sufficient for most use cases

Scripted Pipeline:

• Full Groovy scripting capabilities
• Maximum flexibility
• Steeper learning curve

Basic Declarative Pipeline Structure

#!/usr/bin/env groovy

pipeline {
    agent any  // Where to execute (any available agent)
    
    stages {
        stage('Build') {
            steps {
                echo 'Building...'
                // Your build commands
            }
        }
        
        stage('Test') {
            steps {
                echo 'Testing...'
                // Your test commands
            }
        }
        
        stage('Deploy') {
            steps {
                echo 'Deploying...'
                // Your deployment commands
            }
        }
    }
}

Essential Jenkinsfile Components

1. Agent

Specifies where the pipeline executes:

agent any  // Use any available agent
agent none  // Define agents per stage
agent { label 'linux' }  // Specific agent

2. Tools

Access build tools configured in Jenkins:

tools {
    maven 'maven-3.9.11'  // Name from Tools configuration
    nodejs 'node-16'
}

3. Environment Variables

Define variables accessible throughout the pipeline:

environment {
    APP_VERSION = '1.0.0'
    DOCKER_IMAGE = 'myapp/backend'
    SERVER_CRED = credentials('server-cred-id')  // Loads credentials
}

Built-in Environment Variables:

• BUILD_NUMBER: Current build number
• BUILD_ID: Unique build identifier
• JOB_NAME: Name of the job
• BRANCH_NAME: Git branch being built
• GIT_COMMIT: Git commit hash

View all available variables at: http://your-jenkins/env-vars.html

4. Parameters

Make pipelines reusable with user inputs:

parameters {
    string(name: 'VERSION', defaultValue: '1.0.0', description: 'Version to deploy')
    choice(name: 'ENVIRONMENT', choices: ['dev', 'staging', 'prod'], description: 'Deployment environment')
    booleanParam(name: 'RUN_TESTS', defaultValue: true, description: 'Execute tests?')
}

// Access in pipeline
steps {
    echo "Deploying version ${params.VERSION} to ${params.ENVIRONMENT}"
}

5. Conditional Execution (when)

Execute stages based on conditions:

stage('Deploy to Production') {
    when {
        branch 'main'  // Only on main branch
    }
    steps {
        echo 'Deploying to production...'
    }
}

stage('Run Tests') {
    when {
        expression { params.RUN_TESTS == true }
    }
    steps {
        echo 'Running tests...'
    }
}

6. Post Actions

Execute logic after stages complete:

post {
    always {
        echo 'This runs regardless of pipeline result'
        cleanWs()  // Clean workspace
    }
    success {
        echo 'Pipeline succeeded!'
        // Send success notification
    }
    failure {
        echo 'Pipeline failed!'
        // Send alert to Slack/email
    }
}

Complete Pipeline Example

#!/usr/bin/env groovy

pipeline {
    agent any
    
    tools {
        maven "maven-3.9.11"
    }
    
    parameters {
        string(name: 'VERSION', defaultValue: '', description: 'Version to deploy')
        choice(name: 'ENVIRONMENT', choices: ['dev', 'staging', 'prod'], description: 'Target environment')
        booleanParam(name: 'EXECUTE_TESTS', defaultValue: true, description: 'Run tests?')
    }
    
    environment {
        APP_NAME = "my-java-app"
        DOCKER_REGISTRY = "docker.io"
    }
    
    stages {
        stage('Initialize') {
            steps {
                script {
                    echo "Starting pipeline for ${APP_NAME}"
                    echo "Build number: ${BUILD_NUMBER}"
                    echo "Branch: ${GIT_BRANCH}"
                    
                    if (params.VERSION?.trim()) {
                        echo "Deploying custom version: ${params.VERSION}"
                    }
                }
            }
        }
        
        stage('Build') {
            steps {
                script {
                    echo "Building application..."
                    sh "mvn clean package -DskipTests"
                }
            }
        }
        
        stage('Test') {
            when {
                expression { params.EXECUTE_TESTS }
            }
            steps {
                script {
                    echo "Running tests..."
                    sh "mvn test"
                }
            }
        }
        
        stage('Build Docker Image') {
            steps {
                script {
                    echo "Building Docker image..."
                    sh "docker build -t ${DOCKER_REGISTRY}/${APP_NAME}:${BUILD_NUMBER} ."
                }
            }
        }
        
        stage('Push to Registry') {
            steps {
                script {
                    withCredentials([usernamePassword(
                        credentialsId: 'docker-hub', 
                        usernameVariable: 'USER', 
                        passwordVariable: 'PASSWORD'
                    )]) {
                        sh 'echo "$PASSWORD" | docker login -u "$USER" --password-stdin'
                        sh "docker push ${DOCKER_REGISTRY}/${APP_NAME}:${BUILD_NUMBER}"
                    }
                }
            }
        }
        
        stage('Deploy') {
            when {
                expression { params.ENVIRONMENT == 'prod' }
                branch 'main'
            }
            steps {
                script {
                    def deployConfirm = input(
                        message: "Deploy to production?",
                        ok: "Deploy",
                        parameters: [
                            string(name: 'APPROVER', description: 'Your name')
                        ]
                    )
                    echo "Deploying to production. Approved by: ${deployConfirm}"
                    // Deployment commands here
                }
            }
        }
    }
    
    post {
        always {
            echo "Pipeline completed"
            cleanWs()
        }
        success {
            echo "Build successful! Image: ${DOCKER_REGISTRY}/${APP_NAME}:${BUILD_NUMBER}"
        }
        failure {
            echo "Build failed. Check logs for details."
        }
    }
}

Using External Groovy Scripts

For better organization, separate complex logic into external Groovy files:

Jenkinsfile:

def gv

pipeline {
    agent any
    
    tools {
        maven "maven-3.9.11"
    }
    
    stages {
        stage('Initialize') {
            steps {
                script {
                    gv = load "script.groovy"  // Load external script
                }
            }
        }
        
        stage('Build Jar') {
            steps {
                script {
                    gv.buildJar()  // Call function from script
                }
            }
        }
        
        stage('Build Image') {
            steps {
                script {
                    gv.buildImage()
                }
            }
        }
        
        stage('Deploy') {
            steps {
                script {
                    gv.deployApp()
                }
            }
        }
    }
}

script.groovy:

def buildJar() {
    echo "Building JAR file..."
    sh "mvn clean package"
}

def buildImage() {
    echo "Building Docker image..."
    sh "docker build -t myapp:${BUILD_NUMBER} ."
    
    withCredentials([usernamePassword(
        credentialsId: 'docker-hub', 
        usernameVariable: 'USER', 
        passwordVariable: 'PASSWORD'
    )]) {
        sh 'echo "$PASSWORD" | docker login -u "$USER" --password-stdin'
        sh "docker push myapp:${BUILD_NUMBER}"
    }
}

def deployApp() {
    echo "Deploying application..."
    // Deployment logic
}

return this  // Important: return script for use in Jenkinsfile

Why separate scripts?

• Better organization for complex pipelines
• Reusable functions
• Easier to maintain and test
• Cleaner Jenkinsfile

Pushing Images to Registries

Docker Hub

1. Create Credentials in Jenkins:

• Manage Jenkins → Credentials → Add Credentials
• Type: Username with password
• ID: docker-hub

2. Use in Pipeline:

stage('Push to Docker Hub') {
    steps {
        script {
            withCredentials([usernamePassword(
                credentialsId: 'docker-hub',
                usernameVariable: 'USERNAME',
                passwordVariable: 'PASSWORD'
            )]) {
                sh "docker build -t ${USERNAME}/java-app:${BUILD_NUMBER} ."
                sh 'echo $PASSWORD | docker login -u $USERNAME --password-stdin'
                sh "docker push ${USERNAME}/java-app:${BUILD_NUMBER}"
            }
        }
    }
}

Nexus Repository

Nexus is a private artifact repository commonly used in enterprises.

1. Configure Docker for Insecure Registry (if using HTTP):

On the machine running Docker:

# Edit daemon configuration
sudo vi /etc/docker/daemon.json

# Add:
{
  "insecure-registries": ["3.88.138.50:8082"]
}

# Restart Docker
sudo systemctl restart docker

2. Fix Jenkins Container Permissions (if needed):

docker exec -it -u 0 <container-id> bash
chmod 666 /var/run/docker.sock

3. Create Nexus Credentials in Jenkins:

• Same process as Docker Hub
• ID: nexus-docker

4. Push to Nexus:

stage('Push to Nexus') {
    steps {
        script {
            def nexusUrl = "3.88.138.50:8082"
            
            withCredentials([usernamePassword(
                credentialsId: 'nexus-docker',
                usernameVariable: 'USERNAME',
                passwordVariable: 'PASSWORD'
            )]) {
                sh "docker build -t ${nexusUrl}/java-app:${BUILD_NUMBER} ."
                sh "echo \$PASSWORD | docker login -u \$USERNAME --password-stdin ${nexusUrl}"
                sh "docker push ${nexusUrl}/java-app:${BUILD_NUMBER}"
            }
        }
    }
}

Multibranch Pipeline Deep Dive

Imagine you have:

• A main branch (production code)
• Multiple feature branches (feature/login, feature/payment)
• Bug fix branches (bugfix/cart-issue)

You want to:

• Build and test ALL branches
• Deploy ONLY the main branch to production

Multibranch Pipeline handles this automatically.

How it works:

1. Jenkins scans your repository
2. Finds all branches matching your criteria
3. For each branch with a Jenkinsfile, creates a pipeline
4. Builds automatically when branches change
5. Removes pipelines when branches are deleted

Jenkinsfile with Branch Logic:

pipeline {
    agent any
    
    stages {
        stage('Test') {
            steps {
                echo "Testing branch: ${BRANCH_NAME}"
                sh "mvn test"
            }
        }
        
        stage('Build') {
            steps {
                echo 'Building application...'
                sh "mvn clean package"
            }
        }
        
        stage('Deploy to Staging') {
            when {
                branch 'develop'
            }
            steps {
                echo 'Deploying to staging environment...'
                // Deploy to staging
            }
        }
        
        stage('Deploy to Production') {
            when {
                branch 'main'
            }
            steps {
                echo 'Deploying to production...'
                // Deploy to production
            }
        }
    }
}

Setting up Multibranch Pipeline:

1. New Item → Multibranch Pipeline
2. Branch Sources → Add Git
3. Add repository URL and credentials
4. Configure branch discovery (usually include all branches)
5. Save and scan

Jenkins will automatically discover all branches with a Jenkinsfile and create pipelines for them.

Continue reading in Part 2 where we cover Jenkins Shared Libraries, automated triggers, version management, troubleshooting, best practices, and scaling Jenkins for production.