Coder Legion - Recent posts tagged jenkins

MongoDB DR Drill Automation with Terraform, Python & Jenkins — How We Made Restores Boring

Thu, 02 Jul 2026 15:09:45 +0000

Backups Don't Save You. Restores Do.

We ran a MongoDB restore drill last quarter. It failed — not the restore itself, but the confidence. Nobody in the room was sure the data was actually intact. The service came back up, and we all just stared at each other.

That was the problem. So we fixed it by automating everything.

One Jenkins job now provisions infra, builds the replica set, restores from dumps, validates data integrity, and stores a full audit trail. Here's exactly how it works.

The Goal

Remove every manual, error-prone step from the DR process:

Identical restore flow across all environments
Automated replica set setup — no manual rs.initiate() typos
Real validation that proves data is intact, not just assumed
Full audit trail for post-mortems and compliance reviews

The Pipeline: 5 Stages

1. Infrastructure with Terraform

Every drill starts with clean infra. Terraform provisions EC2s, networking, and persistent volumes from scratch — same starting point every time. No leftover state. No "works on my machine" surprises.

resource "aws_instance" "mongo_node" {
  count         = 3
  ami           = var.mongo_ami
  instance_type = "t3.medium"
  tags = {
    Name = "mongo-dr-node-${count.index}"
    Role = "mongodb-replica"
  }
}

2. Replica Set Creation (Python)

Instead of manually running rs.initiate() and rs.add() and hoping the timing works, a Python script handles the entire setup — ordering, retries, and confirmation.

from pymongo import MongoClient
import time

def init_replica_set(primary_host, secondary_hosts):
    client = MongoClient(f"mongodb://{primary_host}:27017")
    config = {
        "_id": "rs0",
        "members": [{"_id": i, "host": h}
                    for i, h in enumerate([primary_host] + secondary_hosts)]
    }
    client.admin.command("replSetInitiate", config)
    # Wait for PRIMARY election
    for _ in range(30):
        status = client.admin.command("replSetGetStatus")
        if any(m["stateStr"] == "PRIMARY" for m in status["members"]):
            return True
        time.sleep(2)
    raise Exception("Replica set did not elect a PRIMARY in time")

Automating this removes timing issues and misconfiguration. Every replica set comes up the same way.

3. Backup & Restore

Backups are normalized into compressed archives. The restore unpacks a dump and applies it to the fresh nodes:

# Create dump
mongodump --host $SOURCE_HOST --db $DB_NAME \
  --out /backup/dump --gzip

# Restore to DR environment
mongorestore --host $DR_HOST --db $DB_NAME \
  /backup/dump/$DB_NAME --gzip --drop

4. Validation & Comparison — The Part Most Teams Skip

This is the step that actually builds confidence. The validation script:

Checks which collections exist (flags missing collections)
Compares document counts collection by collection
Compares indexes between source and restored DB
Samples _id values for obvious data mismatches

def validate_restore(source_uri, dr_uri, db_name):
    src = MongoClient(source_uri)[db_name]
    dr  = MongoClient(dr_uri)[db_name]

    report = {"status": "pass", "collections": {}}

    for col in src.list_collection_names():
        src_count = src[col].count_documents({})
        dr_count  = dr[col].count_documents({})
        src_idx   = sorted(src[col].index_information().keys())
        dr_idx    = sorted(dr[col].index_information().keys())

        match = (src_count == dr_count) and (src_idx == dr_idx)
        report["collections"][col] = {
            "count_match":  match,
            "source_count": src_count,
            "dr_count":     dr_count,
            "index_match":  src_idx == dr_idx
        }
        if not match:
            report["status"] = "fail"

    return report

Exit code 0 = counts and indexes match → Jenkins passes.
Non-zero = mismatch → Jenkins fails the build immediately.

No more guessing. No more staring at each other in the war room.

5. Jenkins Orchestration

Single Jenkins pipeline. Stages run sequentially, each one gated on the previous:

pipeline {
  agent any
  stages {
    stage('Provision Infra') {
      steps {
        sh 'terraform init && terraform apply -auto-approve'
      }
    }
    stage('Setup Replica Set') {
      steps {
        sh 'python3 scripts/init_replica_set.py'
      }
    }
    stage('Restore MongoDB') {
      steps {
        sh 'bash scripts/restore.sh'
      }
    }
    stage('Validate Restore') {
      steps {
        sh 'python3 scripts/validate_restore.py'
      }
    }
    stage('Archive Logs') {
      steps {
        archiveArtifacts artifacts: 'reports/*.json, logs/*.log'
      }
    }
  }
}

Every run is logged, every report is archived. When auditors ask if restores work — you show them a report with timestamps, counts, and index diffs. Not a gut feeling.

Lessons Learned

Automate infra, not just the restore. Terraform gives you a clean slate every drill. Manual infra setup introduces variability that hides real problems.

Validation is not optional. A restore that "seems fine" is not the same as a restore that is fine. Document count mismatches and missing indexes are easy to catch automatically and impossible to catch by eyeballing logs.

Logs equal trust. The audit trail is what makes your DR process credible to others — engineers, management, auditors. Without it, you're asking people to take your word for it.

Minimal input reduces errors. We trimmed required inputs to just host + DB name and let scripts infer the rest. Less to type = fewer mistakes under pressure.

Practice makes permanent. Each drill found a small improvement. After ten drills, the process was genuinely fast and boring — which is exactly what you want.

The Outcome

We went from a 3-hour manual war room exercise to a single Jenkins job anyone can trigger. The drills are now predictable, repeatable, and quick.

More importantly — everyone on the team believes the restores work, because the validation script proves it every single time.

Boring DR is good DR.

Running MongoDB in production? When did you last drill a full restore? Drop your setup in the comments — curious how teams handle validation.

Implementing an End-to-End CI/CD Pipeline for a Java ATM Simulation Application

Tue, 02 Dec 2025 23:48:32 +0000

Abstract
This case study outlines the design and implementation of a complete CI/CD pipeline for a Java ATM Simulation application. The project aimed to strengthen hands-on skills in DevOps automation, continuous integration, code quality analysis, artifact management, and deployment workflows using real-world tools on AWS.

Project Goals

Develop a fully automated CI/CD pipeline from code commit to deployment.

Integrate Jenkins, Maven, SonarQube, Nexus, and Tomcat into a streamlined workflow.

Deploy services on separate AWS EC2 instances to mimic production infrastructure.

Implement artifact versioning and reliable storage.

Enable automated deployment to an application server with minimal manual intervention.

Tools & Technology Stack

AWS EC2: Dedicated servers for Jenkins, SonarQube, Nexus, and Tomcat.

Jenkins: CI server and pipeline orchestration.

Maven: Build automation and dependency management.

SonarQube: Static code analysis and code quality reporting.

Nexus Repository Manager: Artifact storage and version control.

Apache Tomcat: Application server deployment.

GitHub: Source code management.

Java 11+: Application and tooling requirement.

Jenkins Plugins: Git, Maven, SSH Agent, SonarQube Scanner.

Infrastructure Setup
Four EC2 instances were provisioned, each hosting a separate tool to simulate production:

Jenkins on port 8080

SonarQube on port 9000

Nexus on port 8081

Tomcat on port 8080

Java was installed on all instances, and security groups were configured for inter-tool communication. Each server was validated for accessibility via a browser to ensure proper connectivity.

Pipeline Design & Implementation

Source Control Integration
The ATM Simulation project was pushed to GitHub. Jenkins was configured to clone the repository automatically on each build trigger using a declarative pipeline script.

Build & Test Automation
mvn clean package was used to compile code, execute tests, and generate artifacts. Maven ensured dependency management and consistent builds across environments.

Code Quality Analysis
SonarQube was integrated by updating pom.xml and Jenkins credentials. Running mvn sonar:sonar generated detailed code metrics, highlighting vulnerabilities and enforcing coding standards.

Artifact Management
A Maven repository was created in Nexus. pom.xml and Jenkins settings.xml were configured to upload artifacts via mvn deploy, ensuring version-controlled storage and rollback capability.

Automated Deployment to Tomcat
Tomcat manager roles and credentials were configured. Jenkins deployed the final WAR file, completing the continuous delivery process. Poll SCM triggers enabled automation for every commit.

Advanced Jenkins Configuration
The SSH Agent plugin simulated a master-slave architecture, enabling distributed builds and enhancing pipeline scalability.

Challenges & Solutions

Tool Version Conflicts: Resolved Java compatibility issues for SonarQube and Nexus.

Nexus Deployment Failures: Fixed misconfigurations in pom.xml and settings.xml.

Tomcat Deployment Errors: Corrected manager roles and permissions for successful deployment.

Pipeline Failures: Updated Jenkins plugins and validated pipeline scripts to resolve build issues.

Key Learnings

Hands-on experience in end-to-end CI/CD workflows.

Integration between source control, CI servers, artifact repositories, and application servers.

Best practices in artifact versioning, storage, and automated deployment.

Troubleshooting pipeline failures and ensuring tool compatibility.

Leveraging Jenkins plugins for scalable master-slave architectures.

Real-world automation of software delivery and cloud deployment.

Conclusion
This project provided practical exposure to enterprise-grade DevOps practices, including automation, CI/CD pipeline creation, artifact management, and deployment orchestration. It strengthened problem-solving, tool integration, and cloud deployment skills, demonstrating the ability to implement and manage robust DevOps workflows in real-world environments while reinforcing a commitment to continuous learning in automation and CI/CD.

Mastering Jenkins CI/CD and Cloud Fundamentals: My June–July DevOps Journey ✅

Sun, 27 Jul 2025 19:14:31 +0000

✨ Introduction

Over the past two months, I took a deep dive into the world of CI/CD pipelines, application security, and cloud fundamentals. My goal was to simulate a real-world enterprise environment using tools like Jenkins, Docker, Tomcat, and SonarQube—all while improving my grasp of DevOps principles. The cherry on top: I successfully cleared the AWS Cloud Practitioner certification.

This blog documents my learnings, setups, and the hands-on challenges I tackled. Whether you're starting your DevOps journey or looking to refine your pipeline, I hope this helps.

1. Setting Up Jenkins for Real-World CI/CD

I started by setting up a Jenkins controller and then created custom Jenkins agents using Docker. This allowed me to:

Isolate build environments
Control security boundaries
Reuse Docker images across pipelines

I structured separate pipelines for DEV and QA profiles, ensuring each environment had unique configurations and behavior. Jenkins' flexibility made it easy to parameterize the builds and manage them per environment.

2. Security & Secrets Management

One major focus was improving how environment variables and secrets are handled:

Used Jenkins Credentials Plugin to store secrets like JWT keys and database credentials
Passed these into Docker containers via docker-compose and environment variables in JAVA_OPTS
Ensured no secrets were hardcoded in the source or pipeline definitions

This was crucial for simulating a real enterprise-grade deployment pipeline.

3. Code Quality & Test Coverage

I integrated SonarQube with Jenkins to continuously analyze code quality. Here's what I did:

Installed the SonarQube plugin in Jenkins
Set up a local SonarQube server running in a Docker container
Used Maven goals (sonar:sonar) to trigger analysis from Jenkins

To measure test coverage, I integrated JaCoCo and ensured JUnit tests were run and reported independently.

These tools helped me:

Identify code smells and bugs early
Maintain a clean, modular codebase
Track testing metrics over time

4. Tomcat & Docker Compose Insights

I deployed my Spring Boot application using Tomcat within a Dockerized setup. Key takeaways:

Understood how Tomcat manages servlet deployment
Learned how to pass environment-specific properties (application-qa.properties, etc.) via JAVA_OPTS
Used docker-compose.yml to streamline configuration and ensure consistent environments

This was especially helpful in managing profile-specific behavior.

5. SSH & GitHub Automation

I explored the use of ssh-keygen and how SSH keys can be leveraged to:

Authenticate GitHub repositories from Jenkins
Enable secure, automated Git pulls
Avoid manual authentication during builds

This added a new layer of automation and security to the CI/CD flow.

6. Cloud Achievement: AWS Cloud Practitioner

Parallel to my DevOps practice, I prepared for and passed the AWS Cloud Practitioner certification. Key concepts I explored:

AWS Core Services (EC2, S3, RDS, Lambda)
Identity and Access Management (IAM)
Shared Responsibility Model
Cloud pricing and billing concepts

This knowledge gave me a stronger foundation in cloud architecture and how it supports DevOps.

✅ Key Takeaways

Jenkins pipelines can be secured and structured to simulate enterprise-grade scenarios
Docker agents are essential for isolating builds and improving reproducibility
Tools like SonarQube and JaCoCo provide critical insights into code health and coverage
Managing SSH and environment variables correctly enhances both automation and security
Understanding cloud fundamentals bridges the gap between development and operations

Conclusion

This journey was packed with hands-on learning, challenges, and rewarding moments. I now feel much more confident in designing CI/CD pipelines, securing deployments, and integrating quality gates into builds.

I'd love to hear how others approach similar setups—feel free to connect, comment, or share your tips. You can also find some of my sample configs and scripts on my GitHub.

Thanks for reading!