Coder Legion - Recent posts tagged ansible

⚙️From 4 Hours to 3 Minutes: Automating GCP Ingress Provisioning with Ansible

Tue, 27 Oct 2026 16:36:09 +0000

Context

Every time a new client was onboarded, their web application had to be deployed and exposed to the internet.

As part of the deployment, the infrastructure required several configuration steps across different services before the application was ready to receive traffic.

Problem

Although the deployment workflow was always the same, every new client required repeating the exact same operational tasks:

Creating GCP firewall rules
Retrieving the VM's public IP
Creating or updating Cloudflare DNS records
Installing and configuring Nginx
Generating Let's Encrypt SSL certificates
Validating the routing and HTTPS configuration

Since these tasks were performed manually for every client, the provisioning process typically took around four hours to complete. Besides being time-consuming, it also increased the likelihood of configuration mistakes and inconsistencies.

Solution

After repeating the same workflow several times, I realized the process was entirely deterministic. The only inputs that changed between deployments were values such as the domain name, backend ports, and service account.

Instead of spending hours repeating the same sequence of tasks, I automated the entire workflow with Ansible and integrated it into our CI/CD pipeline.

How the workflow looks

GitLab CI/CD
      │
      ▼
Run Ansible Playbook
      │
      ▼
Create GCP Firewall Rule
      │
      ▼
Retrieve VM Public IP
      │
      ▼
Create / Update Cloudflare DNS Record
      │
      ▼
Install Nginx
      │
      ▼
Generate HTTP Configuration
      │
      ▼
Request Let's Encrypt Certificate
      │
      ▼
Generate HTTPS Configuration
      │
      ▼
Reload Nginx
      │
      ▼
Application Available via HTTPS

A single CI/CD job orchestrates the entire workflow. Ansible handles the infrastructure configuration, DNS updates, reverse proxy setup, and SSL provisioning, making the deployment consistent and repeatable.

Implementation Highlights

Google Cloud Firewall Automation – Creates HTTP/HTTPS firewall rules through the GCP API.
Automatic Public IP Discovery – Retrieves the VM's external IP from the GCP metadata service instead of requiring manual input.
Cloudflare DNS Management – Updates DNS records automatically as part of the deployment.
Dynamic Nginx Configuration – Uses Jinja2 templates to generate HTTP and HTTPS configurations.
Automated SSL Provisioning – Obtains and configures Let's Encrypt certificates without manual intervention.
CI/CD Integration – The entire process runs automatically through GitLab CI.

Results

Before	After
~4 hours to provision and configure	~3 minutes through CI/CD
Multiple manual steps	Fully automated workflow
Manual DNS, firewall and SSL configuration	Managed by Ansible
Repetitive onboarding process	Consistent and repeatable deployment

Beyond the time savings, every deployment now follows the exact same process.

Instead of spending hours repeating the same steps, I simply trigger the pipeline and let the automation do the work.

Check the code

https://github.com/ja-lourenco/ansible-gcp-ingress-proxy

Why win_updates Alone Isn't Enough for Production Windows Patching — My AWX Approach

Wed, 07 Oct 2026 11:41:31 +0000

Most Windows patching guides show you a basic win_updates task and call it done. That works fine in a lab. But when you're managing 30 production servers, you need something that tells you what's going to happen before it happens, handles double reboots gracefully, auto-detects WSUS, and leaves a proper audit trail.

This is the playbook I actually run at work through AWX — four phases, one server at a time, with a human approval checkpoint before anything gets touched.

Pre-check — OS build, pending reboots, disk space, WSUS detection, update scan
Patch — two-pass patching with automatic reboot handling
Post-check — OS build verification, uptime, installed hotfixes
Validate — build comparison before vs after, final ping test

Each phase runs as a separate AWX job template using Ansible tags — so you're in full control of what runs and when. We patch one server at a time (serial: 1) so a failure on one host never cascades to the rest of the fleet.

The Architecture — How It Runs in AWX

Before looking at the playbook itself, here's how I've wired this up in AWX:

AWX Workflow: Windows Patching
│
├── Job Template 1: Pre-Check (tags: precheck)
│ └── [On Success]
├── Job Template 2: Patch (tags: patch)
│ └── [On Success]
├── Job Template 3: Post-Check (tags: postcheck)
│ └── [On Success]
└── Job Template 4: Validate (tags: validate)

Each job template runs the same playbook (windows_patching_with_clear_logs.yaml) but with a different tag. This gives you full visibility in AWX — you can see exactly which phase passed or failed, and stop the workflow at any point before moving to the next phase.

In our environment, all 30 servers are in the same AWX inventory. The playbook uses serial: 1 which means AWX runs through all four phases on server 1 before moving to server 2. One server at a time, fully controlled.

The Playbook Variables

At the top of the playbook I define the key variables that control the patching behaviour:

vars:
  patch_categories:
    - SecurityUpdates
    - CriticalUpdates
    - UpdateRollups
  windowslogs_dir: "windowslogs/windows-patching"
  patch_passes: 2
  upd_search_log: "C:\\Windows\\Temp\\ans_upd_search.log"

patch_categories — the three update types I install. I deliberately exclude FeaturePacks and DefinitionUpdates to keep patching focused on security and stability.

windowslogs_dir — where log files get saved on the Ansible controller, not on the Windows servers. Every host gets its own timestamped log file so you have a full audit trail.

patch_passes — set to 2 because Windows often needs a second pass after a reboot to pick up remaining updates, especially Cumulative Updates and Servicing Stack updates.

Pre-Tasks — Setting Up Log Files

Before any tasks run, I set up timestamped log file paths on the controller:

pre_tasks:
  - name: Build run timestamp on controller (for unique filenames)
    delegate_to: localhost
    run_once: true
    set_fact:
      date_str: "{{ lookup('ansible.builtin.pipe', 'date +%Y%m%d_%H%M%S') }}"
    tags: [always]

  - name: Ensure windowslogs dir exists on controller
    delegate_to: localhost
    run_once: true
    ansible.builtin.file:
      path: "{{ windowslogs_dir }}"
      state: directory
      mode: "0755"
    tags: [always]

  - name: Build per-host log file paths (on controller)
    set_fact:
      precheck_file: "{{ windowslogs_dir }}/{{ inventory_hostname }}_precheck_{{ date_str }}.txt"
      postcheck_file: "{{ windowslogs_dir }}/{{ inventory_hostname }}_postcheck_{{ date_str }}.txt"
      updates_file:  "{{ windowslogs_dir }}/{{ inventory_hostname }}_updates_{{ date_str }}.txt"
    tags: [always]

The tags: [always] means these tasks run regardless of which tag you pass — log paths are always set up first.

Phase 1 — Pre-Check

This is the most important phase. I never patch a server without running pre-checks first.

1. Get the Full OS Build Including UBR

The standard OS build number isn't enough. Windows uses an Update Build Revision (UBR) on top of the major build — so two servers with the same OsBuildNumber might be at different patch levels:

- name: Get OS version/build before patch (full build)
  ansible.windows.win_shell: |
    $ci = Get-ComputerInfo
    $cv = Get-ItemProperty 'HKLM:\SOFTWARE\Microsoft\Windows NT\CurrentVersion'
    $ubr = $cv.UBR
    $major = $ci.OsBuildNumber
    $full = '{0}.{1}' -f $major, $ubr
    [PSCustomObject]@{
      OsVersion     = $ci.OsVersion
      OsBuildNumber = $major
      UBR           = $ubr
      FullBuild     = $full
      Caption       = $ci.OsName
    } | ConvertTo-Json -Compress | Out-String
  register: pre_os
  check_mode: no
  changed_when: false
  tags: [precheck]

I then validate this output — if it comes back as UNKNOWN, the playbook fails immediately:

- name: Validate pre_os output
  ansible.builtin.assert:
    that:
      - pre_os_version     != 'UNKNOWN'
      - pre_os_build_major != 'UNKNOWN'
      - pre_os_build_full  != 'UNKNOWN'
    fail_msg: "Failed to parse OS info on {{ inventory_hostname }}."
  tags: [precheck]

2. Check for Pending Reboots

If a server already has a pending reboot, patching it is risky. I check two registry keys:

- name: Check pending reboot before patch (PowerShell fallback)
  ansible.windows.win_shell: |
    $paths = @(
      "HKLM:\SOFTWARE\Microsoft\Windows\CurrentVersion\Component Based Servicing\RebootPending",
      "HKLM:\SOFTWARE\Microsoft\Windows\CurrentVersion\WindowsUpdate\Auto Update\RebootRequired"
    )
    $pending = $false
    foreach ($p in $paths) { if (Test-Path $p) { $pending = $true } }
    if ($pending) { "true" } else { "false" }
  register: pre_reboot_raw
  check_mode: no
  changed_when: false
  tags: [precheck]

- name: Stop if a reboot is already pending
  ansible.builtin.fail:
    msg: "Reboot is already pending on {{ inventory_hostname }}. Reboot first, then patch."
  when: pre_reboot.reboot_pending | default(false)
  tags: [precheck]

3. Check Disk Space

- name: Check free space on system drive (bytes)
  ansible.windows.win_shell: |
    $sys = ([System.Environment]::GetEnvironmentVariable('SystemDrive'))
    Get-PSDrive -Name $sys.TrimEnd(':') |
    Select-Object Used,Free | ConvertTo-Json -Compress | Out-String
  register: pre_disk
  check_mode: no
  changed_when: false
  tags: [precheck]

4. WSUS Auto-Detection

One of the things I'm most proud of in this playbook. It auto-detects whether WSUS is configured and adjusts accordingly:

- name: Detect WSUS policy (is WSUS enforced?)
  ansible.windows.win_shell: |
    $wuKey = 'HKLM:\SOFTWARE\Policies\Microsoft\Windows\WindowsUpdate'
    $auKey = 'HKLM:\SOFTWARE\Policies\Microsoft\Windows\WindowsUpdate\AU'
    $obj = [ordered]@{
      UseWUServer = $null
      WUServer    = $null
      WUStatus    = $null
    }
    if (Test-Path $auKey) {
      $obj.UseWUServer = (Get-ItemProperty $auKey -ErrorAction SilentlyContinue).UseWUServer
    }
    if (Test-Path $wuKey) {
      $p = Get-ItemProperty $wuKey -ErrorAction SilentlyContinue
      $obj.WUServer  = $p.WUServer
      $obj.WUStatus  = $p.WUStatusServer
    }
    $obj | ConvertTo-Json -Compress | Out-String
  register: wsus_raw
  check_mode: no
  changed_when: false
  tags: [precheck]

- name: Decide update server selection (WSUS vs Microsoft Update)
  set_fact:
    update_server_selection: >-
      {{
        'managed_server'
        if (wsus_use_server == 1 and (wsus_url | length > 0))
        else 'windows_update'
      }}
  tags: [precheck]

5. Async Update Scan

Before touching anything, I do a full scan without downloading or installing:

- name: Kick off update search (no download/install)
  ansible.windows.win_updates:
    state: searched
    category_names: "{{ patch_categories }}"
    server_selection: "{{ update_server_selection }}"
    log_path: "{{ upd_search_log }}"
  register: pre_search_async
  async: 7200
  poll: 0
  tags: [precheck]

- name: Wait up to 20 minutes for search to finish
  async_status:
    jid: "{{ pre_search_async.ansible_job_id }}"
  register: pre_search
  until: pre_search.finished
  retries: 40
  delay: 30
  tags: [precheck]

6. Predict Patching Complexity

- name: Predict complexity (long run / multi-reboot heuristic)
  set_fact:
    candidate_updates: "{{ pre_search.updates | default([]) }}"
    will_be_long: >-
      {{ (pre_search.updates | default([])
          | selectattr('title','search','Cumulative Update|Servicing Stack|Feature Update')
          | list | length) > 0 }}
    predicted_passes: "{{ 2 if (pre_search.updates | default([])
          | selectattr('title','search','Cumulative Update|Servicing Stack')
          | list | length) > 0 else 1 }}"
  tags: [precheck]

7. Save Pre-Check Log

- name: Save pre-check summary (controller)
  delegate_to: localhost
  ansible.builtin.copy:
    dest: "{{ precheck_file }}"
    mode: "0644"
    content: |
      Host: {{ inventory_hostname }}
      Caption: {{ pre_os_caption }}
      OS Version: {{ pre_os_version }}
      OS Build (major): {{ pre_os_build_major }}
      OS Build (full) : {{ pre_os_build_full }}
      Last Boot: {{ pre_uptime.stdout | trim }}
      Pending Reboot: {{ pre_reboot.reboot_pending | default(false) }}
      Disk (bytes): {{ pre_disk.stdout | trim }}
      Candidate Updates: {{ (candidate_updates | length) if candidate_updates is defined else 0 }}
      Predicted Long/Multi-Reboot: {{ will_be_long | default(false) }}
      Suggested Passes: {{ predicted_passes | default(1) }}
      Updates:
      {% for u in candidate_updates %}
        - {{ u.title if (u is mapping and 'title' in u) else (u|string) }}
      {% endfor %}
  tags: [precheck]

Phase 2 — Patch

Once pre-checks pass and I've reviewed the log, I launch the patch job template in AWX. Two passes — first installs and reboots, second catches anything remaining:

- name: PATCH PASS 1 - install updates
  ansible.windows.win_updates:
    category_names: "{{ patch_categories }}"
    server_selection: "{{ update_server_selection }}"
    reboot: yes
    reboot_timeout: 3600
  register: patch_pass1
  tags: [never, patch]

- name: Wait for host to return (pass 1)
  wait_for_connection:
    delay: 20
    timeout: 1200
  when: patch_pass1.reboot_required | default(false)
  tags: [never, patch]

- name: PATCH PASS 2 - install remaining updates
  ansible.windows.win_updates:
    category_names: "{{ patch_categories }}"
    server_selection: "{{ update_server_selection }}"
    reboot: yes
    reboot_timeout: 3600
  register: patch_pass2
  when: patch_passes|int >= 2
  tags: [never, patch]

- name: Wait for host to return (pass 2)
  wait_for_connection:
    delay: 20
    timeout: 1200
  when:
    - patch_passes|int >= 2
    - patch_pass2.reboot_required | default(false)
  tags: [never, patch]

tags: [never, patch] — the never means these tasks are skipped unless you explicitly pass --tags patch. This prevents accidental patching.

Phase 3 — Post-Check

- name: Get OS version/build postchecks (full build)
  ansible.windows.win_shell: |
    $ci = Get-ComputerInfo
    $cv = Get-ItemProperty 'HKLM:\SOFTWARE\Microsoft\Windows NT\CurrentVersion'
    $ubr = $cv.UBR
    $major = $ci.OsBuildNumber
    $full = '{0}.{1}' -f $major, $ubr
    [PSCustomObject]@{
      OsVersion     = $ci.OsVersion
      OsBuildNumber = $major
      UBR           = $ubr
      FullBuild     = $full
    } | ConvertTo-Json -Compress | Out-String
  register: post_os
  tags: [never, postcheck]

- name: Get installed hotfixes postchecks
  ansible.windows.win_shell: >
    Get-HotFix | Sort-Object InstalledOn |
    Select-Object HotFixID, InstalledOn |
    Format-Table -HideTableHeaders | Out-String
  register: hotfixes
  tags: [never, postcheck]

- name: Save post-check summary (controller)
  delegate_to: localhost
  ansible.builtin.copy:
    dest: "{{ windowslogs_dir }}/{{ inventory_hostname }}_postcheck_{{ date_str }}.txt"
    mode: "0644"
    content: |
      Host: {{ inventory_hostname }}
      OS Version (post): {{ post_os_version }}
      OS Build (post, major): {{ post_os_build_major }}
      OS Build (post, full) : {{ post_os_build_full }}
      Last Boot (post): {{ post_uptime.stdout | trim }}
      Installed Updates:
      {{ hotfixes.stdout | trim }}
  tags: [never, postcheck]

Phase 4 — Validate

- name: Summarize patch result
  ansible.builtin.debug:
    msg: |
      Patch Result for {{ inventory_hostname }}:
      - Categories: {{ patch_categories | join(', ') }}
      - Pass1: installed={{ (patch_pass1.updates | default([])) | length }},
               reboot={{ patch_pass1.reboot_required | default(false) }}
      - Pass2: installed={{ (patch_pass2.updates | default([])) | length
               if (patch_pass2 is defined) else 0 }},
               reboot={{ patch_pass2.reboot_required | default(false)
               if (patch_pass2 is defined) else false }}
      - OS Build Before (full): {{ pre_os_build_full | default('N/A') }}
      - OS Build After  (full): {{ post_os_build_full | default('N/A') }}
      - Status: {{
          'PATCH SUCCESSFUL - OS BUILD CHANGED'
          if (pre_os_build_full is defined and post_os_build_full is defined
              and pre_os_build_full != post_os_build_full)
          else 'NO BUILD CHANGE - verify installed hotfixes or CU cadence'
        }}
  tags: [never, validate]

- name: Final ping test
  ansible.windows.win_ping:
  tags: [never, validate]

Setting This Up in AWX

Step 1 — Create Four Job Templates

Job Template	Playbook	Job Tags
Windows Pre-Check	windows_patching_with_clear_logs.yaml	precheck
Windows Patch	windows_patching_with_clear_logs.yaml	patch
Windows Post-Check	windows_patching_with_clear_logs.yaml	postcheck
Windows Validate	windows_patching_with_clear_logs.yaml	validate

Step 2 — Create the Workflow

AWX UI → Templates → Add → Add Workflow Template
Name: Windows Patching Workflow
Click Save → Workflow Visualiser

Chain the four job templates together using On Success:

Pre-Check → [On Success] → Patch → [On Success] → Post-Check → [On Success] → Validate

Step 3 — Add a Manual Approval Node

Between Pre-Check and Patch I add a manual Approval node so someone reviews the pre-check logs before patching starts:

In the Workflow Visualiser → between Pre-Check and Patch
Click + → select Approval
Name: Review Pre-Check Logs Before Patching
Timeout: 4 hours

This is the human checkpoint before anything touches production.

Errors I Hit in Production

Error 1 — WinRM Drops After Reboot and Never Comes Back

Fix — increase the wait_for_connection timeout:

wait_for_connection:
  delay: 20
  timeout: 1800

Error 2 — WSUS Returning No Updates

WSUS hadn't approved the latest Cumulative Update. Fix — override at runtime in AWX by passing as an extra variable:

update_server_selection: windows_update

Error 3 — UBR Not Changing After Patching

Check Get-HotFix output in the post-check log. If new KBs are listed even though the UBR didn't change, only smaller updates installed — worth investigating before signing off.

Error 4 — Async Update Scan Timing Out

Fix — increase async timeout and retries:

async: 7200
retries: 80
delay: 30

What This Gives You

For 30 production Windows servers, this approach gives you:

A full audit trail — pre and post check logs per server per patching run
Safety gates — pending reboot detection, disk space checks and manual approval before patching starts
Intelligence — WSUS auto-detection means the same playbook works across different network segments
Visibility in AWX — each phase is a separate node in the workflow so you can see exactly where something failed
Confidence — build comparison before and after tells you definitively whether the Cumulative Update actually applied

What's Next

The next improvement I'm planning is adding email notifications to this workflow — so when patching fails on any of the 30 servers at 2AM, I get an alert immediately rather than finding out in the morning.

Drop your questions in the comments — happy to help!

— Sireesha

From Shell Scripts to Scalable Automation: Why Ansible?

Sat, 08 Aug 2026 17:39:49 +0000

Most of us start our automation journey with 𝗦𝗵𝗲𝗹𝗹 𝗦𝗰𝗿𝗶𝗽𝘁𝗶𝗻𝗴—and that's exactly how it should be.

It's simple, powerful, and a great way to automate repetitive tasks.

But as your infrastructure grows, you'll eventually hit its limits.

🚨 Imagine this...
You need to configure 100 or even 1,000 servers.
Yes, you can SSH into each server and execute a shell script.
But managing automation at this scale becomes challenging.

⚠️ Challenges with Shell Scripting
❌ No Idempotency — Running the same script multiple times can repeat tasks and cause errors unless you add custom checks.

❌ Limited Error Handling — You must manually validate exit codes ($?) and handle failures yourself.

❌ Complex at Scale — Large scripts quickly become difficult to maintain with retries, logging, validations, and conditions.

❌ Limited Scalability — Shell executes sequentially by default. Parallel execution is possible using & and wait, but managing jobs, failures, and thousands of SSH connections adds complexity.

❌ No Built-in Inventory — You manage server IPs and SSH logic yourself.

❌ Distribution-Specific Logic — Different Linux distributions often require different commands.

✅ So, why Ansible?

Ansible is an open-source, agentless automation tool that follows a push-based architecture.

It connects to remote servers over SSH, pushes tasks from the control node, and executes them without installing any agent on the managed nodes.

Key Advantages
✅ Idempotent by default

✅ Inventory Management

✅ Parallel Execution

✅ Better Error Handling

✅ Configuration Management

✅ Rich Modules

✅ Readable & Scalable YAML Playbooks

💡 Shell Scripting is still a fundamental skill for every DevOps engineer.

But as your infrastructure grows, writing more shell code isn't always the answer.

Sometimes, choosing the right tool is.

The question isn't whether Shell Scripting is good or bad.

The real question is: At what point does your automation stop scaling?

Follow my profile for more practical DevOps content and real-world automation insights. 🚀

Coolify: The Complete Manual Setup Guide (For When the Auto-Install Script Won't Cut It)

Thu, 23 Jul 2026 05:15:31 +0000

Coolify's one-line install script is great — until it isn't. Right now it officially supports Ubuntu 20.04, 22.04, and 24.04 LTS. If you're running anything newer (Ubuntu's already on 26.04 LTS), the script won't work and you're left doing it manually.

This is that manual walkthrough — set up in the order that fits a security-first VPS workflow rather than the order Coolify's own docs use. If you've been following along with the Ansible playbooks from earlier in this series, this picks up right where that left off.

Minimum Hardware Requirements

CPU: 2 cores
Memory: 2 GB RAM
Storage: 30 GB free

Coolify can technically run below this, but it's not recommended.

Prerequisites

Before touching Coolify itself, you'll need:

SSH access to your VPS
CURL installed
Docker Engine installed

If you're reconnecting to a server you've rebuilt or re-provisioned, clear the old fingerprint first:

ssh-keygen -f '/home/your-path/.ssh/known_hosts' -R 'your-vps-ip'

Installing SSH

If you followed the earlier videos in this series, OpenSSH is already installed. If not:

sudo apt update && sudo apt install -y openssh-server

Confirm it's running and check which port it's listening on (you should have already changed this from the default 22 — see the VPS security video):

sudo systemctl status ssh
sudo ss -tulpn | grep ssh

Installing CURL

sudo apt update && sudo apt install -y curl
curl --version

curl and ca-certificates also get installed as part of the apt-update Ansible playbook below, so this may already be handled.

Running the First Ansible Playbook

Connect Ansible to the VPS:

ANSIBLE_HOST_KEY_CHECKING=FALSE ansible -i ./inventory/hosts vpsDemo -m ping --user root --ask-pass

Then run the update playbook:

ansible-playbook ./playbooks/apt-update.yml --user root -e "ansible_port=22" --ask-pass --ask-become-pass -i ./inventory/hosts

If you haven't set up the Ansible inventory and playbooks from the earlier videos, do that first — this guide assumes they're already in place.

Installing Docker Engine

Remove any conflicting packages first:

sudo apt remove $(dpkg --get-selections docker.io docker-compose docker-compose-v2 docker-doc podman-docker containerd runc | cut -f1)

Add Docker's official GPG key and repo:

sudo apt update
sudo apt install ca-certificates curl
sudo install -m 0755 -d /etc/apt/keyrings
sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
sudo chmod a+r /etc/apt/keyrings/docker.asc

sudo tee /etc/apt/sources.list.d/docker.sources <<EOF
Types: deb
URIs: https://download.docker.com/linux/ubuntu
Suites: $(. /etc/os-release && echo "${UBUNTU_CODENAME:-$VERSION_CODENAME}")
Components: stable
Architectures: $(dpkg --print-architecture)
Signed-By: /etc/apt/keyrings/docker.asc
EOF

sudo apt update
sudo apt install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin -y

Verify it worked:

sudo systemctl status docker
sudo docker run hello-world

Creating a Non-Root Admin User

Coolify's own docs assume you're using the root account. Since root login is disabled as part of the security hardening earlier in this series, we create a dedicated user with passwordless sudo instead.

Run the second playbook to create that user:

ansible-playbook ./playbooks/basic-secure.yml --user root -e "ansible_port=22" --ask-pass --ask-become-pass -i ./inventory/hosts

Then set up that user's SSH directory:

sudo mkdir -p /home/your-sudo-user/.ssh
sudo touch /home/your-sudo-user/.ssh/authorized_keys
sudo chmod 700 /home/your-sudo-user/.ssh
sudo chmod 600 /home/your-sudo-user/.ssh/authorized_keys

Setting Up Coolify's Directory Structure

sudo mkdir -p /data/coolify/{source,ssh,applications,databases,backups,services,proxy,webhooks-during-maintenance}
sudo mkdir -p /data/coolify/ssh/{keys,mux}
sudo mkdir -p /data/coolify/proxy/dynamic

Generating and Adding an SSH Key

sudo ssh-keygen -f /data/coolify/ssh/keys/*Emails are not allowed*rnal -t ed25519 -N '' -C your-sudo-user@coolify

Add the public key to the authorized_keys file:

cat /data/coolify/ssh/keys/*Emails are not allowed* | sudo tee -a /home/your-sudo-user/.ssh/authorized_keys

Pulling Coolify's Configuration Files

sudo curl -fsSL https://cdn.coollabs.io/coolify/docker-compose.yml -o /data/coolify/source/docker-compose.yml
sudo curl -fsSL https://cdn.coollabs.io/coolify/docker-compose.prod.yml -o /data/coolify/source/docker-compose.prod.yml
sudo curl -fsSL https://cdn.coollabs.io/coolify/.env.production -o /data/coolify/source/.env
sudo curl -fsSL https://cdn.coollabs.io/coolify/upgrade.sh -o /data/coolify/source/upgrade.sh

Generating Secure Environment Values

Only run these once, on first install. Changing them later can break Coolify. Back them up somewhere safe.

sudo sed -i "s|APP_ID=.*|APP_ID=$(openssl rand -hex 16)|g" /data/coolify/source/.env
sudo sed -i "s|APP_KEY=.*|APP_KEY=base64:$(openssl rand -base64 32)|g" /data/coolify/source/.env
sudo sed -i "s|DB_PASSWORD=.*|DB_PASSWORD=$(openssl rand -base64 32)|g" /data/coolify/source/.env
sudo sed -i "s|REDIS_PASSWORD=.*|REDIS_PASSWORD=$(openssl rand -base64 32)|g" /data/coolify/source/.env
sudo sed -i "s|PUSHER_APP_ID=.*|PUSHER_APP_ID=$(openssl rand -hex 32)|g" /data/coolify/source/.env
sudo sed -i "s|PUSHER_APP_KEY=.*|PUSHER_APP_KEY=$(openssl rand -hex 32)|g" /data/coolify/source/.env
sudo sed -i "s|PUSHER_APP_SECRET=.*|PUSHER_APP_SECRET=$(openssl rand -hex 32)|g" /data/coolify/source/.env

Permissions and Docker Setup

Set correct ownership and permissions:

sudo chown -R 9999:root /data/coolify
sudo find /data/coolify -type d -exec chmod 755 {} \;
sudo find /data/coolify -type f -exec chmod 644 {} \;
sudo chown -R your-sudo-user:your-sudo-user /home/your-sudo-user/.ssh

Create the Docker network Coolify expects:

sudo docker network create --attachable coolify

Add your user to the Docker group:

sudo usermod -aG docker your-sudo-user

Starting Coolify

sudo docker compose --env-file /data/coolify/source/.env -f /data/coolify/source/docker-compose.yml -f /data/coolify/source/docker-compose.prod.yml up -d --pull always --remove-orphans --force-recreate

Confirm it's running:

sudo docker ps

Then visit http://YOUR-SERVER-IP:8000 in your browser.

Setup your SSL Certificates

Coolify's reverse-proxy Traefik does this under the hood automatically.

In your DNS records add two A records.

A     @     your-vps-ip-address        14400
A     *     your-vps-ip-address        14400

In your Coolify dashboard go to settings in the left side column. In the input box marked URL type your https subdomain for Coolify there.

https://coolify.yourdomain.com

Traefik will automatically apply a Let's Encrypt certificate to your Coolify subdomain.

Disable Port 8000

Disable port 8000 on your VPS. If your hosting provider allows you to configure a firewall from your dashboard, disable it from there. To disable port 8000, simply write rules that allow the ports you want and block everything else.

Wrap-Up

That's a full manual Coolify install on a hardened, non-root VPS — no automated script required. From here, Coolify handles the rest: connecting your Git repos, setting up applications, and managing deployments.

If you hit issues with the automated script on a newer Ubuntu release, this manual path should get you unblocked. Questions or corrections welcome in the comments.

I automated my homelab so hard it started naming servers after Tarantino characters

Sat, 18 Jul 2026 00:47:17 +0000

title: I automated my homelab so hard it started naming servers after Tarantino characters
published: true
description: What started as avoiding the Proxmox UI turned into OpenTofu, Ansible, dynamic inventory, and a bunch of bugs I didn't expect.
tags: devops, opentofu, ansible, homelab

cover_image: https://raw.githubusercontent.com/Ayushverma8/homelab/main/docs/blog-cover.png

This started because I was lazy.

Every time I needed a new container on my Proxmox server, I'd open the web UI, click Create CT, pick a template, type a hostname, set the specs, configure networking, hit create, wait, then SSH in and install the same packages I always install. Fifteen minutes of clicking for something I do regularly.

So one evening I wrote a main.tf to automate it. That should have been the end of it. It wasn't.

The first version was embarrassing

provider "proxmox" {
  pm_api_url = "https://192.168.2.250:8006/api2/json"
}

resource "proxmox_lxc" "test_container" {
  target_node  = "proxmox"
  hostname     = "test-lxc"
  cores        = 1
  memory       = 512
}

API URL hardcoded in source. Specs as magic numbers. Want a second container? Copy-paste the whole block. It worked, but anyone looking at this in a code review would have things to say.

Then I kept going

I split it into proper files. Moved every value into variables with defaults. Added validation so cores = 0 or disk_size = "big" would fail at plan time instead of blowing up on the Proxmox API with some cryptic error.

Then I switched to a map with for_each:

lxc_containers = {
  testbox = {}
  docker  = { cores = 4, memory = 4096, disk_size = "64G" }
}

testbox = {} gives you a container with sane defaults. Override only what's different. Adding a container is one line, not a resource block.

The Tarantino thing

I needed hostnames. Didn't want container-1, container-2. That's boring and you can never remember which is which.

So I threw 30 Tarantino characters into a list and used random_shuffle to pick one per container, plus a random_id hex suffix to avoid collisions. Now when I tofu apply, my container might come back as beatrix-a3f1b2 or django-7c9e04. My Proxmox dashboard looks ridiculous and I love it.

SSH keys without managing SSH keys

Instead of copying keys around, the config pulls them from GitHub:

data "http" "github_keys" {
  url = "https://github.com/${var.github_username}.keys"
}

Every apply fetches the latest keys. Add a new key to GitHub, run the pipeline, done. No ssh-copy-id, no Ansible vault for public keys.

Where it got hard: Ansible + DHCP

After creating containers, I wanted Ansible to configure them. Install packages, set timezone, run updates. But the containers use DHCP, so at plan time there's no IP. The output just says ip = "dhcp". Not helpful.

I wrote a script that calls the Proxmox API after boot to get the actual IP:

ip=$(curl -sk "${PM_API_URL}/nodes/${node}/lxc/${vmid}/interfaces" \
  -H "Authorization: PVEAPIToken=..." \
  | jq -r '.data[] | select(.name=="eth0") | .["ip-addresses"][]
    | select(.["ip-address-type"]=="inet") | .["ip-address"]')

Generates an Ansible inventory on the fly. No hardcoded IPs.

Sounds clean, right? It wasn't.

Everything that went wrong

Wrong field name. The script was producing ansible_host: null for every container. Ansible tried to resolve the hostname, which obviously didn't exist in DNS, and died. I spent 15 minutes curling the Proxmox API manually before I noticed: the field is ip-address, not inet-addr. Every example I'd seen online used inet-addr. One jq filter change and it worked.

Race condition I didn't think about. The Makefile ran tofu apply then immediately generated inventory. But the container had literally just booted. DHCP hadn't assigned an IP yet. The inventory came back empty, Ansible ran against zero hosts, and the whole thing "succeeded" with nothing actually provisioned. No errors. No warnings. Just silence.

I added retry logic. The script now polls the API every 5 seconds, up to a minute, waiting for all containers to have IPs before generating inventory. Not elegant, but it works every time.

Lifecycle rules that created a deadlock. I wanted to protect important containers from accidental tofu destroy. OpenTofu has prevent_destroy for this, but it has to be a literal true or false in the code. You can't pass a variable. So I ended up with two resource blocks: proxmox_lxc.protected and proxmox_lxc.unprotected, and a flag in tfvars to control which pool a container landed in.

This worked until I tried to flip the flag. Changing a container from protected to unprotected meant OpenTofu wanted to destroy the protected one and recreate it in the unprotected block. But prevent_destroy blocked the destroy. Complete deadlock. The only fix was tofu state mv to manually move resources between blocks.

I spent an hour on this before deciding it was absurd for a homelab. Deleted the whole thing. One resource block, no flags, no state gymnastics.

The Cloudflare Tunnel rabbit hole. I thought it'd be cool to SSH into my containers from anywhere. Set up Cloudflare Tunnel, created DNS records, installed cloudflared on a container. Everything looked right. Then: remote error: tls: handshake failure. Spent another hour debugging tunnel configs, ingress rules, Access policies.

Then I asked myself when I last needed to SSH into my homelab from outside my house. The answer was never. Deleted the entire Cloudflare integration. Over-engineering is a real disease.

CI that passed locally and failed in Actions. ansible-lint was green on my machine because I had community.general installed from an old project. The GitHub Actions runner didn't have it. Pipeline went red. Then lint complained my variables didn't have a role prefix. Then it complained about a missing newline at the end of a file. Three separate commits to fix linting issues. Should have run it locally first.

Where it ended up

make up

That's the whole thing. Creates containers, waits for IPs, provisions with Ansible, prints this:

============================================
Infrastructure up and provisioned
============================================

SSH into your containers:
  ssh root@192.168.2.85

make down tears it all down. make up again and you get a fresh set. Different Tarantino name, same packages, same config. Ansible is idempotent: first run changed=4, second run changed=0.

What I'd skip next time

The prevent_destroy experiment. Not worth the state complexity for a homelab.

Cloudflare Tunnel. Cool idea, but I didn't need it. Building for a problem you don't have is how features become baggage.

And I'd run the linter locally before pushing. Not after CI tells me what I already should have caught.

The repo

github.com/ayushverma8/homelab

Clone it, fill in your Proxmox credentials, make up. Let me know what Tarantino character you get.

The Ansible Playbook that will Harden Your VPS

Tue, 14 Jul 2026 21:06:17 +0000

Ansible Playbooks — Automate VPS Hardening, User Creation & Removal

Link to youtube Video on these playbooks

Introduction

In this post I'm sharing 3 Ansible playbooks I use to manage my VPS servers. I won't go into great detail on tasks here — if you're new to Ansible check out my previous post and video first.

The three playbooks covered today:

basic-secure.yml — automate VPS hardening
add-vps-user.yml — semi-automated user creation
remove-vps-user.yml — completely remove a user and their privileges

📚 Resources:

Ansible Documentation

Ansible Collection Index

The Basics

A task is broken up into four parts:

Part	Description
Name	A plain-text description of what the task does
Collection	The Ansible content bundle the module belongs to
Module	The tool that executes the action
Parameters	The specific options passed to the module

Playbook 1 — basic-secure.yml

This playbook fully hardens a fresh Ubuntu VPS in a single command. It prompts you for a custom admin username, generates a random 16-character password, configures UFW, installs Fail2Ban, and moves SSH to port 2222.

---
- name: Harden Ubuntu VPS Security Configuration
  hosts: vpsDemo
  gather_facts: true
  vars_prompt:
    - name: 'custom_admin_user'
      prompt: 'Enter the custom username for your main administrator account'
      private: false
  tasks:
    # 1. GENERATE RANDOM PASSWORD
    - name: Generate random password
      set_fact:
        new_admin_password: "{{ lookup('password', '/dev/null chars=ascii_letters,digits,hexdigits length=16') }}"

    # 2. CREATE SUDO USER
    - name: Ensure the custom admin user exists
      ansible.builtin.user:
        name: '{{ custom_admin_user }}'
        password: "{{ new_admin_password | password_hash('sha512') }}"
        shell: /bin/bash
        state: present
        groups: sudo
        append: true

    - name: Allow the admin user to use sudo without a password prompt
      ansible.builtin.copy:
        content: "{{ custom_admin_user }} ALL=(ALL) NOPASSWD:ALL\n"
        dest: '/etc/sudoers.d/{{ custom_admin_user }}'
        mode: '0440'
        validate: /usr/sbin/visudo -cf %s

    # 3. CONFIGURE UFW FIREWALL
    - name: Reset UFW to default settings
      community.general.ufw:
        state: reset

    - name: Set UFW default policies to deny incoming
      community.general.ufw:
        policy: deny
        direction: incoming

    - name: Open Port 80 (HTTP)
      community.general.ufw:
        rule: allow
        port: '80'
        proto: tcp

    - name: Open Port 443 (HTTPS)
      community.general.ufw:
        rule: allow
        port: '443'
        proto: tcp

    - name: Open Custom SSH Port 2222
      community.general.ufw:
        rule: allow
        port: '2222'
        proto: tcp

    - name: Enable UFW Firewall
      community.general.ufw:
        state: enabled

    # 4. INSTALL FAIL2BAN
    - name: Install Fail2Ban
      ansible.builtin.apt:
        name: fail2ban
        state: present
        update_cache: true

    - name: Ensure Fail2Ban is running and enabled on boot
      ansible.builtin.service:
        name: fail2ban
        state: started
        enabled: true

    # 5. HARDEN SSH
    - name: Configure SSH to use custom port 2222
      ansible.builtin.lineinfile:
        path: /etc/ssh/sshd_config
        regexp: '^#?Port\s'
        line: 'Port 2222'
        state: present

    - name: Disable Root SSH Login
      ansible.builtin.lineinfile:
        path: /etc/ssh/sshd_config
        regexp: '^#?PermitRootLogin\s'
        line: 'PermitRootLogin no'
        state: present

    # 6. FIX SYSTEMD SSH SOCKET
    - name: Create systemd override directory for SSH socket
      ansible.builtin.file:
        path: /etc/systemd/system/ssh.socket.d
        state: directory
        mode: '0755'

    - name: Write dual-stack IPv4/IPv6 socket configuration
      ansible.builtin.copy:
        dest: /etc/systemd/system/ssh.socket.d/listen.conf
        mode: '0644'
        content: |
          [Socket]
          ListenStream=
          ListenStream=0.0.0.0:2222
          ListenStream=[::]:2222

    - name: Reload systemd daemon
      ansible.builtin.systemd_service:
        daemon_reload: true

    - name: Restart SSH socket
      ansible.builtin.service:
        name: ssh.socket
        state: restarted

    - name: Restart SSH service
      ansible.builtin.service:
        name: ssh
        state: restarted

    # 7. DISPLAY CREDENTIALS
    - name: Display your new credentials (SAVE THESE IMMEDIATELY)
      ansible.builtin.debug:
        msg:
          - '========================================================'
          - 'NEW SUDO USERNAME: {{ custom_admin_user }}'
          - 'NEW PASSWORD: {{ new_admin_password }}'
          - 'CUSTOM SSH PORT: 2222'
          - '========================================================'

⚠️ Save the displayed credentials immediately — the generated password is only shown once.

Playbook 2 — add-vps-user.yml

Creates a new sudo user with a randomly generated secure password and prints the credentials to your screen.

---
- name: Universal Semi-Automated User Creation Script
  hosts: vpsDemo
  gather_facts: false
  become: true
  vars_prompt:
    - name: 'custom_admin_user'
      prompt: 'Enter the custom username for this new administrator account'
      private: false
  tasks:
    - name: Generate random secure password
      set_fact:
        new_random_password: "{{ lookup('password', '/dev/null chars=ascii_letters,digits length=16') }}"

    - name: Ensure the new user account exists
      ansible.builtin.user:
        name: '{{ custom_admin_user }}'
        password: "{{ new_random_password | password_hash('sha512') }}"
        shell: /bin/bash
        state: present
        groups: sudo
        append: true

    - name: Allow the new user to use sudo without a password prompt
      ansible.builtin.copy:
        content: "{{ custom_admin_user }} ALL=(ALL) NOPASSWD:ALL\n"
        dest: '/etc/sudoers.d/{{ custom_admin_user }}'
        mode: '0440'
        validate: /usr/sbin/visudo -cf %s

    - name: Display New User Credentials
      ansible.builtin.debug:
        msg:
          - '========================================================'
          - 'NEW ADMINISTRATIVE ACCOUNT CREATED SUCCESSFULLY!'
          - 'USERNAME: {{ custom_admin_user }}'
          - 'PASSWORD: {{ new_random_password }}'
          - '========================================================'

Playbook 3 — remove-vps-user.yml

Completely purges a user account, their home directory, mail spool, and sudo privileges from the server.

---
- name: Universal User and Privilege Removal Script
  hosts: all
  gather_facts: false
  become: true
  vars_prompt:
    - name: 'user_to_delete'
      prompt: 'Enter the exact username you want to COMPLETELY delete'
      private: false
  tasks:
    - name: Delete the user's custom sudoers configuration file
      ansible.builtin.file:
        path: '/etc/sudoers.d/{{ user_to_delete }}'
        state: absent

    - name: Remove the user account and purge their files
      ansible.builtin.user:
        name: '{{ user_to_delete }}'
        state: absent
        remove: true   # deletes home directory and mail spool
        force: true    # kills any active processes owned by the user
        ignore_errors: true

    - name: Display Removal Confirmation
      ansible.builtin.debug:
        msg:
          - '========================================================'
          - "SUCCESS: Account '{{ user_to_delete }}' and their sudo privileges"
          - 'have been completely purged from the server.'
          - '========================================================'

Conclusion

I hope these playbooks are useful — grab them, adapt them to your environment and save yourself hours of repetitive manual work. I'll be sharing more playbooks as I build them out.

Blessings. 🙏

Looking for developer templates built on TanStack, Directus, and Tailwind CSS?
🛒 northernrangedigital.lemonsqueezy.com
🌐 northernrangedigital.com

Ansible Installation and Configuration On Ubuntu

Wed, 08 Jul 2026 00:14:52 +0000

Introduction

What is Ansible?

Ansible is an automation language that can describe any IT environment, whether home-lab or large-scale infrastructure. It is easy to learn, beautiful code that reads like clear documentation.

There are many repetitive tasks that we may perform on a server, a VM, or a VPS. The tasks can range from setting up SSH keys to disabling the root user to firewall configs. If you manage multiple servers daily and you have to do the same configuration over and over, Ansible can automate the process, allowing you to increase your productivity.

It requires installing Ansible on the Control Node and just having Python 3 installed on the Managed Node.

What is the Control Node?

It's simply the system that Ansible is installed on that controls the remote machines.

What is the Managed Node?

A remote system, or host, that Ansible controls. Ansible is agent-less, so you don’t need to install Ansible on the managed node; you simply need Python 3 to be installed.

Install Ansible on Ubuntu (Control Node—local machine. Running ZorinOS).

your-command-prompt:~$ sudo apt update

your-command-prompt:~$ sudo apt install software-properties-common

your-command-prompt:~$ sudo add-apt-repository --yes --update ppa:ansible/ansible

your-command-prompt:~$ sudo apt install ansible

Note: Ensure that Python 3 is installed on your remote server. At this time of writing, Ubuntu 26.04 LTS has Python 3.14.4 installed by default. This is enough to get your remote server running with Ansible.

Create an inventory folder with a host file.

This file contains a list of the remote machines, VMs, or VPSs that you want to control using Ansible.

For this step you can use a code editor; I will be using Vs Code.

Create a folder called Ansible and open it in vs code
In that folder, create another folder and name it inventory
Inside the inventory folder, create a file named hosts with no extension

hosts file

Start with an attribute; give it whatever name you want.
[servers]

Then list the servers that you want to manage using their DNS names or their IP addresses.
[servers]
vpsServer ansible_host=10.10.100.45
work-ToRule
10.10.45.62

We can give our hosts an alias by passing in their IP address to any name we wish to give it. In the list above, I have given the VPS with IP address 10.10.100.45 an alias of vpsServer. (IP address, just an example)

Create a folder named Playbook

In this folder we will create the actual yml files that we will use to automate the different processes we want to run on our remote servers. It can be an update, firewall config or install a package.

Our first playbook will update the Ubuntu server and set the right timezone. I will name it apt-update.yml.

Examine the YML file.

- hosts: "*" — this means all remote hosts will be updated. A single host can be selected by either its alias, DNA name, or IP address.

become: true—gives Ansible the administrative authority to act as a sudo user.

serial: 1 —In Ansible, serial: 1 tells the automation engine to manage your servers one by one rather than updating all of them at the same time.

tasks —In Ansible, the tasks section is structured as a list of individual actions. Each task is a YAML block that breaks down into four main parts: the Identity, the Module, the Arguments, and Directives.

- name: This is a plain-text description of what the task does and always starts with a dash or hyphen.

The Collection (community.general): An Ansible Collection is a standardized packaging format used to bundle and distribute Ansible content, including playbooks, roles, modules, and plugins.

The Module (timezone): This is the specialized tool or command line plugin you want to use. It tells Ansible how to execute the action on the server. Instead of writing raw terminal commands, you use abstract modules like apt (for packages), copy (for files), user (for accounts), or service (for daemons).

The Parameters (name: America/Port_of_Spain): These are the specific settings, options, or instructions you pass into the module. It defines the exact parameters of your action. The name: parameter is a mandatory argument required by the community.general.timezone module. It specifies the exact geographical timezone identifier you want to apply to the target Linux operating system.

If you're new to Ansible, you can find a collection of the modules and arguments available @ https://docs.ansible.com/projects/ansible/latest/collections/index.html

Run Playbook

Connect to the remote host in your terminal.

your-command-prompt:~$ ANSIBLE_HOST_KEY_CHECKING=FALSE ansible -i ./inventory/hosts vpsServer -m ping --user root --ask-pass

ANSIBLE_HOST_KEY_CHECKING=FALSE
This is a temporary environment variable that changes how Ansible handles security signatures. It tells Ansible to skip looking inside your local ~/.ssh/known_hosts file. It allows you to connect to a fresh or reinstalled VPS instantly without throwing the strict “Host key verification failed” fingerprint error.

ansible
This invokes the ad-hoc command line engine (used for running single, quick tasks instead of running an entire playbook file)

-i ./inventory/hosts
The -i flag stands for Inventory. It tells Ansible exactly where your infrastructure map file is saved. In this case, you are instructing it to bypass any default system paths and read the specific file named hosts located inside your ./inventory/ folder

vpsServer
This is the specific destination target. Ansible looks into your inventory to find a server or group named exactly vpsServer to run the task against.

-m ping
The module flag. It tells Ansible to execute the built-in ping software module. This is a diagnostic module that logs into the server, checks if Python is available, and returns a successful green"pong"if the link is healthy.

Once you get a green ‘pong’ you can run the next command.

your-command-prompt:~$ ansible-playbook ./playbooks/apt-update.yml --user root -e “ansible_port=22" --ask-pass --ask-become-pass -i ./inventory/hosts

ansible-playbook
This invokes Ansible's playbook execution engine. Unlike the simple ansible command (which runs just one quick ad-hoc task), ansible-playbook is designed to open an automation script file and run a whole sequence of structured tasks in order.

./playbooks/apt-update.yml
This is the relative file path to the exact automation script you want to run. : You are telling Ansible to open the file named apt-update.yml inside your playbooks folder and execute whatever configuration steps (like package updates or installations) are written inside it.

--user root
This asks Ansible to establish the initial SSH network connection using the root account, completely ignoring any other default usernames written inside your configuration files.

-e “ansible_port=22”
The -e flag stands for Extra Variables. This dynamically injects a temporary variable into the execution. Here, you are explicitly forcing Ansible to connect over standard Port 22, overriding any custom ports (like 2222) saved in your host files.

--ask-pass
Tells Ansible that you are using password authentication instead of an SSH key token. It forces the terminal to prompt you for the root SSH login password when you launch the command.

--ask-become-pass
This tells Ansible to prompt you for the privilege escalation (sudo) password.

Note: Since you are logging in directly as the absolute system master (root),--ask-become-pass is actually redundant here because root already has absolute power and never needs to escalate further.

-i ./inventory/hosts
The -i flag maps your Inventory. It tells the playbook engine exactly where to look to find your list of servers. You are directing it to read the specific hosts file located inside your ./inventory/ folder.

Summary of the Execution Flow

When you execute this command from your Control Node terminal, Ansible handles it in this sequence:

It opens your./inventory/hostsfile to look up the IP address of the target server.
It stops and asks you to type the SSH password (due to –ask-pass).
It stops and asks you to type the Sudo password (due to –ask-become-pass).
It bypasses port variables, connects directly to Port 22 as root, opens up your apt-update.yml playbook, and begins executing its tasks.

Conclusion

I want to give a shout-out to Aldo @https://dev.to/aldo_cve. He recommended Ansible in a previous post I published, and I looked it up.

I found Ansible to be a great addition to my workflow in setting up my backend. I hope you guys enjoyed this walk through. You can look forward to more posts where I publish some of the playbooks that I find useful.

From Zero to Infra: Building a Production-Ready Setup Using Our API &amp;amp; Ansible

Thu, 13 Nov 2025 09:47:35 +0000

The Story Begins

One common pattern we’ve observed at Nine is that many of our customers run web applications — think content management systems, web shops, dashboards, or custom APIs. These apps often rely on a tried-and-true combination of infrastructure services: a database to persist data, a cache layer for performance, and a server to run the application logic.

With that in mind, the setup we chose here reflects that real-world use case. We’re not just launching a VM or showing off an API (ok, maybe a little). But we’re also simulating the foundational components of what you’d find in production:

A PostgreSQL database to store orders, users, and product data
A Redis key-value store to cache hot data and reduce load times
A CloudServer VM to run the actual application (e.g. web server,
backend API, etc.)

The goal is to show how these can be provisioned and managed easily, using our API and tools, in a way that mirrors your own workload and challenges.

As the Head of Engineering at Nine, I’ve witnessed our platform evolve, providing developers with powerful tools to automate infrastructure. One of the most potent tools is our public API, giving you the ability to provision and orchestrate your infrastructure programmatically.

This blog post is not just a tutorial; it’s a story about turning a blank slate into a working, production-like environment. We will build and validate the provisioning of the following infrastructure components:

A PostgreSQL database for persistent storage
A Redis key-value store for high-speed caching
A CloudServer VM that could host any application you choose

Each step is written not just to instruct but to explain, giving context to every decision and command. This post assumes you’re a developer or DevOps engineer who may not yet be familiar with Nine’s API, Ansible, or our CLI tool nctl. We’ll guide you from first principles.

Why Ansible?

Before we get into the technical weeds, it’s worth discussing the tool we’ll use to glue everything together: Ansible.

Ansible is an automation engine used for configuration management, application deployment, and general infrastructure orchestration. What makes it so popular?

Agentless: Unlike some other tools, you don’t need to install anything on your remote servers. Ansible connects over SSH.
Human-readable YAML syntax: This makes it easy to learn and collaborate on with others.
Dynamic control flow: You can incorporate conditions, variables, loops, and even execute raw shell commands or API calls.

We chose Ansible because it’s great for scenarios like ours: where we want to orchestrate not just the creation of infrastructure, but also follow-up steps like configuration and validation.

You may have heard of OpenTofu or Terraform – these tools excel at declaratively managing infrastructure states. In fact, since we offer a public API, Nine is also fully compatible with Terraform. You can find example configurations in our Terraform examples repository. Ansible, in contrast, is more procedural and event-driven, which suits this use case well where we need more step-by-step control and conditional logic.

Setting Up Your Toolkit (macOS)

Our journey begins on a MacBook. We’ll install all the tools we need using Homebrew, the de facto package manager for macOS.

If you’re on Linux, you can follow the Ansible installation guide for Linux and install nctl manually from our GitHub releases.

For Windows users, we recommend using WSL2 (Windows Subsystem for Linux) and then following the Linux instructions above. This allows you to use the same Unix-like tooling and commands without leaving Windows.

Install Homebrew

If you don’t have Homebrew yet, install it with:

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

Homebrew lets us easily install other dependencies.

Install Ansible

brew install ansible

Ansible is our primary tool to automate the provisioning, configuration, and validation of infrastructure resources.

Install Nine’s CLI tool: nctl

brew tap ninech/nine
brew install nctl

nctl is Nine’s command-line client. It wraps around our public API and simplifies tasks like creating a VM or database, retrieving secrets, and managing access control. We’ll use it heavily.

Authenticate with Nine’s API

To interact with our API, you’ll need a Bearer token. This token grants authenticated access and is required for both Ansible and nctl.

You can create a service account and retrieve your token either through Cockpit:

https://cockpit.nine.ch/en/customer/api_service_accounts/

Or via the CLI:

nctl auth login

We create an apiserviceaccount named cicd:

nctl create apiserviceaccount cicd

And we retrieve the token for the apiserviceaccount named cicd:

nctl get apiserviceaccounts cicd --print-token

Once you have your token, export it as an environment variable:

export NINE_API_TOKEN="<your-token>"

And don’t forget to export your SSH public key too. We’ll need it to access the VM later:

export SSH_PUBLIC_KEY="$(cat ~/.ssh/id_ed25519.pub)"

You should always treat your API token like a password. Don’t hardcode it in files.

Project Structure

To keep things tidy, let’s create a project directory and define our reusable configuration.

mkdir nine-infra && cd nine-infra

Group Variables

Ansible allows us to keep variables in one place using group vars. Create the folder with:

mkdir group_vars

Here’s what group_vars/all.yml could look like:

# group_vars/all.yml
postgres_name: example-postgres
postgres_version: 15
redis_name: example-redis
vm_name: example-vm
location: nine-cz41
machine_type: nine-small-1
os: ubuntu24.04

This makes our playbooks more readable and makes it easier to reuse or update settings. You can of course adjust the values here — such as the OS version, machine type, or service versions — depending on your needs. To discover which values are supported, you can use the documentation we provide for our API:

Cloud VM machine types and locations
PostgreSQL supported versions and types
KeyValueStore specs

Step 1: Provision PostgreSQL

To provision a managed PostgreSQL instance, we’ll use nctl within an Ansible playbook. This ensures the database is consistently created with our desired configuration.

Create the following file and save it as provision_postgres.yml:

# provision_postgres.yml
- name: Provision PostgreSQL
  hosts: localhost
  gather_facts: false
 
  vars_files:
    - group_vars/all.yml
 
  tasks:
    - name: Check if PostgreSQL already exists
      ansible.builtin.shell: "nctl get postgres {{ postgres_name }}"
      changed_when: false
      failed_when: false
      register: pg_exists
 
    - name: Create PostgreSQL
      ansible.builtin.command: >
        nctl create postgres {{ postgres_name }}
        --postgres-version {{ postgres_version }}
        --machine-type nine-db-xs
        --location {{ location }}
        --allowed-cidrs 0.0.0.0/0
        --wait
      changed_when: "pg_result.rc == 0"
      ignore_errors: true
      when: "pg_exists.rc != 0"
      register: pg_result
 
    - name: Get PostgreSQL connection string
      ansible.builtin.command: "nctl get postgres {{ postgres_name }} --print-connection-string"
      register: pg_conn_str
      changed_when: "pg_conn_str.rc != 0"
 
    - name: Set Postgres connection string fact
      ansible.builtin.set_fact:
        postgres_conn_string: "{{ pg_conn_str.stdout }}/postgres"
 
    - name: Debug Postgres connection string
      ansible.builtin.debug:
        var: postgres_conn_string

Run it using:

ansible-playbook provision_postgres.yml

Be aware: This step may take some time to complete.

First up is our database. A PostgreSQL instance gives us a durable, reliable data store. It’s a common backend choice for modern apps.

We use nctl to create the instance, and then immediately retrieve its connection string so we can use it later.

Validation:

Use the connection string to confirm the instance is reachable:

psql <postgres_connection_string>

If the connection succeeds, you’re ready to move on. Keep in mind that the creation of a DB can take some time since we always provision a dedicated database. Soon we will offer Shared DBs as well which will speed up that scenario by a lot. Also please be aware that we – for simplicity reasons of this blog post – did allow 0.0.0.0/0 to access all our services. I do recommend you to not set up your infrastructure in this way.

Step 2: Provision Redis

Redis will act as our key-value store. Just like PostgreSQL, we’ll use an Ansible playbook to automate its provisioning.

Create the following file as provision_redis.yml:

# provision_redis.yml
- name: Provision Redis
  hosts: localhost
  gather_facts: false
 
  vars_files:
    - group_vars/all.yml
 
  tasks:
    - name: Check if Redis already exists
      ansible.builtin.shell: "nctl get keyvaluestore {{ redis_name }}"
      changed_when: false
      failed_when: false
      register: redis_exists
 
    - name: Create Redis
      ansible.builtin.command: >
        nctl create keyvaluestore {{ redis_name }}
        --memory-size 1Gi
        --location {{ location }}
        --allowed-cidrs 0.0.0.0/0
        --wait
      changed_when: "redis_result.rc == 0"
      ignore_errors: true
      when: "redis_exists.rc != 0"
      register: redis_result
 
    - name: Get Redis connection info
      ansible.builtin.command: nctl get keyvaluestore {{ redis_name }} -o yaml
      register: redis_info
      changed_when: "redis_info.rc != 0"
 
    - name: Parse Redis connection info
      ansible.builtin.set_fact:
        redis_host: "{{ redis_info.stdout | from_yaml | json_query('status.atProvider.connection.address') }}"
        redis_password: "{{ redis_info.stdout | from_yaml | json_query('status.atProvider.connection.password') }}"
 
    - name: Debug Redis details
      ansible.builtin.debug:
        msg: "Redis host: {{ redis_host }}, password: {{ redis_password }}"

Run it using:

ansible-playbook provision_redis.yml

Be aware: This step may take some time to complete.

Next, we spin up a Redis key-value store. Redis is often used for caching, session storage, rate limiting, or pub/sub. It’s fast and memory-based.

We again use nctl to create it, then extract the address and password from the YAML output.

Validation:

Try connecting to Redis using the CLI:

redis-cli -h <redis_host> -a <redis_password>

If the response is PONG, you’re good to go.

Step 3: Provision Cloud VM

With the database services in place, let’s bring up the virtual machine that will act as our compute node. This VM can later be configured to host applications or scripts.

Save the following playbook as provision_vm.yml:

# provision_vm.yml
- name: Provision Cloud VM
  hosts: localhost
  gather_facts: false
 
  vars_files:
    - group_vars/all.yml
 
  tasks:
    - name: Check if VM already exists
      ansible.builtin.shell: "nctl get cloudvirtualmachine {{ vm_name }}"
      changed_when: false
      failed_when: false
      register: vm_exists
 
    - name: Create VM using nctl (wait until ready)
      ansible.builtin.command: >
        nctl create cloudvirtualmachine {{ vm_name }}
        --hostname {{ vm_name }}
        --location {{ location }}
        --machine-type {{ machine_type }}
        --os {{ os }}
        --public-keys "{{ lookup('env', 'SSH_PUBLIC_KEY') }}"
        --wait
      changed_when: "vm_create_output.rc == 0"
      ignore_errors: true
      when: "vm_exists.rc != 0"
      register: vm_create_output
 
    - name: Show VM creation output
      ansible.builtin.debug:
        var: vm_create_output.stdout_lines
      when: "vm_exists.rc != 0 and vm_create_output.rc == 0"

Then apply it with:

ansible-playbook provision_vm.yml

Be aware: This step may take some time to complete.

Now let’s bring up the virtual machine. This will be our compute layer — where your app (or anything else) can run.

We’ll use nctl again, this time to provision a VM with your preferred OS image and SSH key.

Validation:

ssh root@<vm_ip>

This confirms that the server is alive, reachable, and ready for configuration.

Putting it all together

Wrapping Up

We’ve now laid the groundwork for a complete cloud-native deployment:

A secure, managed PostgreSQL instance
A fast Redis service ready for caching or queues
A fully provisioned and reachable virtual machine

All provisioned step-by-step via Ansible and Nine’s CLI and API — no manual clicking, no guessing.

From here, you can begin configuring your app, deploying code, and scaling out your architecture – all documented in code and easily accessible.

Written by Tim Grethler, Head of Engineering at Nine Internet Solutions AG.

Exploring Ansible Adhoc Commands For Beginners

Sat, 15 Feb 2025 08:16:00 +0000

Ansible is a powerful automation tool that simplifies IT tasks such as configuration management, application deployment, and task automation. One of its essential features is the ability to run ad hoc commands. In this blog post, we'll delve into Ansible ad hoc commands, their syntax, and various use cases.

What Are Ansible Adhoc Commands?

Ansible ad hoc commands are single-line commands used to perform quick tasks without writing a full Ansible playbook. They are convenient for executing simple operations on remote hosts immediately. This makes them ideal for tasks like checking system information, managing files, and installing packages.

Syntax of Ansible Adhoc Commands

The basic syntax of an Ansible ad hoc command is as follows:

ansible [host-pattern] -m [module] -a "[module options]"

where,

ansible: The command to run Ansible.

[host-pattern]: Specifies the target hosts or groups of hosts.

-m [module]: Specifies the Ansible module to use.

-a "[module options]": Specifies the arguments or options for the module.

Example:
To ping all hosts in the inventory:

ansible all -m ping

Common Use Cases of Ansible Adhoc Commands

Checking Connectivity

To check the connectivity between the Ansible control node and the remote hosts, you can use the ping module:

ansible all -m ping

This command will return a "pong" response from each reachable host.

Gathering System Information

The setup module can be used to gather detailed system information from the remote hosts:

ansible all -m setup

This command returns a JSON output containing various facts about the target systems.

Managing Files

To create a directory on remote hosts using the file module:

ansible all -m file -a "path=/tmp/test_dir state=directory"

This command creates a directory named test_dir in the /tmp directory on all hosts.

Installing Packages

The yum module (for RHEL-based systems) can be used to install packages:

ansible all -m yum -a "name=httpd state=present"

This command installs the httpd package on all hosts.

Running Commands

To execute a shell command on remote hosts, you can use the command module:

ansible all -m command -a "uname -a"
ansible all -m shell -a "uname -a"

This command retrieves the kernel version and system information from all hosts.

Copying Files

To copy a file from the Ansible control node to the remote hosts, you can use the copy module:

ansible all -m copy -a "src=/path/to/source dest=/path/to/destination"

This command copies the specified file to the target destination on all hosts.

Fetch Files

To transfer files from the remote hosts to the Ansible control node, you can use fetch module:

ansible all -m fetch -a "src=/path/to/source dest=/path/to/destination flat=yes"

This command will get files from remote machine to local machine.

Tip: Use Ansible ad hoc commands for quick, one-time tasks instead of writing full playbooks. This is especially useful for troubleshooting, system checks, and immediate configuration changes.

Caution: Be cautious when running ad hoc commands on multiple hosts, as changes take effect immediately without confirmation. Always test commands on a single host or in a controlled environment before executing them on a large scale.

Note: While ad hoc commands are powerful, they do not maintain state. For long-term configuration management, use Ansible playbooks to ensure consistency across your infrastructure.

Conclusion

Ansible ad hoc commands are a powerful tool for performing quick, on-the-fly tasks. Their simplicity and flexibility make them an essential part of any Ansible user's toolkit. Whether you're checking connectivity, gathering system information, managing files, or installing packages, ad hoc commands provide a straightforward way to get the job done. Start experimenting with them today to see how they can streamline your IT operations!

Ansible in Automation, Configuration management and Devops

Mon, 03 Feb 2025 08:42:07 +0000

What is Ansible?

In the fast-paced world of software development, automation and efficient workflows are key to success. Enter Ansible, an open-source tool that has transformed the way we approach automation, configuration management, continuous deployment, and overall DevOps practices.

Ansible is an IT automation tool that allows you to manage systems, deploy software, and orchestrate more advanced IT tasks. Its simplicity and powerful features have made it a favorite among developers and operations teams.

How Ansible Works

Unlike other automation tools that require agents to be installed on managed systems, Ansible is agentless. It connects to remote machines using SSH (Secure Shell) and executes tasks using predefined modules. These tasks are written in YAML-based playbooks, making it easy to define and manage automation workflows.

Ansible operates from a control node, which sends commands to managed nodes, ensuring that configurations and deployments are consistent across all systems.

The Benefits of Ansible

- Simple Automation: Ansible uses a straightforward YAML syntax, making
it easy to write automation scripts, also known as playbooks. This
simplicity reduces the learning curve and allows teams to get started
quickly.

- Efficient Configuration Management With Ansible: You can ensure that
your systems are configured consistently. By defining the desired
state of your infrastructure in playbooks, Ansible automatically
adjusts any discrepancies, maintaining a uniform environment.

- Continuous Deployment: Ansible streamlines the deployment process,
allowing you to automate the entire lifecycle of your
applications.From code integration to testing and deployment, Ansible
ensures that updates are delivered smoothly and reliably.

- Enhanced DevOps Practices: Ansible fosters collaboration between
development and operations teams. By using a single tool for both
automation and configuration management, it bridges the gap and
improves communication, leading to faster and more efficient
workflows.

Real-World Applications

- Infrastructure as Code (IaC): Ansible allows you to define and manage your infrastructure as code. This means you can version-control your infrastructure alongside your application code, making it easier to reproduce and manage environments.

- Application Deployment: Deploying applications manually can be time-consuming and error-prone. Ansible automates the deployment process, ensuring that every deployment is consistent and reducing the risk of human error.

- Configuration Management: Maintaining consistent configurations across multiple servers is challenging. Ansible simplifies this by allowing you to define configurations once and apply them across all your systems.

- Continuous Integration and Continuous Deployment (CI/CD): Ansible integrates seamlessly with CI/CD pipelines, automating the build, test, and deployment processes. This ensures that new features and bug fixes are delivered to users quickly and reliably.

Getting Started with Ansible

To install Ansible on a Linux system, use the following command:

sudo apt update && sudo apt install ansible -y

For Red Hat-based systems:

sudo yum install ansible -y

Writing Your First Ansible Playbook

Here's an example of a simple Ansible playbook that installs Apache on a remote server:

- name: Install Apache Server
  hosts: web_servers
  become: yes
  tasks:
    - name: Install Apache
      apt:
        name: apache2
        state: present

Run the playbook using:

ansible-playbook apache.yml

Tip: If you’re managing multiple environments (e.g., development, staging, production), consider using Ansible Vault to encrypt sensitive information like API keys, database passwords, and SSH credentials within your playbooks.

Caution: Be cautious when running Ansible as root (using become: yes). If a playbook contains misconfigured tasks, it may cause unintended changes or even break critical system components. Always test your playbooks in a staging environment before deploying them in production.

Note: Ansible follows an idempotent approach, meaning that running the same playbook multiple times will not cause unnecessary changes. It only applies modifications if the system state differs from the defined configuration.

Conclusion

Ansible has revolutionized the way we approach automation, configuration management, continuous deployment, and DevOps practices. Its simplicity, efficiency, and power make it an invaluable tool for any development and operations team. By adopting Ansible, you can streamline your workflows, improve collaboration, and deliver software more efficiently.

Set up a streams for Apache Kafka cluster with Ansible

Wed, 11 Dec 2024 20:21:43 +0000

Based on the project Apache Kafka, streams for Apache Kafka is a powerful, innovative distributed event streaming platform. Using a fully distributed architecture, a streams for Apache Kafka cluster requires you to setup and deploy numerous services, ensure they are all properly configured and, even more importantly, that each component is functional and can communicate with each other.

For all these reasons, applying an automation solution such as Red Hat Ansible Automation Platform to deploy and manage streams for Apache Kafka makes a lot of sense, especially as Ansible now has a collection dedicated for the product.

Overview

As mentioned, a typical implementation of streams for Apache Kafka includes several components, each deployed on numerous target systems. Depending on the organization of the servers, some of those components may actually be collocated.

At the core, however, there are two main parts: Zookeepers instances, often abbreviated zk, in charge of the orchestration of the cluster, and the brokers, responsible of processing the request.

Beyond those foundational elements, a streams for Apache Kafka deployment can include other services, such as Kafka Connect, to help integrate with numerous systems. We'll first tackle the deployment of the cluster itself before discussing this extra service.

Throughout this article, we will document step-by-step how to deploy each of those components using Ansible. To be thorough and easy to follow, however, we are going to first describe the setup of the Ansible controller itself to ensure the reader has all the information necessary to properly reproduce this demonstration.

Preparing the Ansible controller

Install Ansible

In Ansible lexicon, the system executing the automation tool itself is named the "controller". The machines handled by the controller are designated under the term "targets".

To set up a controller on Red Hat Enterprise Linux 9 (RHEL 9), the very first step is to enable the Ansible Automation Platform subscription on the system:

# subscription-manager repos --enable=ansible-automation-platform-2.4-for-rhel-9-x86_64-rpms

Then, Ansible can be installed simply by using dnf:

# dnf install -y ansible-core

Configure Ansible to use Ansible automation hub

As mentioned above, to smoothly set up and manage our streams for Apache Kafka cluster, we are going to leverage an Ansible collection dedicated to the product. This extension for the automation engine is available on the Ansible automation hub. This means that, to be able to connect to this repository of content, Ansible needs to be configured to utilize it (instead or on top of Ansible Galaxy). This configuration lives in the ansible.cfg configuration file:

[defaults]
...
[inventory]
...
[galaxy]
server_list = automation_hub, galaxy
[galaxy_server.galaxy]
url=https://galaxy.ansible.com/
[galaxy_server.automation_hub]
url=https://cloud.redhat.com/api/automation-hub/
auth_url=https://sso.redhat.com/auth/realms/redhat-external/protocol/openid-connect/token
token=<insert-your-token-here>

Please refer to this documentation on how to configure Ansible to Automation Hub and then execute the following command to install the Red Hat provided collection for streams for Apache Kafka.

Once you have properly configured Ansible to utilize automation hub, you can simply run the following command to install the Ansible collection for streams for Apache Kafka:

$ ansible-galaxy collection install redhat.amq_streams

Note: Execute this command as the same system user used to run the playbook later on. Indeed, another user, on the same machine, won't have access to the installed collection.

Prepare targets

For any systems managed by Ansible, SSHD must be running and a SSH pubkey, associated to root user needs to have been deployed to ensure that Ansible can connect to it. For RHEL 9 system, the required subscription for streams for Apache Kafka (and RHEL) must be enabled. Please refer to this documentation to learn the necessary steps to achieve this.

An inventory of the targets needs to be provided to Ansible. The simplest way to achieve this is using a plain inventory file, such as this one:

[all]

[brokers]
broker*.example.com

[zookeepers]
zk*.example.com

[topic_writer]
broker1.example.com

For testing purpose, you can use localhost, which also removes the need to set up SSHD (as there will be no SSH connection between Ansible and localhost):

[all]
localhost ansible_connection=local

[brokers]
localhost ansible_connection=local

[zookeepers]
localhost ansible_connection=local

[topic_writer]
localhost ansible_connection=local

Each of the components of the streams for Apache Kafka cluster used different ports, so they can be all deployed on one single system. Obviously, this is not recommended for production. It allows you, however, to run the playbooks provided in this article with only one machine (physical, virtual, or containerized).

Note: Execute twice every playbook in this demonstration to verify that they are indeed idempotent (meaning, the second time does not perform any change).

Automating the deployment of the Zookeepers instances

Now that we have a working Ansible controller and an assortment of target machines ready to be provisioned, we can start the setup of our cluster. The first step will be, obviously, to deploy the zk instances. Indeed, they are being orchestrating the cluster, and having them already running before the brokers make sense.

Thanks to the Ansible collection for streams for Apache Kafka, the playbook to set up the zk nodes is reduced to the bare minimum:

---
- name: "Ansible Playbook to install a Zookeeper ensemble with auth"
  hosts: zookeepers
  gather_facts: yes
  vars_files:
    - vars.yml
  collections:
    - redhat.amq_streams
  roles:
    - role: redhat.amq_streams.amq_streams_zookeeper

  post_tasks:
    - name: "Validate that Zookeeper deployment is functional."
      ansible.builtin.include_role:
        name: redhat.amq_streams.amq_streams_zookeeper
        tasks_from: validate.yml

Note that the host's value matches the name of the inventory's group associated with the machines responsible for running Zookeeper.

In essence, the playbook above just use the role redhat.amq_streams.amq_streams_zookeeper provided by the collection. In the post_tasks: segment, it utilizes as well that validate.yml, a handy playbook also supplied by the collection, to check that the zk instances deployed are indeed functional.

The specific configuration of the Zookeepers node is provided by the variables in the vars.yml:

---
amq_streams_common_download_dir: "/opt"

In our demonstration here, we chose to deploy the binaries and files of streams for Apache Kafka in the /opt folder, on the target system. As this particular configuration will also be used for the broker (see below), it has been placed into a separate vars.yml file. This way, this configuration can be shared with the playbook managing the broker instance, as we'll cover in the next section.

You can now run the playbooks to deploy zk on the targets belonging to the 'zookeepers' group:

$ ansible-playbook -i inventory -e @service_account.yml zookeepers.yml

Note: KRaft can be employed as an alternative to Zookeepers for orchestration, however the Ansible collection for streams for Apache Kafka does not yet supports deployment using KRaft.

Automating the deployment of the brokers

Now that the zk nodes are ready to orchestrate our cluster, we can look into the deployment of the broker’s nodes. Here again, the Ansible collection for streams for Apache Kafka is doing the heavy lifting:

---
- name: "Ansible Playbook to install Kafka Brokers with authentication"
  hosts: brokers
  vars_files:
    - vars.yml
  vars:
    amq_streams_replication_factor: "{{ groups['brokers'] | length }}"
    amq_streams_broker_offsets_topic_replication_factor: "{{ amq_streams_replication_factor }}"
    amq_streams_broker_transaction_state_log_replication_factor: "{{ amq_streams_replication_factor }}"
    amq_streams_broker_transaction_state_log_min_isr: "{{ amq_streams_replication_factor }}"
  collections:
    - redhat.amq_streams
  roles:
    - name: amq_streams_broker
  post_tasks:
    - name: "Validate that Broker deployment is functional."
      ansible.builtin.include_role:
        name: amq_streams_broker
        tasks_from: validate.yml

Similar to the playbook above, we use the role provided by the streams for Apache Kafka collection in order to deploy the broker. As mentioned in the previous section, we also include the vars.yml to utilize the configuration shared between the zk instances and brokers.

A few variables were also added in order to configure the broker instances. In the example above, we used the number of instances to determine the value of the replication factor.

We can now run this playbook in order to set up our cluster:

$ ansible-playbook -i inventory -e @service_account.yml brokers.yml

Creating topics in the cluster

With our zk and brokers nodes completely provisioned, our cluster is now ready. For applications to be able to consume or publish events (or messages), topics must be created.

Here again, these changes to the setup are fully automated by a third playbook, also leveraging the capabilities of the Ansible collection for streams for Apache Kafka:

---
- name: "Ansible Playbook to ensure topics are created in Red Hat AMQ Streams cluster"
  hosts: topic_writer
  gather_facts: no
  vars_files:
    - vars.yml
  vars:
    amq_streams_broker_topic_to_create_name: "myTopic"
    amq_streams_broker_topics:
      - name: "{{ amq_streams_broker_topic_to_create_name }}"
        partitions: 1
        replication_factor: "{{ groups['brokers'] | length }}"
  collections:
    - middleware_automation.amq_streams

  tasks:
    - name: "Create topic: {{ amq_streams_broker_topic_to_create_name }}"
      ansible.builtin.include_role:
        name: amq_streams_broker
        tasks_from: topic/create.yml
      loop: "{{ amq_streams_broker_topics }}"
      loop_control:
        loop_var: topic
      vars:
        topic_name: "{{ topic.name }}"
        topic_partitions: "{{ topic.partitions }}"
        topic_replication_factor: "{{ topic.replication_factor }}"

  - name: "Describe topic: {{ amq_streams_broker_topic_to_create_name }}"
    ansible.builtin.include_role:
      name: amq_streams_broker
      tasks_from: topic/describe.yml
    loop: "{{ amq_streams_broker_topics }}"
    loop_control:
      loop_var: topic
    vars:
      topic_name: "{{ topic.name }}"

The target host for this playbook is all the members of the group topic_writer. As shown on the manifest above, this group only contains one of the brokers. Indeed, to create the topic for the entire cluster, any broker can be utilized and there is no need to repeat the operation on all the nodes.

Here again, the vars.yml is used in this playbook to give access the shared variables between all of them. In our demonstration, the folder sheltering the streams for Apache Kafka deployment on the target is the only common parameters.

Note: With a real life use case, there would be the most likely more common configuration between the components, however, for the sake of simplicity, our example only have this parameter shared.

While we only create one topic, we used a list and the associated for loop to demonstrate how this playbook can be used to ensure the availability of as many topics as needed.

Let's run this playbook to create our topic inside the cluster. We don't need to pass our service account credentials anymore, as this playbook mustn't perform any product installation and thus does not require to access the Red Hat Customer portal to retrieve the necessary zips:

$ ansible-playbook -i inventory topics.yml -e amq_streams_common_download_dir=/opt

Note that we need to specify where streams for Apache Kafka is located on the target system, because this playbook has not performed the installation; thus it does not know where the streams for Apache Kafka binaries and config files lives. If the topics deployment were part of the same playbook as the setup, this would not be necessary.

Also note that this playbook, like the previous one, is idempotent: if we run it again, no change occurs on the target system as the required topics are already created inside the cluster:

$ ansible-playbook -i inventory topics.yml

Automating the deployment of Kafka Connect

With the cluster now fully functional, we can study an additional component of the rich ecosystem of Red Hat AMQ: Kafka Connect. This supplemental service will allow the cluster to interact with numerous external systems, typically a datasource of some kind, like traditional RDMS or in-memory cache. In essence, it helps data's conversion from a remote service into messages that can be processed by the cluster. Or, the other way around, transforming messages from the cluster into records persisted or manipulated by an external system.

Here again, setting up this service on the target system is made extremely easy by the Ansible collection for streams for Apache Kafka:

---
- name: "Ansible Playbook to install a Kafka Connect ensemble with auth"
  hosts: connect
  gather_facts: yes
  vars_files:
    - vars.yml
  collections:
    - redhat.amq_streams
  tasks:
    - name: "Ensure Kafka Connect is running"
      ansible.builtin.include_role:
        name: redhat.amq_streams.amq_streams_connect
  vars:
    connectors:
      - { name: "file", path: "connectors/file.yml" }
      
  post_tasks:
    - name: "Validate that Kafka Connect deployment is functional."
      ansible.builtin.include_role:
        name: redhat.amq_streams.amq_streams_connect
        tasks_from: validate.yml

The sole configuration specific to Kafka Connect is the file connector. It is needed only because Kafka Connect can be started without any connectors. In a real-life use case, it's not an issue as, obviously, such a component would not be deployed with a set of predefined connectors matching the requirement of the applications utilizing the cluster. In our demonstration, we add this connector only for the installation to be successful:

$ ansible-playbook -i inventory connect.yml -e @service_account.yml

Note: In our demonstration we install each component on the same host, which means that the Red Hat service authentication data could be omitted in the command line above, as Kafka is already available on the target. However, in a real-life scenario, Kafka Connect will most likely run on a different system and thus require those identification information to set up the service.

Summary

Thanks to both Ansible and its collection for streams for Apache Kafka, we have deployed a fully operational and ready to be used cluster, with only a set of simple playbooks. This setup works on one machine, but could be effortlessly utilized for hundreds of systems. The only change required is to modify the inventory accordingly.

From now on, managing the cluster is also made easier by Ansible. New topics can handily be added to the configuration and deployed. Advanced features, such as authentication between components, can be implemented leveraging functionalities of the streams for Apache Kafka collection. Last but not the least, Ansible can help automate the update of the cluster. It's a difficult subject that needs to be handled according to the usage made of the system, so there is no "out of the box" solution for this, however the collection and the primitives of Ansible, provides a lot assistance to manage such critical operation.