Coder Legion - Recent posts tagged chaosengineering

Chaos Network Proxy

Sun, 21 Dec 2025 17:42:58 +0000

The Story: Why I built Chaos Proxy

We've all been there. The feature works perfectly on localhost. The E2E tests pass with flying colors. Then we deploy to production, and users on 3G networks start complaining that the app freezes, crashes, or—worst of all—charges them twice.

I realized that our CI pipelines were living in a fantasy world of 0ms latency and 100% uptime.

I wanted to simulate "Bad Network" conditions automatically in GitHub Actions, specifically for mobile apps and backend idempotency checks. I tried mocking requests in Playwright, but that didn't cover native Android/iOS emulators. I tried local proxies, but they were hard to script.

So I built Chaos Proxy —a cloud-based, programmable Chaos Proxy designed for CI/CD.

How it works

Debuggo isn't just a GUI tool. It’s an API-first platform. You can treat your network infrastructure like code.

Create: Your CI script calls our API to spin up an isolated, ephemeral proxy container.
Connect: You route your E2E test traffic (Web, Android, iOS) through this proxy.
Break: You send API commands to inject latency, trigger 503 errors, or tamper with headers in real-time.

Demo: Simulating 503 Errors in Chrome (Visual)

Key Features

API for CI/CD: Spin up and destroy proxies programmatically. No long-living servers to manage.
The "Rage Click" Test: Inject 3 seconds of latency into specific endpoints (e.g., /api/pay) to ensure your UI disables buttons correctly before the user clicks twice.
Native Mobile Support: Since it works at the network level (HTTP Proxy), it supports Android Emulators and iOS Simulators perfectly.
Response Fuzzing: Automatically tamper with JSON bodies to see if your app crashes on malformed data.

⚡️ See it in action
Here is how simple it is to inject a 503 Service Unavailable error into your checkout flow using curl:

curl -X PUT https://api.debuggo.app/v1/sessions/$SESSION_ID/rules \
  -H "Authorization: Bearer $API_KEY" \
  -d '{
    "rules": [
      {
        "url_pattern": "*/api/checkout",
        "failure_rate": 50,
        "error_code": 503
      }
    ]
  }'

Demo: Automating Network Chaos via Terminal (CLI)

How you can get involved

I just launched the Public API Beta. I am looking for QA Engineers and DevOps folks who are tired of "flaky" apps and want to build true resilience.

Try the Free Tier: You can start manually or via API for free.
Break your App: Try the "Rage Click" test (Tip #5 on our blog).
Feedback: Let me know what integration you need next!

Stop trusting localhost. Start testing reality.

Announcing Chaos Proxy API: Automate Network Chaos in CI/CD

Sun, 21 Dec 2025 14:51:08 +0000

Moving Beyond "Localhost" Testing

Until now, Debuggo has been a fantastic tool for manual testing. You spin up a proxy, connect your phone, and verify how your app handles a 503 error or high latency. It works great for ad-hoc debugging.

But manual testing doesn't scale.

You cannot ask your QA team to manually verify "Offline Mode" handling on every single Pull Request. You cannot manually check if your payment gateway handles double-clicks correctly before every deploy.

To build truly resilient apps, you need Continuous Chaos.

Today, we are launching the Chaos Proxy API. Now you can programmatically create proxies, configure chaos rules, and tear them down—all within your CI/CD pipeline (GitHub Actions, GitLab CI, Jenkins).

Architecture: How it works in CI

The API gives you full control over the lifecycle of a Chaos Proxy directly from your pipeline scripts:

Create: Spin up a fresh, isolated proxy instance on demand (POST /sessions).
Configure: Apply chaos rules (latency, errors, body tampering) via JSON (PUT /rules).
Certify: Download the CA certificate to install on Android Emulators or iOS Simulators (GET /certs).
Test: Run your E2E suite (Playwright, Appium, Cypress) routing traffic through the proxy.
Destroy: Clean up resources when the test finishes (DELETE /sessions).

Real-World Example: GitHub Actions
Here is a complete workflow. This script spins up a proxy, injects a 3-second latency to simulate a slow network, runs tests to ensure the UI handles "Rage Clicks" correctly, and then shuts everything down.

name:  Chaos E2E Tests

on: [push]

jobs:
  chaos-test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3

      # 1. Start the Proxy
      - name:  Start Debuggo Proxy
        id: start_proxy
        run: |
          RESPONSE=$(curl -s -X POST https://chaos-proxy.debuggo.app/api/v1/sessions \
            -H "Authorization: Bearer ${{ secrets.DEBUGGO_API_KEY }}")
          
          # Extract and save details to ENV
          echo "PROXY_ID=$(echo $RESPONSE | jq -r .id)" >> $GITHUB_ENV
          echo "PROXY_HOST=$(echo $RESPONSE | jq -r .host)" >> $GITHUB_ENV
          echo "PROXY_PORT=$(echo $RESPONSE | jq -r .port)" >> $GITHUB_ENV
          echo "PROXY_AUTH=$(echo $RESPONSE | jq -r .auth)" >> $GITHUB_ENV

      # 2. Configure Chaos (The "Bad 3G" Simulation)
      - name:  Configure Chaos Rules
        run: |
          curl -X PUT https://chaos-proxy.debuggo.app/api/v1/sessions/$PROXY_ID/rules \
            -H "Authorization: Bearer ${{ secrets.DEBUGGO_API_KEY }}" \
            -H "Content-Type: application/json" \
            -d '{
              "rules": [
                {
                  "url_pattern": "*/api/checkout",
                  "delay": 3000,
                  "error_code": null
                }
              ]
            }'

      # 3. Run Tests
      - name:  Run Playwright Tests
        run: |
          # Route traffic through the authenticated proxy
          export HTTPS_PROXY="http://$PROXY_AUTH@$PROXY_HOST:$PROXY_PORT"
          npx playwright test

      # 4. Cleanup (Always run this, even if tests fail)
      - name:  Cleanup
        if: always()
        run: |
          curl -X DELETE https://chaos-proxy.debuggo.app/api/v1/sessions/$PROXY_ID \
            -H "Authorization: Bearer ${{ secrets.DEBUGGO_API_KEY }}"

API Reference
Use these endpoints to integrate Chaos Proxy into your custom scripts.

Authentication
Authenticate all requests by including your API Key in the header. You can generate a key in your Dashboard Settings.

Authorization: Bearer dbg_ci_YOUR_KEY

Start Proxy Session
Creates a new isolated proxy container. Returns the host, port, and credentials.
- Endpoint: POST /api/v1/sessions
- Response:

{
  "id": "sess_abc123",
  "host": "proxy-us-east.debuggo.app",
  "port": 10245,
  "auth": "user:pass"
}

Configure Rules
Updates the chaos logic in real-time. You can change rules mid-test (e.g., test success first, then inject failure).
- Endpoint: PUT /api/v1/sessions/{session_id}/rules
- Body Example:

{
  "rules": [
    {
      "url_pattern": "*/api/v1/checkout",
      "failure_rate": 100,
      "error_code": 503,
      "delay": 0
    },
    {
      "url_pattern": "*/api/v1/search",
      "delay": 2000
    }
  ]
}

Download CA Certificate
Retrieves the Root CA certificate. Essential for automated setup of Android Emulators or iOS Simulators in CI.
- Endpoint: GET /api/v1/certs/ca.pem
- Usage:

curl -O https://chaos-proxy.debuggo.app/api/v1/certs/ca.pem
# Then install via adb

Stop Session
Stops the proxy and releases the port.
- Endpoint: DELETE /api/v1/sessions/{session_id}

Why automate Chaos?

Catch Regressions in "Unhappy Paths" Developers often break error handling logic because they rarely see errors locally. Automating a 500 Error test ensures your "Something went wrong" screen never breaks.
Validate Idempotency By injecting latency into your payment endpoints during CI, you can verify that your backend correctly handles duplicate requests (Rage Clicks) before they reach production.
Native Mobile Testing Unlike Playwright’s built-in page.route (which only works in a browser context), Debuggo works at the system level. This allows you to test Native Android and iOS apps running in emulators within your CI pipeline.

Ready to break your build (on purpose)? Get your API Key.

The "Spinner of Death": Why Localhost Latency is Lying to You

Fri, 19 Dec 2025 15:19:20 +0000

The "Localhost" Bias
We've all been there.

On your machine, the API responds in 5ms. The UI updates instantly. You click "Submit," the modal closes, and you move on to the next ticket. Status: Done. ✅

But on a user's 4G connection in a subway tunnel, that same API call takes 2 seconds.

Because you tested on localhost (Gigabit Fiber), you missed critical race conditions:

️ The Double-Click Bug: The user clicks "Submit" twice because "nothing happened," charging their credit card twice.
The Infinite Spinner: The loader gets stuck forever because a packet was dropped.
️ Race Conditions: Data arrives out of order, overwriting the user's input.

Your app feels fast because you are cheating. 0ms latency is a lie.

The Wrong Solution: time.sleep()
I often see tests that look like this:

# ❌ Don't do this
page.click("#submit")
time.sleep(2) # Simulating "network lag"
expect(page.locator(".success")).to_be_visible()

Why this fails: `sleep()` just pauses the test execution script. The browser engine itself is still blazing fast. It doesn't simulate network queues, slow handshakes, or constrained bandwidth. You aren't testing the network; you're just making your test suite slower.

The Right Solution: Network Throttling (CDP)

To test this properly in automation, you need to talk directly to the browser engine. You need to tell Chrome: "Pretend you are on a terrible 50kb/s connection."

We can do this using the Chrome DevTools Protocol (CDP) within Playwright. This forces the browser to handle packet delays and loading states exactly as a real user would experience.

The Code (Python + Playwright)
Here is how to inject a "Bad 3G" connection into your test:

from playwright.sync_api import Page, expect

def test_slow_network_handling(page: Page):
    # 1. Connect to Chrome DevTools Protocol (CDP)
    # This gives us low-level access to the browser
    client = page.context.new_cdp_session(page)
    
    # 2.  CHAOS: Emulate "Bad 3G"
    # Latency: 2000ms (2 seconds)
    # Throughput: 50kb/s (Very slow)
    client.send("Network.emulateNetworkConditions", {
        "offline": False,
        "latency": 2000, 
        "downloadThroughput": 50 * 1024,
        "uploadThroughput": 50 * 1024
    })

    page.goto("https://myapp.com/search")
    
    # 3. Trigger the slow action
    page.fill("#search-box", "Playwright")
    page.click("#search-btn")

    # 4. Resilience Assertion
    
    # Check 1: Does the UI prevent double submission?
    expect(page.locator("#search-btn")).to_be_disabled()
    
    # Check 2: Does the user get immediate feedback?
    expect(page.locator(".loading-spinner")).to_be_visible()

Why this matters: This test proves your UI provides feedback. If a user clicks a button and waits 2 seconds with no visual feedback, they will assume the app is broken.

But wait, what about Mobile Apps?
The script above is perfect for automated CI pipelines running Chrome. But CDP has a major limitation: It doesn't work on a physical iPhone or Android device.

If you are a Mobile Developer or manual QA, you can't "attach Playwright" to the phone in your hand to simulate a subway tunnel.

The Manual Alternative (System-Level Proxy)

To test latency on a real device without writing code, you need a System-Level Proxy that sits between your phone and the internet.

You can use desktop tools like Charles Proxy (if you enjoy configuring Java apps and firewalls), or you can use a cloud-based tool like Chaos Proxy (which I'm building).

It allows you to simulate "Subway Mode" (2s latency) on any device—iPhone, Android, or Laptop—just by connecting to a Wi-Fi proxy.

The Workflow:

Create a "Chaos Rule" (e.g., Latency = 2000ms).
Connect your phone to the proxy via QR code.
Watch your app struggle (and then fix it).

Summary
Stop trusting Localhost. It hides your worst bugs.
Automated: Use Playwright + CDP to inject latency in your E2E tests.
Manual/Mobile: Use a Chaos Proxy to test resilience on physical devices.

Happy testing!

If you found this useful, check out my previous post: Stop Testing Success. Kill the Database.

Don't Break Your WiFi: How to Simulate 503 Errors in a Single Browser Tab

Sun, 07 Dec 2025 07:15:00 +0000

Testing frontend error handling is a pain.

You want to verify that your React/Vue app shows a nice "Retry" button when the API fails. Or you want to check if the Skeleton Loader looks good during a 10-second delay.

The traditional way: Open Chrome DevTools, toggle "Offline" mode. The problem: It kills the connection completely. You can't simulate a slow server that returns a specific HTTP 503 error code.

The other way: Use a system-wide proxy (like Charles). The problem: It breaks your Spotify, Slack, and Zoom calls while you test.

In this guide, I'll show you how to use FoxyProxy and Debuggo Chaos Proxy to inject network failures only into a specific browser tab, leaving the rest of your machine online.

Video Tutorial (1:57)
See the full setup in action:
youtube.com/watch?si=_ItP2sPSbkGy83KU&embeds_referring_euri=https%3A%2F%2Fdev.to%2F&source_ve_path=Mjg2NjQsMTY0NTA2&v=NbPPoqCv3mY&feature=youtu.be

The Stack

FoxyProxy Standard: A browser extension (Chrome/Firefox) that routes traffic based on patterns.
Chaos Proxy Debuggo: A cloud chaos proxy that adds delays and errors.

Step 1: Configure the Chaos Rule
Let's break the internet (selectively). In the Debuggo dashboard, I set up a rule for httpbin.org:

Delay: 10 seconds (I really want to see that loading spinner).
Status: 503 Service Unavailable.
Probability: 50%.

Step 2: Set up FoxyProxy
Instead of configuring the proxy in MacOS System Settings (which affects the whole OS), we do it in the browser extension.

Click Add New Proxy.
Host: chaos-proxy.debuggo.app
Port: 11997 (Your allocated port).
Auth: Enter the username/password provided by the service.

Tip: You can set "URL Patterns" in FoxyProxy to only route traffic for your staging API (e.g., *api.staging.com*) through the chaos proxy.

Step 3: The Moment of Truth
Navigate to your target URL.

First load: You might see it load instantly. Why? Browsers are smart. They reuse existing TCP connections. Fix: Refresh the page or open a new tab to force a new connection through the proxy.

Second load:

The browser tab spins... and spins... (The 10s delay works!).
Finally, you see the 503 Error Page injected by the proxy.

Why do this?
This setup allows you to audit your UI's resilience:

Infinite Scroll: Does the app crash if the "Load More" request times out?
Form Submission: Does the "Submit" button re-enable if the server returns an error?
Error Boundaries: Does your React Error Boundary actually catch the crash?

Chaos Engineering isn't just for backend infrastructure. Frontend developers need to verify that their UI allows users to recover gracefully from network glitches. Using a browser-scoped proxy is the least intrusive way to do it.

Try creating your first rule at chaos-proxy.debuggo.app.

Stop Mocking Everything: How to Test API Resilience in Your Terminal (Curl + Chaos Proxy)

Fri, 05 Dec 2025 07:15:00 +0000

"It works on my machine."

We've all said it. But does your bash script work when the third-party API takes 15 seconds to respond? Does your backend service gracefully handle a sudden spike of 503 errors from a payment provider?

Writing a full mock server just to test a simple retry logic in a script is often overkill. In this tutorial, I'll show you a faster way: Chaos Engineering directly in the terminal.

We will use curl and a cloud-based Chaos Proxy to inject failures into real network requests without changing a single line of your application code.

The Problem: Localhost is Too Perfect
When developing locally, network latency is near zero. APIs either work (200 OK) or they don't (Connection Refused). But in production, you face:

High Latency: The server is busy.
Intermittent Failures: 5% of requests drop.
Throttling: You hit the rate limit.

Simulating this in a terminal usually involves complex iptables rules or local tools like tc (Traffic Control). There is an easier way.

The Solution: Cloud Chaos Proxy
Instead of configuring your OS, we will route specific requests through a proxy that "breaks" the traffic according to rules we define.

I'll be using chaos-proxy.debuggo.app for this, but the concept applies to any programmable proxy.

Video Guide (1:27)
Prefer watching? Here is the 90-second workflow:

Step-by-Step Tutorial

1. Define the Failure
First, we need to tell the proxy what to break.

Target: httpbin.org (or your API domain).
Delay: 7 seconds (Simulate lag).
Failure Rate: 1 (100% of requests will fail).
Error Code: 503 Service Unavailable.

2. Trust the Certificate (The "One-Time" Setup)
Since we are intercepting HTTPS traffic, we need to trust the proxy's CA certificate.

Download the mitmproxy-ca-cert.pem from the dashboard.

MacOS: Add it to Keychain Access -> System and set "Always Trust".

Linux: Copy to /usr/local/share/ca-certificates/ and update.

3. The Magic Command
Now, we use curl with the -x (proxy) flag.

curl -v -x http://user:*Emails are not allowed*:13979 https://httpbin.org/get

Understanding the Output
When you run this command multiple times:

Scenario A (The Chaos): You'll notice the terminal "hangs" for 7 seconds (our delay). Then:

< HTTP/1.1 500 Internal Server Error
< content-length: 56
< content-type: text/plain
...
Debuggo Chaos Injection: 500 Error

Scenario B (Success): The other 50% of the time, the request passes through to the real server:

< HTTP/1.1 200 OK
...
{
  "args": {}, 
  "headers": { ... }
}

Real World Use Cases

Testing CI Pipelines: Verify that your deployment scripts don't crash if a dependency is slow.
Cron Jobs: Ensure your nightly data sync retries correctly on failure.
Quick Sanity Checks: Before pushing code, verify how the API client handles a 503 error.

Conclusion
You don't need heavy infrastructure to test network resilience. A simple proxy setup allows you to inject chaos into any HTTP client—be it curl, wget, or your Python/Node.js scripts.

Happy Breaking!