Coder Legion - Recent posts tagged apify

A scraper isn’t a monitor: detecting new Telegram posts with Apify and n8n

Tue, 22 Sep 2026 02:46:31 +0000

My Actor run succeeded. It returned five recent Telegram posts, each with an ID, timestamp, and permalink.

Then I asked the question the automation actually needed to answer:

Which posts are new?

The honest answer was: the current run cannot know by itself.

Nothing had failed. The Actor had done its job. It had retrieved what was visible now. But “new” is not a property contained in one snapshot; it is the difference between the current snapshot and a previous one.

That distinction is the difference between a scraper that runs on a schedule and a monitor you can trust.

In this tutorial, I’ll show the state layer I added around an Apify Actor:

a stable identity for every record;
state partitioned by Telegram channel;
a deliberate first-run policy;
duplicate prevention across executions;
a warning when the polling window may be too small; and
fail-closed handling for missing or malformed state.

The same design works for price trackers, job monitors, lead feeds, inventory checks, and any other workflow that repeatedly asks, “What changed?”

Implementation note: the code and evidence for this article are in the corrected companion package. That workflow deliberately stops at notification candidates; I have not treated untested delivery as finished.

Retrieval and change detection are different jobs

The Actor reads Telegram’s public t.me/s/ preview and returns recent messages in newest-first order. Each item includes fields such as:

{
  "channel": "telegram",
  "id": 454,
  "url": "https://t.me/telegram/454",
  "date": "2026-07-19T17:58:20+00:00",
  "text": "For all the details on these new features...",
  "views": 1180000,
  "scraped_at": "2026-08-05T14:13:17.444080+00:00"
}

That is enough to answer:

What is visible now?

It is not enough to answer:

What appeared since the previous successful check?

The second question requires historical information. That history can live inside an Actor, in a database, in a caller-supplied cursor, or in the workflow that calls the Actor. For this implementation, I kept the Actor stateless and made the downstream n8n workflow own the comparison state.

The boundary looks like this:

Schedule
   │
   ▼
Run Apify Actor ──► current message window
                         │
                         ▼
              validate and partition records
                         │
                         ▼
              compare with durable seen state
                         │
              ┌──────────┴──────────┐
              ▼                     ▼
      notification candidates   status and warnings
              │                     │
              └──────────┬──────────┘
                         ▼
                  persist new state

The experiment that exposed the missing layer

I first called the Actor from an AI client through the Apify MCP server. I retained four one-call conditions:

Condition	What the caller knew before the run	Could it determine new IDs?
Baseline	No previous result	Not asked; the result became the baseline
Same conversation	Previous IDs were still in context	Yes
Fresh session	No reachable previous result	No; newness was undetermined
Fresh session with prior IDs	A small state file plus comparison rules	Yes

All four Actor runs succeeded and returned the same five IDs, 454 through 450. In the fresh session with no baseline, retrieval still worked, but comparison did not.

That result matters because “no new records” and “I cannot determine whether records are new” are not the same answer. A reliable monitor must preserve that distinction.

The runs also exposed a tempting identity bug: every message’s views value changed between two observations, while the IDs stayed stable. Comparing whole records would have classified all five messages as different. Comparing the stable identity classified them correctly as the same messages.

Choose identity before choosing storage

Telegram’s Bot API defines message_id as unique inside a chat. That means id alone is not a reusable global key. The monitor uses this partitioned identity instead:

(normalized channel, message ID)

For example:

function normalizeChannel(value) {
  return String(value ?? '')
    .trim()
    .replace(/^@/, '')
    .toLowerCase();
}

function messageKey(channel, id) {
  const normalizedChannel = normalizeChannel(channel);
  if (!normalizedChannel) {
    throw new Error('Channel is required.');
  }

  if (!Number.isSafeInteger(id) || id <= 0) {
    throw new Error('Message ID must be a positive safe integer.');
  }

  return JSON.stringify([normalizedChannel, String(id)]);
}

views and scraped_at are deliberately excluded. They can change without creating a new message. Text can also be edited, so an ID-only state detects new messages but does not detect edits; edit detection is a separate contract.

This is the first design rule:

Use immutable source identity for new-item detection. Do not let mutable attributes redefine the record.

Store a baseline, not a vague memory

A minimal prior-state file for one channel can be as small as:

{
  "schemaVersion": 1,
  "channel": "telegram",
  "identityField": "id",
  "sourceRunId": "example-prior-run-id",
  "seenMessageIds": [454, 453, 452, 451, 450]
}

The n8n workflow uses the same contract in a more compact form. Its global workflow state is partitioned by normalized channel, and positive integer IDs are stored as merged ranges:

{
  "schemaVersion": 1,
  "channels": {
    "telegram": {
      "initializedAt": "<timestamp from the first successful run>",
      "baselineFloorId": 450,
      "seenRanges": [[450, 454]],
      "highestObservedId": 454
    }
  }
}

Range compression is an implementation optimization, not part of the general monitoring rule. Set comparison requires stable identifiers; range compression additionally assumes suitable integer IDs.

The workflow reads this state with n8n’s $getWorkflowStaticData('global'). n8n documents three important constraints: static data is intended to be small, it is saved after successful triggered production executions, and it can be unreliable under high-frequency execution. That makes it acceptable for this small serialized demonstration, not a universal production database.

For growing state, overlapping executions, or operational querying, move the same identity and comparison contract to a Data Table or external database.

Make the first run a policy decision

On the first successful execution, every returned ID is unseen. Calling all of them “new” may be mathematically consistent, but it is usually terrible monitoring behavior: the workflow sends a burst of alerts for posts that existed before monitoring started.

This implementation defaults to baseline mode:

Fetch the first non-empty window.
Validate it.
Store the IDs.
Send no notification candidates.
Begin new-item detection on the next successful run.

An explicit alert mode remains available for workflows that really do want the first window delivered.

An empty result does not initialize the baseline. Otherwise a transient empty response could create an empty history, and the next normal result could replay the entire window as new.

Compare, classify, then persist

The core comparison is set difference:

new IDs = current IDs − previously seen IDs

But the order around that expression matters:

Validate the response shape and channel partition.
Reject unusable IDs.
Deduplicate the current window by the partitioned identity.
Compare against previously seen identities.
Apply the first-run and historical-backfill policies.
Generate notification candidates.
Persist every valid observed ID, including records filtered out by keywords.

That final choice gives keyword filters at-most-once behavior. Changing a keyword later does not replay posts the workflow already processed.

State corruption also fails closed. If a non-empty state object has no supported schema or contains malformed channel state, the workflow stops instead of silently treating history as empty. Resetting broken state would turn old records into a fresh alert storm.

The complete comparator and the n8n Code-node implementation are in the public companion repository. The comparator is dependency-free and accepts either a bare dataset array or an MCP-style { "items": [...] } envelope.

Treat a full polling window as a warning

Suppose the workflow requests five messages and receives five. It does not know whether exactly five were available or whether older unseen messages fell outside the returned window.

The workflow therefore sets windowSaturated using the raw Actor response count:

const windowSaturated = rows.length >= Number(monitor.limit);

The check happens before deduplication. Duplicate or invalid rows should not hide the fact that the raw window reached its cap.

Saturation means:

The configured window may be too small for the polling interval.

It does not prove that a message was missed. It tells the operator to consider a larger limit, a shorter interval, or a proper catch-up strategy.

The same reasoning applies in the other direction: an ID that was previously seen but is absent from a capped current window is not automatically deleted. It may simply have fallen beyond the window.

MCP proved the information boundary; n8n owns the recurring state

The two caller paths in this project are intentionally separate:

The experiment used an AI client and the Apify MCP server to test what could be determined with and without prior state.
The recurring implementation uses n8n and Apify’s synchronous REST endpoint to fetch dataset items, compare them with n8n workflow state, and emit candidates.

They call the same Actor, but they are not one combined MCP-to-n8n pipeline.

In n8n, the HTTP Request node calls:

POST https://api.apify.com/v2/acts/{ACTOR_ID}/run-sync-get-dataset-items

The Apify API documents this endpoint as running an Actor synchronously and returning its dataset items. Store the Apify API token in an n8n Bearer Auth credential; do not paste it into the workflow JSON.

What I verified

The corrected companion package includes:

a dependency-free comparator with 23 deterministic tests;
an importable n8n workflow whose embedded state engine has 11 deterministic tests;
a fail-closed state schema;
first-run baseline suppression;
per-channel state partitions;
duplicate prevention across triggered runs; and
saturation reporting based on the raw response count.

I also retained two successful triggered n8n executions:

Execution	Result
First triggered run	Created independent baselines for two channels; zero notification candidates
Second triggered run	Reused the persisted state; all ten returned IDs were already seen

The second execution reading the first execution’s state is the important result. It demonstrates persistence across triggered production runs rather than only inside one editor session.

What I did not verify

The evidence has limits:

No genuinely new live Telegram ID appeared during the retained runs. The positive new-ID path is covered by deterministic constructed tests.
The corrected public workflow stops at notification candidates; Telegram delivery is intentionally disabled and untested.
Every observed result window reached the configured limit of five, so the experiment does not establish catch-up completeness.
n8n workflow static data was tested only with small state and serialized executions.
The MCP experiment and n8n workflow ran at different times and are not a one-variable performance benchmark.

Those boundaries do not weaken the state contract. They define exactly what the implementation proves—and what the next production test must cover.

The checklist I now use for every scheduled Actor

Before turning any recurring Actor call into a monitor, I answer these questions:

What is the stable identity? Partition it by source, tenant, channel, or account where necessary.
Which fields are mutable? Keep them out of identity unless change detection is explicitly the goal.
Where does the baseline live? Give one layer clear ownership.
What happens on the first non-empty run? Baseline or alert—never accidental behavior.
What happens when state is missing or malformed? Fail closed instead of replaying history.
Can the polling window saturate? Surface that condition before deduplication.
What does absence mean? In a capped window, it does not prove deletion.
Can executions overlap? If yes, use storage with explicit concurrency guarantees.

A scheduled scraper repeatedly tells you what exists. A monitor tells you what changed without inventing certainty, replaying old records, or hiding gaps.

The difference is not another API call. It is a state contract.

Resources

AI assistance disclosure: I used AI tools during drafting, editing, and QA. I reviewed the final article and supporting evidence and take responsibility for its claims and conclusions.

How to Debug Webhooks Like a Pro: Building an Enterprise-Grade Webhook Testing Tool

Wed, 31 Dec 2025 21:56:31 +0000

Debugging webhooks is one of the most frustrating experiences in modern web development. You configure a Stripe payment hook, wait for a transaction, and... nothing happens. Was the request sent? Was it malformed? Did your server even receive it?

In this tutorial, I'll show you how I built an enterprise-grade webhook debugging tool using Node.js and the Apify platform, and share the key patterns that make it production-ready.

The Problem: Why Webhook Debugging Is So Painful

Every developer who has integrated with Stripe, GitHub, Shopify, or any webhook-based service has experienced these issues:

No visibility into what data is being sent
Localhost is unreachable from external services
Tunneling tools like ngrok add complexity and security concerns
Failed webhooks require reconfiguring the source service

The traditional workflow looks like this:

1. Deploy code to staging
2. Configure webhook URL
3. Trigger event
4. Check logs (if they exist)
5. Realize data is wrong
6. Repeat steps 1-5 indefinitely

This is incredibly slow. What if you could capture, inspect, and replay webhooks instantly?

The Solution Architecture

Here's the high-level architecture of a webhook debugging server:

┌─────────────┐     ┌──────────────────┐     ┌─────────────┐
│   Stripe    │────▶│  Webhook Server  │────▶│   Dataset   │
│   GitHub    │     │  (Express.js)    │     │   (Logs)    │
│   Shopify   │     └────────┬─────────┘     └─────────────┘
└─────────────┘              │
                             ▼
                    ┌──────────────────┐
                    │  SSE Stream      │
                    │  (Real-time UI)  │
                    └──────────────────┘

The core components are:

Express server to receive webhook requests
Logging middleware to capture full request details
SSE broadcasting for real-time monitoring
Persistence layer for replay functionality

Building the Core Middleware

The heart of the system is a middleware that captures every detail of incoming requests:

// logger_middleware.js
import { nanoid } from "nanoid";

function createLoggerMiddleware(webhookManager, options, broadcast) {
  return async (req, res) => {
    const startTime = Date.now();
    const webhookId = req.params.id;

    // Validate the webhook exists and hasn't expired
    if (!webhookManager.isValid(webhookId)) {
      return res.status(404).json({
        error: "Webhook ID not found or expired",
      });
    }

    // Capture full request details
    const event = {
      id: nanoid(10),
      timestamp: new Date().toISOString(),
      webhookId,
      method: req.method,
      headers: maskSensitiveHeaders(req.headers),
      query: req.query,
      body: parseBody(req.body),
      contentType: req.headers["content-type"],
      remoteIp: req.ip,
      processingTime: Date.now() - startTime,
    };

    // Send response immediately
    res.status(options.defaultResponseCode).send(options.defaultResponseBody);

    // Background: Store and broadcast
    await storeEvent(event);
    broadcast(event); // SSE to all connected clients
  };
}

Key insight: We respond to the webhook source immediately, then handle storage and broadcasting asynchronously. This ensures sub-10ms response times.

Security: Timing-Safe Authentication

When implementing API key authentication, a common mistake is using simple string comparison:

// ❌ VULNERABLE to timing attacks
if (providedKey === expectedKey) {
  return { isValid: true };
}

An attacker can measure response times to guess your key character-by-character. Here's the secure approach:

// ✅ SECURE - Uses timing-safe comparison
import { timingSafeEqual } from "crypto";

export function validateAuth(req, authKey) {
  if (!authKey) return { isValid: true };

  const providedKey = extractKeyFromRequest(req);
  if (!providedKey) {
    return { isValid: false, error: "Missing API key" };
  }

  const expectedBuffer = Buffer.from(authKey);
  const providedBuffer = Buffer.from(providedKey);

  // timingSafeEqual requires same-length buffers
  if (expectedBuffer.length !== providedBuffer.length) {
    // Perform dummy comparison to avoid timing shortcut
    timingSafeEqual(expectedBuffer, expectedBuffer);
    return { isValid: false, error: "Invalid API key" };
  }

  const isValid = timingSafeEqual(expectedBuffer, providedBuffer);
  return { isValid, error: isValid ? null : "Invalid API key" };
}

This prevents timing-based attacks by ensuring the comparison always takes the same amount of time, regardless of how many characters match.

Real-Time Monitoring with Server-Sent Events

Instead of polling for new webhooks, we use Server-Sent Events (SSE) for instant updates:

// Set up SSE endpoint
const clients = new Set();

app.get("/log-stream", (req, res) => {
  // SSE headers
  res.setHeader("Content-Type", "text/event-stream");
  res.setHeader("Cache-Control", "no-cache");
  res.setHeader("Connection", "keep-alive");
  res.flushHeaders();

  // Track this client
  clients.add(res);

  // Clean up on disconnect
  req.on("close", () => clients.delete(res));
});

// Broadcast to all connected clients
function broadcast(data) {
  const message = `data: ${JSON.stringify(data)}\n\n`;
  clients.forEach((client) => client.write(message));
}

Tip: Add a heartbeat to keep connections alive through proxies:

setInterval(() => {
  clients.forEach((client) => client.write(": heartbeat\n\n"));
}, 30000);

Rate Limiting with LRU Eviction

To prevent abuse without running out of memory, we implement an in-memory rate limiter with LRU eviction:

export class RateLimiter {
  constructor(limit, windowMs, maxEntries = 1000) {
    this.limit = limit;
    this.windowMs = windowMs;
    this.maxEntries = maxEntries;
    this.hits = new Map(); // Maintains insertion order for LRU
  }

  middleware() {
    return (req, res, next) => {
      const ip = req.ip;
      const now = Date.now();

      let userHits = this.hits.get(ip);

      if (!userHits) {
        // Evict oldest entry if at capacity
        if (this.hits.size >= this.maxEntries) {
          const oldest = this.hits.keys().next().value;
          this.hits.delete(oldest);
        }
        userHits = [];
      } else {
        // LRU: Re-insert to move to end
        this.hits.delete(ip);
      }

      // Filter to recent window only
      const recentHits = userHits.filter((h) => now - h < this.windowMs);

      if (recentHits.length >= this.limit) {
        return res.status(429).json({ error: "Rate limit exceeded" });
      }

      recentHits.push(now);
      this.hits.set(ip, recentHits);
      next();
    };
  }
}

The LRU pattern ensures that frequently accessed IPs stay in memory while inactive ones get evicted first.

Hot-Reloading Configuration

One advanced feature is zero-downtime configuration updates. Using Apify's Actor events, we can update settings without restarting:

Actor.on("input", async (newInput) => {
  console.log("[SYSTEM] Applying new settings...");

  // Update authentication
  currentAuthKey = newInput.authKey;

  // Update rate limits
  rateLimiter.limit = newInput.rateLimitPerMinute || 60;

  // Recompile custom scripts
  if (newInput.customScript !== oldScript) {
    compiledScript = new vm.Script(newInput.customScript);
  }

  console.log("[SYSTEM] Hot-reload complete!");
});

This allows you to change API keys, rate limits, or custom scripts while the server is live.

Frequently Asked Questions

Q: How do I handle large payloads?

Set a maximum payload size and reject oversized requests early:

app.use(bodyParser.raw({ limit: "10mb", type: "*/*" }));

Q: Can I validate incoming webhook data?

Yes! Use JSON Schema validation to reject malformed payloads automatically.

Q: How do I replay a failed webhook?

Store the original request data and expose a /replay/:id endpoint that re-sends it to a new destination.

Conclusion

Building an enterprise-grade webhook debugging tool requires attention to several key areas:

Performance: Respond immediately, process asynchronously
Security: Use timing-safe comparisons for authentication
Monitoring: Implement SSE for real-time visibility
Resilience: Add rate limiting with memory-safe eviction
Flexibility: Support hot-reloading for zero-downtime updates

The full implementation is open-source and available on GitHub. You can also try it live on the Apify Store.

For more advanced patterns like custom scripting, request forwarding, and JSON schema validation, check out the project's workflow playbooks.

Happy debugging!