Twitter (X) Scraper in Python: Tweet Metrics, SQLite Snapshots, and the Walls You'll Hit (2026)

Wed, 14 Oct 2026 09:11:05 +0000

Most people who search for a "Twitter scraper" don't need the whole firehose. They have a list of tweet URLs (a launch, a campaign, posts from a few creators) and want to know how many likes and replies each one has, and how that changes over a week. That job is still doable in plain Python without logging in. Pulling a user's whole timeline is a different story, and I'll be honest about where it breaks.

This guide builds a small tracker: tweet URLs go in, and a SQLite table of engagement snapshots comes out.

What still works without logging in

When a site embeds a tweet, the embed widget loads its data from a public JSON endpoint:

https://cdn.syndication.twimg.com/tweet-result?id=<tweet_id>&token=<token>

It returns the tweet text, creation date, author (name, handle, verified flags), media, favorite_count (likes) and conversation_count (replies). There's no API key and no cookie. The token is derived from the tweet ID with a small formula from the open-source embed code. Today the endpoint seems to accept any non-empty token, but an empty one returns {}, so compute it the same way the widget does in case that changes.

What you don't get matters just as much: no retweet count, no quote count, no view count, and no way to list a user's tweets. Keep that in mind before you build a dashboard on it.

Setup

python -m venv .venv && source .venv/bin/activate
pip install requests

Step 1: Turn URLs into IDs (and IDs into dates)

Tweet IDs are "snowflakes": the top bits are a timestamp in milliseconds since X's custom epoch. That means you can get the post time from the ID alone, which is handy for sorting before you've fetched anything. This only holds for tweets from November 2010 onward, when snowflake IDs started. Older IDs are small sequential numbers, so prefer the created_at field once you have the tweet.

import math
import re
from datetime import datetime, timezone

ID_RE = re.compile(r"(?:twitter|x)\.com/[^/]+/status(?:es)?/(\d+)")
TWITTER_EPOCH_MS = 1288834974657


def tweet_id(url_or_id: str) -> str:
    if url_or_id.isdigit():
        return url_or_id
    m = ID_RE.search(url_or_id)
    if not m:
        raise ValueError(f"not a tweet URL: {url_or_id}")
    return m.group(1)


def posted_at(tid: str) -> datetime:
    ms = (int(tid) >> 22) + TWITTER_EPOCH_MS
    return datetime.fromtimestamp(ms / 1000, tz=timezone.utc)

Step 2: Compute the embed token

The embed code does ((id / 1e15) * Math.PI).toString(36) and then strips every 0 and the dot. Python has no float-to-base-36, so here's a short port:

DIGITS = "0123456789abcdefghijklmnopqrstuvwxyz"


def _to_base36(x: float, frac_digits: int = 12) -> str:
    whole, frac = int(x), x - int(x)
    out = ""
    while whole:
        out = DIGITS[whole % 36] + out
        whole //= 36
    out = (out or "0") + "."
    for _ in range(frac_digits):
        frac *= 36
        d = int(frac)
        out += DIGITS[d]
        frac -= d
    return out


def embed_token(tid: str) -> str:
    return re.sub(r"0+|\.", "", _to_base36((int(tid) / 1e15) * math.pi))

Step 3: Fetch one tweet, politely

import random
import time
import requests

SESSION = requests.Session()
SESSION.headers["User-Agent"] = (
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
    "(KHTML, like Gecko) Chrome/129.0 Safari/537.36"
)
ENDPOINT = "https://cdn.syndication.twimg.com/tweet-result"


def fetch_tweet(tid: str, retries: int = 4) -> dict | None:
    for attempt in range(retries):
        r = SESSION.get(
            ENDPOINT,
            params={"id": tid, "token": embed_token(tid), "lang": "en"},
            timeout=20,
        )
        if r.status_code == 404:
            return None  # deleted, protected, or never existed
        if r.status_code == 429 or r.status_code >= 500:
            time.sleep(2 ** attempt + random.random())
            continue
        r.raise_for_status()
        data = r.json()
        if not data or data.get("__typename") != "Tweet":
            return None  # empty object or a tombstone (withheld, age-gated...)
        return data
    raise RuntimeError(f"gave up on {tid} after {retries} tries")

Three things to notice. A 404 is normal and not an error: tweets get deleted and accounts go private. An empty {} or a non-Tweet type also means "nothing usable". And retry only on 429 and 5xx, with backoff and jitter.

Step 4: Store snapshots in SQLite

One row per tweet per run. Never overwrite, because the history is the useful part.

import sqlite3

db = sqlite3.connect("tweets.db")
db.executescript("""
CREATE TABLE IF NOT EXISTS tweets (
    id TEXT PRIMARY KEY, handle TEXT, posted_at TEXT, text TEXT
);
CREATE TABLE IF NOT EXISTS snapshots (
    id TEXT, fetched_at TEXT, likes INTEGER, replies INTEGER,
    PRIMARY KEY (id, fetched_at)
);
""")


def track(urls: list[str]) -> None:
    now = datetime.now(timezone.utc).isoformat(timespec="seconds")
    for url in urls:
        tid = tweet_id(url)
        t = fetch_tweet(tid)
        if t is None:
            print(f"skip {tid}: unavailable")
            continue
        db.execute(
            "INSERT OR IGNORE INTO tweets VALUES (?, ?, ?, ?)",
            (tid, t["user"]["screen_name"], t.get("created_at") or posted_at(tid).isoformat(), t["text"]),
        )
        db.execute(
            "INSERT OR REPLACE INTO snapshots VALUES (?, ?, ?, ?)",
            (tid, now, t.get("favorite_count", 0), t.get("conversation_count", 0)),
        )
        time.sleep(1.5 + random.random())  # stay well under any rate limit
    db.commit()


if __name__ == "__main__":
    track(open("tweets.txt").read().split())

Put one URL per line in tweets.txt and run it from cron every few hours.

Step 5: See what moved

SELECT t.handle, s.id,
       MAX(s.likes) - MIN(s.likes)     AS likes_gained,
       MAX(s.replies) - MIN(s.replies) AS replies_gained,
       COUNT(*)                        AS snapshots
FROM snapshots s JOIN tweets t USING (id)
WHERE s.fetched_at >= datetime('now', '-7 days')
GROUP BY s.id
ORDER BY likes_gained DESC;

For tracking a few hundred known tweets, that's the whole thing, and it has been stable for me.

The walls you'll hit

No timelines. The tracker only works when you already know the tweet IDs. The embed timeline endpoint (syndication.twitter.com/srv/timeline-profile/...) answered my very first request with 429 Rate limit exceeded, and logged-out x.com profile pages render almost nothing useful in raw HTML.

Missing metrics. No retweets, quotes, or views here. If those numbers matter, this endpoint isn't enough.

Guest tokens and GraphQL. The web app gets data from internal GraphQL operations (UserByScreenName, UserTweets and so on). You can get a guest token without logging in, but the operation IDs and required features flags change with each web release, and X keeps narrowing what guests can see. Expect to fix that code often.

Logged-in cookies. Scraping with your own session cookies works until the account gets rate-limited, challenged, or locked. Don't do it with an account you care about, and read X's terms before you scale anything.

The official API. It's the clean route, but the free tier is very limited for reads and the paid tiers are priced for companies. Check the current plans on X's developer portal before you plan around it.

When I stop doing it myself

If the job grows from "these 200 tweets" into "every new post from these 50 accounts, with views and retweets", the guest-token treadmill eats more time than it saves. That's where I use an API. For example, ScrapeAtlas* has GET /v1/twitter/user-tweets?handle=<handle> (newest first, with optional since/until) and GET /v1/twitter/profile?handle=<handle>. Each tweet comes back with legacy.favorite_count, retweet_count, reply_count, quote_count, views.count and its URL:

import os
import requests

r = requests.get(
    "https://api.scrapeatlas.com/v1/twitter/user-tweets",
    params={"handle": "python_tip", "since": "2026-10-01T00:00:00Z"},
    headers={"x-api-key": os.environ["SCRAPEATLAS_API_KEY"]},
    timeout=60,
)
r.raise_for_status()
body = r.json()
if not body["coverage"]["complete"]:
    print("partial result:", body["coverage"]["warnings"])
for tw in body["tweets"]:
    lg = tw["legacy"]
    print(tw["url"], lg["favorite_count"], lg["retweet_count"], tw.get("views", {}).get("count"))

Check coverage.complete and warnings every time. A partial page is labelled as one, and you should treat it that way rather than as "this account posted less".

Whichever route you take, keep the snapshot table. The numbers that matter are the changes over time, not one reading. If you've found a more stable logged-out way to list a user's recent tweets, I'd like to hear it in the comments.

*I build it.

Coder Legion - Recent posts tagged twitter