This repository was archived by the owner on Jan 23, 2023. It is now read-only.
Repository navigation
This repository was archived by the owner on Jan 23, 2023. It is now read-only.
Missing tweets in the timeline. #147
Description
I can't prove it with a specific example, obviously, but I notice it a 2nd time already.
Two days ago I've closed cawbird, and only now remembered, that it was closed.
So I open it, and scroll back till I find the last tweets I've seen and then some more.
And I notice some earlier tweets that I have not seen the last time I've had cawbird open.
I'm not 100% sure, so I'd like to ask people to take a look if they can see it, too.
PS It might be cawbird or it may be the twitter api, or something else.
Activity
IBBoard commentedon May 1, 2020
I wondered whether I'd seen this a few times, but I think it's technically out of our control. This may get a little technical. TL;DR: We're not doing anything wrong, it's just the way Twitter works, but we might be able to work around it and do better.
Because of its size, Twitter's database is "eventually consistent". Twitter specifically said once:
What that means is that if someone posts a tweet from Japan and then you query from the UK, it's entirely possible that the server that you're asking for updates doesn't yet know about the new tweet. Someone querying from Japan would get the new tweet, though, because they'd ask a server that knew it.
By the time you close and re-open Cawbird then Twitter's database has eventually become consistent (for that tweet) and so it appears where it was missed before.
Technically, we're doing everything correct. We're retrieving new tweets from the timeline since the last tweet we saw. Which is fine. But it might mean that there are occasions when we pull tweet every tweet up to X, then a new one appears at X-1â , but our next request is "give me everything after X" and so we miss X-1.
What I've considered (but never done) is to adjust the "everything since" parameter and push it back a little bit to give us some overlap to get the "eventually consistent" tweets that appear later. The problem is that you're then balancing "are we doing enough overlap to catch everything?" (we don't know exactly how long "eventually" takes) with "are we wasting bandwidth pulling too many duplicate tweets?".
â IDs are a combination of timestamp, the worker that accepted the tweet, and a sequence number to avoid duplication (src)
j1warren commentedon May 1, 2020
Maybe create a configurable parameter for overlap and when displaying those 'lost' tweets highlight them and log the sufficient overlap in logs for these lost&found tweets.
So after time we could know what overlap covers 100% cases.
IBBoard commentedon May 1, 2020
Yeah, the first approach would definitely be to put in a bigger overlap than I think is necessary and then log what gets missed while I ran it locally for a while (and pushed the code here for others to use, if they could rebuild their own).
There might always be a "long tail" to the distribution, though. One second might cover 95%, 15 seconds covers 99%, 60 seconds covers 99.9%, and so on. It's unlikely ("eventually consistent" is normally quite quick, because otherwise the database is fairly useless) but unlikely is not impossible.
We could go back an entire hour. But we can only pull 200 tweets in one go (and we do it every two minutes) so rapidly changing feeds might miss or delay tweets because they won't fit in the 200 tweet cap!
But as I said, there's only so much we can do.
I guess the other question is what we do with the UI, which will partly depend on how often it occurs and how far back they go. Do we just insert them where they should be (and so you might miss them if they're below where you're currently looking) or do we put them at the top and show them slightly out of order (but we sort by tweet ID at the moment, so what do we sort on?)
Re #147: Add candidate for fetching missed tweets
IBBoard commentedon May 17, 2020
Okay, a first cut approach with a bit of debugging is in place. I've not seen anything yet, but it is Sunday evening and people aren't posting that much.
Note that the numbers won't just end "âĶ00000" because the IDs are built by ORing binary values, not the base-10 numbers.
Current overlap is ~8s. To find the Unix timestamps of tweets do
((id>>22)/1000) + 1288834974(because Twitter works to milliseconds and has its own epoch!)j1warren commentedon May 18, 2020
Great! Compiled, will run this for a while to see, too.
Re #147: Rework missed tweet logging
Re #147: Try to fix rogue "unread" message
j1warren commentedon May 19, 2020
Got some
IBBoard commentedon May 19, 2020
That's helpful, thanks. I need a script to pull these apart rather than doing it by hand!
Quick check of the first one says that it's got the same timestamp (1589828955) but then the first bits of the rest of it differs. I'll have to check more, though, because I must be doing something wrong as I'm finding that the "missed" one has a higher value in the rest of the bits (which should make it later!)
IBBoard commentedon May 20, 2020
Okay, a one-liner (ish) in Bash (mostly).
Change
tweetidand re-run to get the different values.That gives us:
The biggest gap we see there is 4 seconds (1262480092174757888 was loaded before we saw 1262480074911203328). And I've also realised that the rest of the bits in the ID aren't important, because as soon as the time is earlier then the whole ID is earlier! (obviously)
So, given that it is only occasional tweets, they're within a few seconds of the latest tweet, and you've gone several polls without any update, I think it's probably safe to just insert them in-line (i.e. where they should be, and not at the top of the window).
Re #147: Tidy up logic and logging
IBBoard commentedon May 20, 2020
Okay, I think that fixes it and has just enough useful logging for the final version without being a complete debug version.
I'll try to keep an eye on it for a bit to see how much we can reduce the overlap (because one minute of overlap when we're polling every two minutes is quite a lot!). Any further examples of time gaps would be good.
j1warren commentedon May 21, 2020
FYI, I've ran for 2 days with those 2 commits from 3 days ago - didn't have a single missed tweet logged.
Will recompile with 7d04c85.
IBBoard commentedon May 23, 2020
Okay, so it's definitely a rare event then. And so far we might get away with ~10 seconds (which I guess will be 8 seconds, because we're doing a binary mask)
Re #147: Reduce home timeline overlap to ~8s
Re #147: Add candidate for fetching missed tweets
j1warren commentedon Sep 30, 2020
I know there's probably nothing to be done, but I've just restarted cawbird and found some tweets that were missing from my previous long-running instance.
Maybe new twitter API will fix this.
IBBoard commentedon Oct 3, 2020
@j1warren Can you remember what kind of a gap it was? And was it many or just a few? And from one user or many?
We can try increasing the overlap, but the more we do that then the more often people are going to re-fetch known tweets as well.
At the end of the day, Twitter is using an "eventually consistent" database, so nothing we do is ever going to be 100% perfect.
j1warren commentedon Oct 7, 2020
@IBBoard
I scrolled till 2 days back, and it was a 2 single tweets from different people.
I have a feeling it might the twitter itself, importing them after hour/two into the feed because if their own internal delays.
I still haven't implemented #2, otherwise I'd tell you the exact timings.
IBBoard commentedon Oct 7, 2020
And you're certain they weren't there? Because sometimes I use my mobile, see a tweet in the middle of a batch of tweets that I was sure wasn't there, and then go back and find it was there and I just blanked it ð
If you did implement #2 then you'd have to track the retrieval time as well to see where it fitted in the chronology. I'm hoping it was still a small gap, but just too long for us to pick up. Because if they are adding tweets after an hour or more then we stand no chance! (But that is technically possible for an "eventually consistent" database)
IBBoard commentedon Oct 13, 2020
And sometimes there's just nothing we can do.
Opened Cawbird. There's a thread from Tim Bray showing tweets 1, 4, 5, 6 and 7. But it should have been in the middle of a batch, so it doesn't make sense.
Close and re-open and now I see 1, 2, 3, 6 and 7. So even between requests the API isn't always consistent!
Re #292: Don't notify on tweet overlap