Cawbird consumes a lot of memory over time

Describe the bug
It's not noticeable (at least not on my machine) but if you leave Cawbird running then it'll slowly consume more and more memory.

This isn't just to do with new tweets stacking up when it isn't auto-scrolling, because scrolling to the top and waiting for it to clear the older tweets doesn't reduce the memory usage by much.

To Reproduce
Steps to reproduce the behavior:

Open Cawbird and set it so that won't scroll with new tweets
Leave it running for a few hours so that 100+ new tweets appear
Check memory usage with ps aux | grep "[c]awbird" (column 4 is system RAM percentage, 5 is absolute amount)
Scroll to the top
Check the memory usage again

Expected behavior
At step 3, Cawbird is using some extra memory compared to startup, but not much.

At step 5, it's back down to startup levels.

** Actual behaviour **
At step 3:
ibboard 4665 0.6 4.1 1612168 1015320 ? Sl 10:36 1:33 /usr/bin/cawbird --gapplication-service
At step 5:
ibboard 4665 0.7 3.9 1505916 965580 ? Sl 10:36 2:02 /usr/bin/cawbird --gapplication-service
(Values after ~5hrs running)

System details:

OS: openSUSE Tumbleweed
Version: 1.0.5
Package type: official RPM

Additional context
If anyone knows about understanding memory usage of an app (beyond running a basic Valgrind and seeing if it reports memory leaks) then please help!

Possible candidate cause (but I'm just starting to learn C/memory analysis tools):

90.71% (92,283,097B) (heap allocation functions) malloc/new/new[], --alloc-fns, etc.
->44.25% (45,017,836B) 0x6353639: ??? (in /usr/lib64/libpixman-1.so.0.36.0)
| ->44.25% (45,017,836B) 0x63536D8: ??? (in /usr/lib64/libpixman-1.so.0.36.0)
|   ->44.25% (45,017,836B) 0x534FD6E: ??? (in /usr/lib64/libcairo.so.2.11600.0)
|     ->44.24% (45,007,364B) 0x1A3894: load_animation (CbMediaDownloader.c:110)
|     | ->44.24% (45,007,364B) 0x1A4354: cb_media_downloader_load_threaded (CbMediaDownloader.c:376)
⋮

I noticed this as well. Does cawbird delete posts (and more important: their media attachments) from memory overtime and during scrolling?
My behaviour is that I often scroll quite far to the bottom to read the last hours of timeline, then jump back to the top and then manually scroll upwards when new content appears.

Re #142: Initial debugging

It should delete them, and from a quick bit of debugging then I can see that it is disposing of media when it removes tweets (so scrolling back would use lots of memory as it loads tweets and images, but get to the top and it'll purge the old ones and free the memory). The media for the thumbnails does appear to be coming out at up to 1200×1200, though, so scroll back a few hours past a load of pictures and those thumbnails could quickly consume quite a bit of memory.

This probably wouldn't be so much of a problem if we weren't doing stuff in C! I'm just starting on some debugging code to understand what it's doing.

Trying to work out whether we are actually using a lot of memory, or whether VSZ is just a misleading number and the OS would reclaim a load if it needed to.

Progression of ps aux | grep [c]awbird

ibboard  20281  0.3  1.3 1108224 342884 pts/0  Sl+  12:10   0:43 build/cawbird
…
ibboard  20281  0.2  2.4 1196256 611972 pts/0  Rl+  12:10   1:17 build/cawbird
…
ibboard  20281  0.3  2.3 1241736 577716 pts/0  Sl+  12:10   1:32 build/cawbird
…
ibboard  20281  0.3  2.3 1241736 577716 pts/0  Sl+  12:10   1:34 build/cawbird
…
ibboard  20281  0.4  2.2 2116432 540372 pts/0  Sl+  12:10   2:15 build/cawbird

Given that we're potentially (in my case) loading dozens of 1200x1200px images and holding that it the Cb.Media object before scaling dynamically to ~400x400px then that may explain the 500MB resident (RSS) because each one is 32MB as a bitmap and I don't know what optimisation (if any) GdkPixmap might do. The name "pixmap" suggests none.

For reference, free -m is showing:

              total        used        free      shared  buff/cache   available
Mem:          23982        5177       11158         432        7646       18040
Swap:          5113           0        5113

So I'm nowhere near needing the OS to collect any free'd memory.

Yeah, I think we might be freeing memory, and we just need to do something more sensible with images.

Steps:

Load Cawbird
Run ps aux | grep [c]awbird
- VSZ and RSS: 1092628 367024
Scroll down one tweet
Scroll up one tweet
Cawbird prunes old tweets
Run ps aux | grep [c]awbird
- VSZ and RSS: 1092628 200632

So it has pruned ~160MB of RSS, but VSZ has remained the same. Which suggests it was free'ing the media correctly, but VSZ doesn't instantly come down?

Interestingly, I just came across this article, which has a different command to the basic process listing that I'd been using above:

ps -eo 'pid user rss:8 size:8 cmd' --sort 'rss'

Comparing the two, I get:
ibboard 2890 3.7 1.9 1282952 479896 ? Sl 18:50 1:22 /usr/bin/cawbird --gapplication-service
2890 ibboard 479896 496716 /usr/bin/cawbird --gapplication-service

So the virtual size is much closer to the resident size. man ps says that size (rather than vsz) is "approximate amount of swap space that would be required if the process were to dirty all writable pages and then be swapped out", which does feel like "actual memory usage" (i.e. vsz includes loads of spare, unused space).

Although this other article suggests that VSZ is everything that has been allocated, not just looked at. Which leaves me wondering if there's something we should be doing to make it get un-allocated.

I hate low-level languages and memory management!

Just ran /usr/bin/time -f "RSS: %MkiB" cawbird, which the Daniel Lange article in the previous post (first link) said will show maximum RSS when I quit. I only ran for five minutes or so, but it only gave RSS: 280056kiB.

I think this is to do with thumbnails an unnecessary scaling rather than any kind of memory leaks.

So, the solution is simple. Stop showing images!

Or I could do something better with the thumbnails and caching and see if that helps. A quick hack that I did the other night affected the displayed image size as well.

So, the solution is simple. Stop showing images!

Does it help? It shoots up to ~>500 Mb in just 3 srolls and I'm hiding images in my timeline. Scrolling and loading new tweets is the trigger. Will running Valgrind help?

I've had a quick look at the code and it looks like it downloads images and stores them as objects in memory regardless of whether you have images visible in the timeline or not. That's good for responsiveness and consistency (if you turn them on then you won't be waiting for them to be downloaded) but not for memory usage.

The suggestion of stopping showing images was a non-serious solution of fixing memory usage by just not showing images at all.

What metric are you using for ">500MB"? There's a couple of options in the previous comments that very rarely get to that level for me, even after long runs. And my first comment after the initial report is valgrind output.

What metric are you using for ">500MB"? There's a couple of options in the previous comments that very rarely get to that level for me, even after long runs.

I ran memory profiler and plotter, here's a graph, I made 3 full scrolls,opened 2 tweets one with picture one without,photos were hidden in timeline. If I leave it like this it goes around 570-600 Mb in task manager which is more than my browser with two tabs open. Is this normal? I missed that output.

But which version of "memory usage" is that using? RSS (resident set size - current actual memory usage) or VSZ (virtual size - effectively the potential memory usage, including stuff it's not using). All of which is made to look worse by the fact that those numbers may include the size of all of the shared libraries (gtk, etc), which then get double-counted across the system.

At least that's what I understand from a lot of reading around. Memory usage isn't a simple "this app is using this amount of my memory" thing.

As for what's "normal", whatever it does now is normal for the way it is written. Valgrind isn't showing any memory leaks, so we're using exactly what GTK/libsoup/various other libraries need to use. So the only way to get better (lower) is to improve what we hold in memory.

Re #142: Distinguish thumb size from real size

Re #142: Only scale an image when necessary

Re #142: Set default sizes for thumb width/height

Re #142: Fix third-party images

Re #142: Destroy cairo context after use

Re #142: Add support for loading full-res images

Re #142: Tidy up debugging

Re #142: Fix >2048px images resizing windows

Re #142: Delay image loading until required

Re #142: Show "large" size to avoid resize on load

Re #142: Load all images from Tweet

Re #142: Swap out assertions for early return

Re #142: Move some responsibilities to Media

Okay, that should be a good bit of optimisation. We'll still slowly eat memory if people have "scroll on new tweets" disabled and don't look at it for hours on end, but there's not much we can do about that without getting far more complex (and less responsive for many people, and prone to issues) with "load as it comes in to view" behaviour.

Navigation Menu

Repository navigation

Cawbird consumes a lot of memory over time #142

19 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Cawbird consumes a lot of memory over time #142

Description

Activity

IBBoard commented on Apr 25, 2020

schmittlauch commented on Apr 26, 2020

IBBoard commented on Apr 26, 2020

IBBoard commented on Apr 26, 2020

IBBoard commented on Apr 26, 2020

IBBoard commented on Apr 28, 2020

IBBoard commented on Apr 28, 2020

bbhtt commented on Jul 6, 2020

IBBoard commented on Jul 7, 2020

bbhtt commented on Jul 8, 2020

IBBoard commented on Jul 8, 2020

19 remaining items

IBBoard commented on Aug 27, 2020

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions