Cawbird crashes on playing video

Describe the bug
After clicking a video for playback, cawbird at random times would crash

To Reproduce
It happens randomly so no way to repo it

Expected behavior
Expect it not to crash

System details:

OS: Linux Mint 19.3
Version: cawbird_1.0.4-1+12.1_amd64
Package type: Deb

Additional context
Ran cawbird in gdb and after it halted, i killed it and this was the output

Thread 1 "cawbird" received signal SIGINT, Interrupt.
__lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
135	../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S: No such file or directory.

bt full output
cawbird video crash.txt

I'm not entirely sure that that's a Cawbird bug. It looks like it's happening deep in GStreamer, and Google shows a lot of other near identical errors in a variety of apps.

In fact, I can't even see any Cawbird calls in that stack trace, so it's going to be REALLY tricky to pin down!

Can you test with the video of Llandudno's goats that my "ibbtwtr" account just RTed and then test with GStreamer directly with gst-play-1.0 https://video.twimg.com/ext_tw_video/1243329209918988295/pu/vid/720x1280/kWa3JQP6QqlrW052.mp4?tag=10. That should tell us whether it's a pure GStreamer issue.

The other thing would be to run Cawbird with G_MESSAGES_DEBUG=cawbird cawbird and see what it prints at the command-line around the crash.

I've had it crash on a video and then reopen cawbird to play the same video again and no crash, so no way to pin it down, which is why i provided the gdb.

I'm running cawbird with the argument you provided and will let you know what shows up after the crash.

So it crashes everytime i click on this video.
https://twitter.com/Emirates247/status/1247209995005038594

G_MESSAGES_DEBUG=cawbird cawbird

[xcb] Unknown request in queue while appending request
[xcb] Most likely this is a multi-threaded client and XInitThreads has not been called
[xcb] Aborting, sorry about that.
cawbird: ../../src/xcb_io.c:151: append_pending_request: Assertion `!xcb_xlib_unknown_req_pending' failed.
Aborted (core dumped)

Odd. Just tested that and it works fine for me. I wonder if it's an XCB/GStreamer issue? I know odds and ends about the Linux graphics stack, but not enough.

Do you get any of the logging lines about "Buffering" or does it crash straight away?

I tried now, after a restart, and it played the first time and before it finished playing, i clicked it to close and then clicked it again to open and crash.

Thread 52 "multiqueue4:src" received signal SIGABRT, Aborted.
[Switching to Thread 0x7fffd673b700 (LWP 4900)]
__GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
51	../sysdeps/unix/sysv/linux/raise.c: No such file or directory.

bt full output
cawbird video crash 2.txt

Do you get any of the logging lines about "Buffering" or does it crash straight away?

Where can i try and pull these 'buffering' lines you mentioned?

When I load a video, I get the following debug:

(cawbird:5218): cawbird-DEBUG: 15:44:34.260: Buffering: 1
…
(cawbird:5218): cawbird-DEBUG: 15:44:34.260: Buffering: 100
(cawbird:5218): cawbird-DEBUG: 15:44:34.261: ASYNC DONE

There may be no lines or many lines in between, depending on the size of the video and how quickly it buffers.

It might be worth us having some extra logging around that as well to log what's being opened and buffered, but that's a minor thing.

The new backtrace still appears to have no direct references to our code! I did find this comment suggesting that it might be an nVidia optimisation bug, though.

Can you try __GL_THREADED_OPTIMIZATIONS=0 cawbird and see if it still happens?

(cawbird:5218): cawbird-DEBUG: 15:44:34.260: Buffering: 1
…
(cawbird:5218): cawbird-DEBUG: 15:44:34.260: Buffering: 100
(cawbird:5218): cawbird-DEBUG: 15:44:34.261: ASYNC DONE

Yes I've seen lines like this if it doesn't crash

The new backtrace still appears to have no direct references to our code! I did find this comment suggesting that it might be an nVidia optimisation bug, though.

I'm running on Intel

Can you try __GL_THREADED_OPTIMIZATIONS=0 cawbird and see if it still happens?

Yes i tried it and it still happens.

Recorded a video to potentially help.
https://www.youtube.com/watch?v=PfZeDjf4qLA

I'm running on Intel

Well, that's a bit odd. I didn't expect similar problems to hit both Intel and nVidia!

Recorded a video to potentially help.

Okay, so the window doesn't even appear. And no buffering shows up, so it's early in the process. I'll see what logging I can add, and use my home:IBBoard:desktop OBS (rather than home:IBBoard:cawbird) to build a new version at some point.

[edit] I don't suppose LIBGL_ALWAYS_SOFTWARE=1 cawbird behaves any differently, does it? (it appears to be a Mesa environment variable rather than an nVidia-specific one)

[edit] I don't suppose LIBGL_ALWAYS_SOFTWARE=1 cawbird behaves any differently, does it? (it appears to be a Mesa environment variable rather than an nVidia-specific one)

No difference. I tested it on Xubuntu 19.10 and no crash, so possibly it limited to just 18.04, which is the base for Linux Mint 19.3.

I observe the same problem, that I could resolve by adding a call to X.init_threads () in the main vala program before initializing gstreamer. The requirement for the libX11 locking infrastructure is important, because (in my case) several gstreamer elements make a lot of raw X11 requests during vaapi/libva initialization (Intel GPU), in different threads, almost simultaneously. The risk of a race condition is pretty high in that case.

Okay, interesting. Presumably that's X-specific, though, and might not work on Wayland?

I wonder whether "we haven't properly initialised the X server threads" also explains #171. It would be really useful if it did!

Re #123: Try initialising X threads to fix crashes

@philipzae

@philipzae or @fbellet - can you test Master and see if that fixes the problem?

Huh. Fair enough. It didn't crash for me, so I assumed it was doing it right! I'll try adding that tomorrow.

What error were you getting with the updated code? Because both gdk_init() and gdk_init_check() says "GTK+ initializes GDK in gtk_init() and so this function is not usually needed by GTK+ applications", which suggests that we should be fine.

(Unless it's doing something odd and we're pre-GTK at that point)

What happens in my case is that neither gdk_init() nor gtk_init() are called before calling gdk_display_get_default(), the null display causes XInitThreads() to be skipped.

(gdb) b	gtk_init
Breakpoint 2 at 0x7ffff78f17d0
(gdb) b gdk_init
Breakpoint 3 at 0x7ffff75bef50
(gdb) b	cb_utils_init_gui
Breakpoint 4 at 0x4ab64b: file ../src/CbUtils.c, line 713.
(gdb) b	XInitThreads
Breakpoint 5 at 0x7ffff6b8e890
(gdb) c
Continuing.

Breakpoint 4, cb_utils_init_gui () at ../src/CbUtils.c:713
713	  GdkDisplay *display = gdk_display_get_default ();
(gdb) n
721	  if (GDK_IS_X11_DISPLAY (display))
(gdb) print display
$1 = (GdkDisplay *) 0x0
(gdb) n
726     }

Another possibility is maybe to call init_gui() after gtk_init() is normally called, from there:

(gdb) c
Continuing.

Thread 1 "cawbird" hit Breakpoint 2, 0x00007ffff78f17d0 in gtk_init () from /lib64/libgtk-3.so.0
Missing separate debuginfos, use: dnf debuginfo-install GConf2-3.2.6-27.fc31.x86_64 dbus-glib-0.110-7.fc32.x86_64 dconf-0.36.0-1.fc32.x86_64
(gdb) bt
#0  0x00007ffff78f17d0 in gtk_init () from /lib64/libgtk-3.so.0
#1  0x00007ffff77bb63d in gtk_application_startup () from /lib64/libgtk-3.so.0
#2  0x000000000041929d in cawbird_real_startup (base=0x8031d0) at ../src/Cawbird.vala:255
#3  0x00007ffff7189ae6 in _g_closure_invoke_va () from /lib64/libgobject-2.0.so.0
#4  0x00007ffff71a2759 in g_signal_emit_valist () from /lib64/libgobject-2.0.so.0
#5  0x00007ffff71a2cd3 in g_signal_emit () from /lib64/libgobject-2.0.so.0
#6  0x00007ffff72b3f52 in g_application_register () from /lib64/libgio-2.0.so.0
#7  0x00007ffff72b434e in g_application_real_local_command_line () from /lib64/libgio-2.0.so.0
#8  0x00007ffff72b4696 in g_application_run () from /lib64/libgio-2.0.so.0
#9  0x0000000000415e0a in _vala_main (args=0x7fffffffd508, args_length1=1) at ../src/main.vala:28
#10 0x0000000000415f4a in main (argc=1, argv=0x7fffffffd508) at ../src/main.vala:18
(gdb) up
#1  0x00007ffff77bb63d in gtk_application_startup () from /lib64/libgtk-3.so.0
(gdb) up
#2  0x000000000041929d in cawbird_real_startup (base=0x8031d0) at ../src/Cawbird.vala:255
255         base.startup ();

I assume from a gstreamer point-of-view, that it doesn't matter if XInitThreads() is called sometime after gst_init() as long as it's before the playbin pipeline is constructed.
Update: ignore my last comment, XInitThreads() must be called very early, else I get segfaults in libcairo.

The problem with calling gtk_init() before XInitThreads() is that it triggers a segfault later in libcairo. So I think we should call gdk_init() and not gtk_init() just to obtain the valid display.

0x00007ffff68fed74 in pthread_mutex_lock () from /lib64/libpthread.so.0
[...]
(gdb) bt
#0  0x00007ffff68fed74 in pthread_mutex_lock () from /lib64/libpthread.so.0
#1  0x00007ffff6ba96bf in XrmQGetResource () from /lib64/libX11.so.6
#2  0x00007ffff6b8504a in XGetDefault () from /lib64/libX11.so.6
#3  0x00007ffff747f6e8 in _cairo_xlib_screen_get_font_options () from /lib64/libcairo.so.2
#4  0x00007ffff7482464 in _cairo_xlib_surface_get_font_options () from /lib64/libcairo.so.2
#5  0x00007ffff7455bd9 in cairo_surface_get_font_options () from /lib64/libcairo.so.2
#6  0x00007ffff740a9b2 in _cairo_gstate_ensure_scaled_font () from /lib64/libcairo.so.2
#7  0x00007ffff740e913 in _cairo_gstate_show_text_glyphs () from /lib64/libcairo.so.2
#8  0x00007ffff74658bb in cairo_show_glyphs () from /lib64/libcairo.so.2
#9  0x00007ffff68e5c8f in pango_cairo_renderer_show_text_glyphs () from /lib64/libpangocairo-1.0.so.0
#10 0x00007ffff68e5f75 in pango_cairo_renderer_draw_glyph_item () from /lib64/libpangocairo-1.0.so.0
#11 0x00007ffff756ddf3 in pango_renderer_draw_glyph_item () from /lib64/libpango-1.0.so.0
#12 0x00007ffff756e8e7 in pango_renderer_draw_layout_line () from /lib64/libpango-1.0.so.0
#13 0x00007ffff756ed64 in pango_renderer_draw_layout () from /lib64/libpango-1.0.so.0
#14 0x00007ffff68e6702 in pango_cairo_show_layout () from /lib64/libpangocairo-1.0.so.0
#15 0x00007ffff796726f in gtk_render_layout () from /lib64/libgtk-3.so.0
#16 0x00007ffff78db180 in gtk_label_render () from /lib64/libgtk-3.so.0
#17 0x00007ffff781eae1 in gtk_css_custom_gadget_draw () from /lib64/libgtk-3.so.0
#18 0x00007ffff7824669 in gtk_css_gadget_draw () from /lib64/libgtk-3.so.0
#19 0x00007ffff78d9235 in gtk_label_draw () from /lib64/libgtk-3.so.0
#20 0x00007ffff7a4d0b4 in gtk_widget_draw_internal () from /lib64/libgtk-3.so.0
[...]
#63 0x00007ffff7a5b6fa in gtk_window_draw () from /lib64/libgtk-3.so.0
#64 0x00007ffff7a4d0b4 in gtk_widget_draw_internal () from /lib64/libgtk-3.so.0
#65 0x00007ffff7a568f0 in gtk_widget_render () from /lib64/libgtk-3.so.0
#66 0x00007ffff78f2b66 in gtk_main_do_event () from /lib64/libgtk-3.so.0
#67 0x00007ffff75caed9 in _gdk_event_emit () from /lib64/libgdk-3.so.0
#68 0x00007ffff75dc8e1 in _gdk_window_process_updates_recurse_helper () from /lib64/libgdk-3.so.0
#69 0x00007ffff75de511 in gdk_window_process_updates_internal () from /lib64/libgdk-3.so.0
#70 0x00007ffff75de708 in gdk_window_process_updates_with_mode () from /lib64/libgdk-3.so.0
#71 0x00007ffff7189ae6 in _g_closure_invoke_va () from /lib64/libgobject-2.0.so.0
#72 0x00007ffff71a2759 in g_signal_emit_valist () from /lib64/libgobject-2.0.so.0
#73 0x00007ffff71a2cd3 in g_signal_emit () from /lib64/libgobject-2.0.so.0
#74 0x00007ffff75d50de in gdk_frame_clock_paint_idle () from /lib64/libgdk-3.so.0
#75 0x00007ffff75bf02d in gdk_threads_dispatch () from /lib64/libgdk-3.so.0
#76 0x00007ffff7eb5331 in g_timeout_dispatch () from /lib64/libglib-2.0.so.0
#77 0x00007ffff7eb478f in g_main_context_dispatch () from /lib64/libglib-2.0.so.0
#78 0x00007ffff7eb4b18 in g_main_context_iterate.constprop () from /lib64/libglib-2.0.so.0
#79 0x00007ffff7eb4be3 in g_main_context_iteration () from /lib64/libglib-2.0.so.0
#80 0x00007ffff72b472d in g_application_run () from /lib64/libgio-2.0.so.0
#81 0x0000000000415e1a in _vala_main (args=0x7fffffffd508, args_length1=1) at ../src/main.vala:28
#82 0x0000000000415f5a in main (argc=1, argv=0x7fffffffd508) at ../src/main.vala:18

why not just use the GDK_WINDOWING_X11 macro ?

https://developer.gnome.org/gdk3/stable/gdk3-General.html#GDK-WINDOWING-X11:CAPS

(for example like there : https://github.com/GNOME/gnome-control-center/blob/master/shell/main.c#L42)

Because the example I found had the "ifdef wayland… if …; ifdef X11… if …" pattern in it and I copied it, and C is an awful language and I'm kludging together what I can because the whole thing should just work without me having to mess with this? 😉

From what I read, the extra "IF" is there in case the app is built for cross-platform usage. The ifdef is kept/stripped at compile time, so it makes sure that only the relevant blocks make it into the code (won't try to compile Wayland code when the dev doesn't have and doesn't want Wayland libraries installed) but then makes sure that it only gets called on platforms running X11 even if they can run Wayland as well. But that's just what I read.

Trying to follow this through: we've got to init X11 threads before GStreamer. And we've got to init GDK before X11 threads (so we can get a display). And GTK automatically inits GDK. But we can't rely on initialising GTK for initialising the GDK display that we need, because that makes X11 crash? Wonderful. I guess that means we can't move all the init code after the new Cawbird() call, then!

Re #123: Init GDK to make sure we have a display

Okay, does that fix it?

Yes, it works for me now! Thanks a lot for the fix.

Update: it works when launched from command-line, but crashes when started from the gnome desktop:

Nov  9 22:28:23 bonobo systemd[1275]: Started dbus-:1.2-uk.co.ibboard.cawbird@9.service.
Nov  9 22:28:23 bonobo kernel: cawbird[3720966]: segfault at 1 ip 00007ffff7ec246e sp 00007fffffffdf20 error 4 in libglib-2.0.so.0.6400.6[7ffff7e7e000+86000]
Nov  9 22:28:23 bonobo kernel: Code: 85 c0 75 b8 4d 89 ef 0f 1f 44 00 00 4c 8b 6d a8 e9 31 fa ff ff 0f 1f 80 00 00 00 00 48 8b 45 80 48 85 c0 74 37 4d 85 f6 74 32 <8b> 00 85 c0 74 2c 49 8b 06 48 8b 38 e8 d1 d2 fd ff 49 89 c4 4d 85
Nov  9 22:28:23 bonobo systemd[1]: Started Process Core Dump (PID 3720967/UID 0).
Nov  9 22:28:23 bonobo systemd[1275]: dbus-:1.2-uk.co.ibboard.cawbird@9.service: Succeeded.
Nov  9 22:28:23 bonobo systemd-coredump[3720968]: Process 3720966 (cawbird) of user 1000 dumped core.#012#012Stack trace of thread 3720966:#012#0  0x00007ffff7ec246e g_option_context_parse (libglib-2.0.so.0 + 0x6046e)#012#1  0x00007ffff75beca2 gdk_parse_args (libgdk-3.so.0 + 0x31ca2)#012#2  0x00007ffff75bef3d gdk_init_check (libgdk-3.so.0 + 0x31f3d)#012#3  0x00007ffff75bef5d gdk_init (libgdk-3.so.0 + 0x31f5d)#012#4  0x00005555555ce6ba cb_utils_init_gui (cawbird + 0x7a6ba)#012#5  0x0000555555578b58 _vala_main (cawbird + 0x24b58)#012#6  0x00007ffff69b5042 __libc_start_main (libc.so.6 + 0x27042)#012#7  0x0000555555578a5e _start (cawbird + 0x24a5e)
Nov  9 22:28:23 bonobo systemd[1]: systemd-coredump@131-3720967-0.service: Succeeded.

I fixed the args passing crash with this modification:

-  Cb.Utils.init_gui(args[1:args.length]);
+  Cb.Utils.init_gui(ref args);

and

-    public void init_gui ([CCode (array_length_pos = 0.1)] string[] args);
+    public void init_gui ([CCode (array_length_pos = 0.1)] ref unowned string[] args);

Does it looks okay to you?

Well, it looks like modified code. I don't have a clue what's right in this context, so I can't be certain that it's okay 😄 But if it stops the arguments crashing things then that's good.

[Edit] Also, it does match the GStreamer approach, which is a good sign!

It works for me! Thanks again.

@philipzae

@philipzae can you check if this fixes it for you?

What we've got now follows what Gnome Devs told me and works for other people, so I assume it's okay.

Navigation Menu

Repository navigation

Cawbird crashes on playing video #123

1 remaining item

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Cawbird crashes on playing video #123

Description

Activity

IBBoard commented on Apr 1, 2020

philipzae commented on Apr 1, 2020

philipzae commented on Apr 7, 2020

IBBoard commented on Apr 7, 2020

philipzae commented on Apr 7, 2020

philipzae commented on Apr 9, 2020

IBBoard commented on Apr 9, 2020

philipzae commented on Apr 9, 2020

IBBoard commented on Apr 9, 2020

philipzae commented on Apr 9, 2020

fbellet commented on Nov 8, 2020

IBBoard commented on Nov 8, 2020

IBBoard commented on Nov 8, 2020

1 remaining item

IBBoard commented on Nov 8, 2020

IBBoard commented on Nov 9, 2020

fbellet commented on Nov 9, 2020

fbellet commented on Nov 9, 2020

fbellet commented on Nov 9, 2020

IBBoard commented on Nov 9, 2020

IBBoard commented on Nov 9, 2020

fbellet commented on Nov 9, 2020

fbellet commented on Nov 10, 2020

IBBoard commented on Nov 10, 2020

fbellet commented on Nov 10, 2020

IBBoard commented on Nov 14, 2020

IBBoard commented on Dec 27, 2020

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions