rtsp: AV1 ingested over RTMP is undecodable over RTSP since v1.16.0 (works over HLS) #6001

Which version are you using?

v1.19.3 (regression bisected to v1.16.0; last good is v1.15.6)

Which operating system are you using?

Linux amd64 standard

Describe the issue

When AV1 is ingested over RTMP and read back over RTSP, the RTSP output is undecodable. FFmpeg decodes 0 frames and reports:

[rtsp] Unexpected fragment continuation in AV1 RTP packet
[rtsp] Forbidden bit set in AV1 OBU header (0x80)
[rtsp] AV1 OBU_TILE_LIST (should not be there!) to be ignored but is fragmented

ffprobe on the RTSP URL returns av1 with profile=unknown, width=0, height=0.

Three observations that narrow it down:

1. It is a regression. The exact same publish + read commands work on v1.15.6 (346 frames decoded, av1,Main,640,480) and fail on v1.16.0, v1.16.2, v1.17.1, v1.18.2 and v1.19.3 — all with byte-identical symptoms. v1.15.6 is the release immediately preceding v1.16.0, so the change landed in v1.16.0.

2. Only the RTP output is affected. From the same RTMP publish on v1.19.3, reading over HLS decodes fine (576 frames) while RTSP decodes 0. So ingest and internal handling of AV1 are fine; what produces unusable output is the RTP packetization.

3. It is not about fragmentation size. It also fails with a tiny stream (160x120 @ 60 kbps) where no OBU needs to be split across packets.

It is not specific to FFmpeg as a publisher either — it reproduces identically with a completely unrelated publisher (an Android app with its own RTMP implementation). Conversely, publishing to v1.19.3 over RTSP (so MediaMTX forwards RTP it received rather than building it) works fine, 482 frames decoded.

As a possible pointer: the v1.15.6...v1.16.0 diff removes the whole internal/codecprocessor/ package, including internal/codecprocessor/av1.go (137 lines) and av1_test.go, as part of "use the same filtering process for every codec" (#5324), which the release notes describe as applying features like RTP packet resizing generically to every codec. I have not verified that this is the cause.

Describe how to replicate the issue

1. Start MediaMTX v1.16.0 or newer with the default configuration.

2. Publish AV1 over RTMP:

ffmpeg -re -f lavfi -i "testsrc=size=640x480:rate=24" -t 35 \
  -c:v libsvtav1 -preset 12 -g 48 -b:v 1200k \
  -f flv rtmp://127.0.0.1:1935/ctrl

3. Read it back over RTSP — 0 frames decoded:

ffmpeg -rtsp_transport tcp -i rtsp://127.0.0.1:8554/ctrl -map 0:v -f null -

4. For contrast, read the same stream over HLS — decodes fine:

ffmpeg -i http://127.0.0.1:8888/ctrl/index.m3u8 -map 0:v -f null -

5. Repeat step 2-3 against v1.15.6: RTSP decodes normally.

Tested with ffmpeg 8.0.1 and SVT-AV1 v2.3.0 in every run, so the client side is constant across all versions.

MediaMTX configuration

Default configuration. The runs above only relocated ports so two versions could run side by side, and disabled the servers that were not under test:

# equivalent to the defaults; set via env vars in my runs
rtspAddress: :8654
rtmpAddress: :2935
rtpAddress: :8100
rtcpAddress: :8101
webrtc: no
srt: no
api: no
logLevel: debug
dumpPackets: yes

The failure also reproduces with a completely untouched mediamtx.yml on the standard ports.

MediaMTX logs

2026/07/28 17:58:36 INF MediaMTX v1.19.3, linux, amd64
2026/07/28 17:58:36 INF [RTSP] started with listeners on :8654 (TCP/RTSP), :8100 (UDP/RTP), :8101 (UDP/RTCP)
2026/07/28 17:58:36 INF [RTMP] started with listener on :2935 (TCP/RTMP)
2026/07/28 17:58:36 INF [RTMP] [conn 127.0.0.1:55108] opened
2026/07/28 17:58:40 INF [path ctrl] stream is available and online, 1 track (AV1)
2026/07/28 17:58:40 INF [RTMP] [conn 127.0.0.1:55108] is publishing to path 'ctrl'
2026/07/28 17:58:44 INF [RTSP] [conn 127.0.0.1:39904] opened
2026/07/28 17:58:44 INF [RTSP] [session bd33e963] created by 127.0.0.1:39904
2026/07/28 17:58:44 INF [RTSP] [session bd33e963] is reading from path 'ctrl', with TCP, 1 track (AV1)

The server logs no error of its own; only the reader fails.

Packet dump

I generated .pcapng files with dumpPackets: yes and can attach them — but since the repro above is deterministic with default config and pure ffmpeg, they may not be needed. Decoding the RTSP dump, these are the AV1 aggregation headers MediaMTX v1.19.3 emits (byte after the RTP header), for reference:

seq=14202 M=1 len=  17 agg=0x20 Z=0 Y=0 W=2 N=0
seq=14203 M=1 len= 315 agg=0x20 Z=0 Y=0 W=2 N=0
seq=14204 M=0 len=1452 agg=0x78 Z=0 Y=1 W=3 N=1
seq=14205 M=0 len=1452 agg=0xd0 Z=1 Y=1 W=1 N=0
...
seq=14210 M=1 len= 951 agg=0x90 Z=1 Y=0 W=1 N=0
seq=14211 M=0 len=1452 agg=0x40 Z=0 Y=1 W=0 N=0   <- W switches to 0 here
seq=14212 M=0 len=1452 agg=0xc0 Z=1 Y=1 W=0 N=0
seq=14213 M=0 len=1452 agg=0xc0 Z=1 Y=1 W=0 N=0

The Z/Y chain itself is self-consistent and the length fields add up to the payload size, so I could not pin down the malformation by inspection — I am only reporting the symptom and the bisection. Note that W alternates between 0 and non-zero across temporal units within the same stream; I do not know whether that is relevant.

Happy to attach the pcapng files or run further tests on any version if useful.