Queue index file cannot be recovered if it has messages with TTL on 3.8.17+

[annieblomgren]

Hi,

We've had two different clusters upgrading to 3.8.19 where the message store got corrupted:

1. First case is a single node with Erlang 23.3.1.
   After the upgrades a queue got into the state 'down'.

2. Second case is a 3 node cluster with Erlang 24.0.2.
   A whole vhost was down.
   We used strace to find the "faulty" .idx file and removed it, after that the vhost could start so not all messages were lost.

We can share full logs and if of interest the data folder privately.

[johanrhodin]

More here: #3253 (comment)
and in this rabbitmq-users post: https://groups.google.com/g/rabbitmq-users/c/ekV9tTBRZms/m/kzZKsITSAwAJ

[annieblomgren]

It has happened again to a single node cluster running Erlang 24.0.4.
A common thing for all 3 cases so far is that the faulty queue has used TTL.

[carlhoerberg]

We at CloudAMQP have stopped provisioning 3.8.19 because so many customers gets corrupted messages stores from this version ⚠️

[michaelklishin]

Most recent queue index changes were #2954 and #3041 in 3.8.17.

[michaelklishin]

#2954 seems to be a lot more relevant than #3041. As of #2954, index files by default store 2048 entries instead of 16384. This value
is stored in the virtual host directory as .config. Existing installations should have the original value stored there. New installations will
use rabbit.queue_index_segment_entry_count or 2048 if it's not set. Setting rabbit.queue_index_segment_entry_count to 16384 should effectively undo #2954 for newly created virtual hosts.

Nodes will log

Setting segment_entry_count for vhost … with … queues to …

on boot and the .config file is a single Erlang term right now, and can easily be inspected.

Without knowing if this was an upgrade and from what version, we can't tell what segment entry count value should be in effect. But you
can inspect those values both using rabbitmq-diagnostics environment and by inspecting the dot files files under the virtual host directory (which would be under the message store root).

[michaelklishin]

Worth mentioning that #3068 was a follow-up PR that makes us back to the original segment count value when virtual host config file does not exist.

[annieblomgren]

Have been able to reproduce it in versions 3.8.19, 3.8.18 and 3.8.17.
It works in 3.8.16.

To reproduce:

1. perftest command bin/runjava com.rabbitmq.perf.PerfTest -x 1 -y 0 -u "throughput-test-1" -a --id "test 1" --message-properties expiration=1 -H $URL -f persistent -s 8000
2. Restart RabbitMQ

Setting segment_entry_count for vhost 'X' with 0 queues to '2048'

[michaelklishin]

@annieblomgren thanks for the details, so this is with a brand new node and not an upgraded one?

[michaelklishin]

so this also includes per-message TTL, which makes me wonder if #3041 can be just as relevant.

[annieblomgren]

@michaelklishin yes tested on brand new nodes too

[michaelklishin]

We could reproduce and so far it seems that #3041 is the root cause. Reverting it seems to reliably make the issue go away.

[michaelklishin]

@annieblomgren can you please try this alpha build? https://github.com/rabbitmq/rabbitmq-server-binaries-dev/releases/tag/v3.8.21-alpha.2

[michaelklishin]

We are ready to produce 3.8.21 but it would be nice to make sure that we are not the only ones who see a behavior change after e63d89a

[johanrhodin]

We'll take it for a spin during the next 24h

[johanrhodin]

I haven't been able to reproduce with RabbitMQ 3.8.21-alpha.2 (was able with 3.8.18).
Have a long running job doing this over and over for a few hours running now.

[annieblomgren]

yeah @michaelklishin can't reproduce it on v3.8.21-alpha.2 👍
thank you for the fast response.

[lhoguin]

Thanks for the assist!

[michaelklishin]

3.9.3 and 3.8.21 are released.