Asterisk 22.10.1 / ASL3 3.10.5 — repeatable app_rpt.so SIGSEGV in __ao2_ref

K6IAN · September 17, 2026, 12:40pm

After upgrading my ASL3 node to Asterisk 22.10.1 / ASL3 3.10.5, I began experiencing repeated Asterisk SIGSEGV crashes. I have retained systemd core dumps from the crashes.

System:

Debian 12 (Bookworm), x86_64
ASL3 node: 525240
asl3: 3.18.2-2.deb12
Problem version:
- asl3-asterisk: 2:22.10.1+asl3-3.10.5-1.deb12
- asl3-asterisk-modules: 2:22.10.1+asl3-3.10.5-1.deb12
- asl3-asterisk-config: 2:22.10.1+asl3-3.10.5-1.deb12
rxchannel = Voter/525240
duplex = 0

The first crashes I observed occurred after the system upgraded to 22.10.1/3.10.5. On September 16 I captured two separate crashes approximately 22 minutes apart.

Both crashes produced the exact same module offsets, despite ASLR changing the absolute addresses:

#0 __ao2_ref (asterisk + 0x70ee1)

#1 app_rpt.so + 0x2749b

#2 app_rpt.so + 0x1cf4b

#3 app_rpt.so + 0x3ca61

#4 asterisk + 0x1e212d

First retained crash:

Wed 2026-09-16 22:02:04 PDT

PID 1011578

SIGSEGV

#0 __ao2_ref asterisk + 0x70ee1

#1 app_rpt.so + 0x2749b

#2 app_rpt.so + 0x1cf4b

#3 app_rpt.so + 0x3ca61

Second retained crash:

Wed 2026-09-16 22:24:05 PDT

PID 1042410

SIGSEGV

#0 __ao2_ref asterisk + 0x70ee1

#1 app_rpt.so + 0x2749b

#2 app_rpt.so + 0x1cf4b

#3 app_rpt.so + 0x3ca61

The kernel fault for the second crash was:

segfault at fffffffffffffffd

ip 0000560930d4cee1

error 5 in asterisk

The earlier crashes I examined showed the same fffffffffffffffd fault pattern.

I initially investigated a custom connpgm script because it runs when nodes connect and also runs once per minute from cron. I added flock protection to prevent simultaneous executions and added launch/finish logging.

However, the 22:24 crash occurred with the flock protection already active. Immediately before the crash:

22:24:01 killmultinodes: LAUNCH

22:24:02 killmultinodes: FINISH status=0

22:24:03 Asterisk SIGSEGV

That execution did not issue a disconnect, strike, violation, or other Asterisk command. Therefore I don’t currently have evidence that the external script directly caused this crash.

I have now performed a controlled downgrade of the complete matched Asterisk package set to:

asl3-asterisk 2:22.9.0+asl3-3.9.3-1.deb12

asl3-asterisk-modules 2:22.9.0+asl3-3.9.3-1.deb12

asl3-asterisk-config 2:22.9.0+asl3-3.9.3-1.deb12

Asterisk is currently running normally on 22.9.0/3.9.3 with the same node configuration. I’ve temporarily held these packages so the system doesn’t automatically upgrade back to 3.10.5 while I test.

I still have both 3.10.5 core dumps present under systemd-coredump. The crashing thread in both dumps is identical at the module-offset level.

Questions for the developers:

Does app_rpt.so + 0x2749b → +0x1cf4b → +0x3ca61 → __ao2_ref correspond to a known issue in 3.10.5?
Are debug-symbol packages available that would allow me to provide a fully symbolized backtrace?
Would the developers like either/both core dumps or additional coredumpctl/journal output?

I can also reproduce/provide additional logging if there are specific diagnostics you’d like enabled.

W5GLE · September 17, 2026, 2:54pm

Core dumps, and especially backtraces, are much more informative when the debugging symbols are available.

Can you please install the asl3-asterisk-dbgsym and asl3-asterisk-modules-dbgsym packages on your system.

Then, if you can repro the problem, you can either use the /var/lib/asterisk/scripts/ast_coredumper command to analyze a generated core file (generally found in the /var/lib/asterisk directory) or with gdb you can exec the thread apply all backtrace command.

Also, any details on how to repro the problem/crash would be very helpful (e.g. "use this conf file", "exec this command", ...)

Issues · AllStarLink/app_rpt · GitHub create a new issue if you do not find one that matches your issue.

K6IAN · September 17, 2026, 3:33pm

I was able to do additional analysis on both retained core dumps using the exact matching Asterisk/app_rpt binaries and debug symbols from the version that generated the crashes.

The production node has already been rolled back, so I did not reinstall the problem version to obtain the symbols. Instead, I downloaded and extracted the matching 3.10.5 packages and dbgsym packages separately and analyzed the retained cores against those binaries.

Version that generated both cores:

asl3-asterisk 2:22.10.1+asl3-3.10.5-1.deb12

asl3-asterisk-config 2:22.10.1+asl3-3.10.5-1.deb12

asl3-asterisk-modules 2:22.10.1+asl3-3.10.5-1.deb12

The matching app_rpt.so Build ID is:

49209e6f5cad860a750a3544aff85accab745f94

I verified that the corresponding dbgsym file has the same Build ID and contains debug information.

Crash #1 — 2026-09-16 22:02 PDT

Core PID: 1011578

The crashing thread resolves to:

#0 __ao2_ref()

#1 rpt_conf_get_muted()

#2 process_link_channel()

#3 rpt_link_connect()

Specifically:

#0 0x0000558d866b4ee1 in __ao2_ref()

#1 0x00007fd12b62749b in rpt_conf_get_muted()

#2 0x00007fd12b61cf4b in process_link_channel()

#3 0x00007fd12b63ca61 in rpt_link_connect()

Crash #2 — 2026-09-16 22:24 PDT

Core PID: 1042410

The crashing thread resolves to the same path:

#0 __ao2_ref()

#1 rpt_conf_get_muted()

#2 process_link_channel()

#3 rpt_link_connect()

Specifically:

#0 0x0000560930d4cee1 in __ao2_ref()

#1 0x00007fc539e2749b in rpt_conf_get_muted()

#2 0x00007fc539e1cf4b in process_link_channel()

#3 0x00007fc539e3ca61 in rpt_link_connect()

Although the absolute addresses differ because of ASLR, the app_rpt.so offsets are identical in both independent cores:

rpt_conf_get_muted app_rpt.so + 0x2749b

process_link_channel app_rpt.so + 0x1cf4b

rpt_link_connect app_rpt.so + 0x3ca61

Using addr2line against the exact matching 3.10.5 app_rpt.so debug symbols resolves those addresses to:

rpt_conf_get_muted

./apps/app_rpt/rpt_bridging.c:437

process_link_channel

./apps/./apps/app_rpt.c:4838

rpt_link_connect

./apps/app_rpt/rpt_link.c:943

So both retained crashes independently show the same path:

rpt_link_connect()

-> process_link_channel()

    -> rpt_conf_get_muted()

        -> \__ao2_ref()

            -> SIGSEGV

The kernel reported the fault address as:

fffffffffffffffd

I don’t currently have a deterministic reproduction procedure. The crashes occurred during normal node/link activity.

I also investigated an external multi-link monitoring script because it interacts with link state. I added flock protection to prevent overlapping executions and instrumented its activity. A controlled multi-link violation/disconnect test did not reproduce the Asterisk crash.

For the 22:24 crash specifically, the monitoring script launched at approximately 22:24:01 and completed normally at 22:24:02 with status 0. That invocation logged no strike, violation, or disconnect. The Asterisk segfault occurred immediately afterward at approximately 22:24:03.

There were also previous periods where Asterisk crashed repeatedly while the monitoring script continued its normal one-minute schedule, including crashes occurring between script executions. So I have not been able to establish the script as the direct trigger.

The node configuration relevant to app_rpt is:

Node: 525240

rxchannel = Voter/525240

duplex = 0

linktolink = yes

Production has now been rolled back to:

asl3-asterisk 2:22.9.0+asl3-3.9.3-1.deb12

asl3-asterisk-config 2:22.9.0+asl3-3.9.3-1.deb12

asl3-asterisk-modules 2:22.9.0+asl3-3.9.3-1.deb12

Those packages are currently held while I observe stability. I don’t consider the upgrade the proven cause at this point; the rollback is being used as an A/B comparison.

Both original 3.10.5 core dumps have been retained. I also have the complete thread apply all backtrace full output for both cores and can provide either the full backtraces or the core files if they would be useful.

If there is a specific variable/object around rpt_conf_get_muted() at rpt_bridging.c:437 that would be useful to inspect from the retained cores, let me know what GDB commands you would like run and I can provide that output as well.

W5GLE · September 17, 2026, 3:47pm

app_rpt: fix __ao2_ref SIGSEGV in mute detection (#1248) by hardenedpenguin · Pull Request #1251 · AllStarLink/app_rpt · GitHub Will resolve your issue, you wont see it until a 3.10.6 is released. It still has to work its way thew the review/commit process.