OpenSIPS version you are running
version: opensips 3.4.17 (x86_64/linux)
flags: STATS: On, DISABLE_NAGLE, USE_MCAST, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, HP_MALLOC, DBG_MALLOC, FAST_LOCK-ADAPTIVE_WAIT
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535
poll method support: poll, epoll, sigio_rt, select.
git revision: 3e7ad039d
main.c compiled on with gcc 13
The impacting code is unchanged on the 3.4, 3.5/3.6 and master branches, so don't yell at me for running 3.4....
Crash Core Dump
Can't share the core file... it has live production SIP traffic, privacy laws and all. Below is the sanitized, symbolized backtrace of the crashing a UDP worker, captured from a matching debug-symbol build. Don't think it'll be needed... the issue is pretty clear.
Program terminated with signal SIGABRT, Aborted.
#4 __GI_abort () at ./stdlib/abort.c:79
#5 fm_free (fm=<optimized out>, p=0x7442006cba10, ...) at mem/f_malloc_dyn.h:240
#6 _shm_free (ptr=0x7442006cba10, file="t_reply.c", function="t_on_reply", line=174) at ../../evi/../mem/shm_mem.h:550
#7 t_on_reply (ref=...) at modules/tm/t_reply.c:174
#8 w_t_on_reply (msg=..., go_to=...) at modules/tm/tm.c:1212
#9 do_action (...) at action.c:1048
... run_action_list / do_action / run_top_route <-- script request route
#28 receive_msg (...) at receive.c:230
in_buff = "ACK sip:<redacted>@<redacted> SIP/2.0\r\nVia: SIP/2.0/UDP <redacted>;branch=...;rport\r\n..."
#29 udp_read_req (...) at net/proto_udp/proto_udp.c:186
#30 handle_io (...) at net/net_udp.c:295
The allocator aborts because the pointer being freed is already on the free list:
CRITICAL:core:fm_free: freeing already freed shm pointer (0x...), first free: <garbage: recycled SIP message text> - aborting!
The first free field is overwritten with unrelated recycled SIP text, i.e. the chunk was already freed and reused.
(Line numbers above are from the 3.4.17 build; on current master the same code is t_on_reply at modules/tm/t_reply.c:161 with shm_free() at line 176.)
Describe the traffic that generated the bug
A stateful, in-dialog ACK arriving over UDP that matches an already-established INVITE transaction (loaded via t_check_trans()), where the configuration script calls t_on_reply() (and/or t_on_failure()) while routing that ACK. Because ACKs are not absorbed as retransmissions the way INVITEs are, a retransmitted ACK is re-run through the request route; when two UDP workers process near-simultaneous copies of the same ACK, both execute t_on_reply() against the same shared transaction cell. And yeah, calling t_on_reply() for an ACK probably has no useful purpose, this is still a problematic bug.
Root cause: t_on_reply(), t_on_negative() (script t_on_failure) and t_on_branch() (maybe others?) do an unlocked read-modify-write on a field of the shared transaction cell:
holder = (!t || t==T_UNDEFINED) ? &goto_on_reply
: (route_type==BRANCH_ROUTE ? &TM_BRANCH(t,_tm_branch_index).on_reply : &t->on_reply);
if (*holder)
shm_free( *holder ); /* <-- two workers read the same pointer, both free it */
*holder = ref ? dup_ref_script_route_in_shm(ref, 0) : NULL;When get_t() returns an already-existing transaction (the ACK case), holder points into shared memory. Two processes read the same *holder, and both call shm_free() on it. The second shm_free() triggers the allocator's double-free detector, which calls abort() while holding the global shm allocator lock. On a FAST_LOCK build the orphaned lock then makes every other process spin at 100% CPU; the whole opensips server starts to lock up but with sockets still open and accepting traffic - but nothing is processed) until it is restarted/reboots. Frequently observe a second process aborting on the same pointer moments later, consistent with concurrent access to one shared cell.
This matches the existing comment above these setters in t_reply.c, which documents that the reply route is meant to be staged in private memory before t_relay and consumed inside t_relay and that it "cannot [be set] ... after t_relay when a reply may arrive." Setting it directly on an already-created (shared) transaction from the request route is outside that intended path, and there is no lock protecting the shared field.
To Reproduce
This is a race condition... not easily reproduced. Under load, we were seeing it a problematic amount of times in a production environment.
But in theory:
- A config that, in its request route, routes in-dialog ACKs through logic that calls
t_on_reply()/t_on_failure() on a transaction matched via t_check_trans() (e.g. an RTP-anchoring path shared by INVITE/ACK). This is also our mitigation... guard against calling t_on_reply() for ACKs.
- Drive in-dialog calls over UDP such that ACKs are retransmitted (e.g. delayed/duplicated 2xx → duplicate ACKs), so multiple UDP workers process copies of the same ACK concurrently.
- Under load, a worker aborts in
fm_free (double-free), holding the shm lock; remaining processes spin at 100% CPU and the proxy stops processing traffic until restarted.
Relevant System Logs
CRITICAL:core:fm_free: freeing already freed shm pointer (0x...), first free: <recycled SIP text> - aborting!
CRITICAL:core:sig_usr: segfault in process pid: <pid>, id: <n> # other workers, walking the corrupted free list
WARNING:core:utimer_ticker: utimer task <tm-utimer> already scheduled NNNNN ms ago (now ...), delaying execution # timer starved while all procs spin on the orphaned shm lock
OS/environment information
- Operating System: Ubuntu 24.04 (x86_64)
- OpenSIPS installation: debs (apt.opensips.org)
- other relevant information:
FAST_LOCK-ADAPTIVE_WAIT build; SIP over UDP; carrier-facing proxy with in-dialog ACK handling that calls t_on_reply.
Additional context
Proposed fix: perform the free/reassign of the (possibly shared) route-ref via an atomic swap under the transaction reply lock, allocating/freeing outside the lock to avoid nesting the reply lock around the shm lock. The lock must be taken only in REQUEST_ROUTE, because FAILURE_ROUTE and (with onreply_avp_mode) ONREPLY_ROUTE already run under LOCK_REPLIES and call these handlers — taking the non-recursive reply lock there would deadlock? The same three setters (t_on_reply, t_on_negative, t_on_branch) should share the fix. Seems like there have been a number of double-free errors against shm fixed recently... TLS had one for 3.4.18, #3767, etc. Maybe this is more systemic than I'm aware... Or maybe a lock-free fix is to skip the update when the transaction already exists in REQUEST_ROUTE, making setting on_reply on an already-created transaction a no-op?
And to note - guarding against calling t_on_reply() against ACKs did mitigate the issue.
Something to note - the orphaned lock was observed causing the attendant process to throw a GPF... which left opensips in an extra sour state. Not any more or less recoverable and I didn't run it down... tm is the root cause.
/usr/sbin/opensips[196362]: CRITICAL:core:sig_usr: segfault in attendant (starter) process!
kernel: traps: opensips[196362] general protection fault ip:55b75ed79816 sp:7ffd0b264e70 error:0 in opensips[118816,55b75ec80000+205000]
I'll post a draft PR with a proposed fix...
OpenSIPS version you are running
The impacting code is unchanged on the
3.4,3.5/3.6andmasterbranches, so don't yell at me for running 3.4....Crash Core Dump
Can't share the core file... it has live production SIP traffic, privacy laws and all. Below is the sanitized, symbolized backtrace of the crashing a UDP worker, captured from a matching debug-symbol build. Don't think it'll be needed... the issue is pretty clear.
The allocator aborts because the pointer being freed is already on the free list:
The
first freefield is overwritten with unrelated recycled SIP text, i.e. the chunk was already freed and reused.(Line numbers above are from the 3.4.17 build; on current
masterthe same code ist_on_replyatmodules/tm/t_reply.c:161withshm_free()at line 176.)Describe the traffic that generated the bug
A stateful, in-dialog ACK arriving over UDP that matches an already-established INVITE transaction (loaded via
t_check_trans()), where the configuration script callst_on_reply()(and/ort_on_failure()) while routing that ACK. Because ACKs are not absorbed as retransmissions the way INVITEs are, a retransmitted ACK is re-run through the request route; when two UDP workers process near-simultaneous copies of the same ACK, both executet_on_reply()against the same shared transaction cell. And yeah, callingt_on_reply()for an ACK probably has no useful purpose, this is still a problematic bug.Root cause:
t_on_reply(),t_on_negative()(scriptt_on_failure) andt_on_branch()(maybe others?) do an unlocked read-modify-write on a field of the shared transaction cell:When
get_t()returns an already-existing transaction (the ACK case),holderpoints into shared memory. Two processes read the same*holder, and both callshm_free()on it. The secondshm_free()triggers the allocator's double-free detector, which callsabort()while holding the global shm allocator lock. On aFAST_LOCKbuild the orphaned lock then makes every other process spin at 100% CPU; the whole opensips server starts to lock up but with sockets still open and accepting traffic - but nothing is processed) until it is restarted/reboots. Frequently observe a second process aborting on the same pointer moments later, consistent with concurrent access to one shared cell.This matches the existing comment above these setters in
t_reply.c, which documents that the reply route is meant to be staged in private memory beforet_relayand consumed insidet_relayand that it "cannot [be set] ... after t_relay when a reply may arrive." Setting it directly on an already-created (shared) transaction from the request route is outside that intended path, and there is no lock protecting the shared field.To Reproduce
This is a race condition... not easily reproduced. Under load, we were seeing it a problematic amount of times in a production environment.
But in theory:
t_on_reply()/t_on_failure()on a transaction matched viat_check_trans()(e.g. an RTP-anchoring path shared by INVITE/ACK). This is also our mitigation... guard against callingt_on_reply()for ACKs.fm_free(double-free), holding the shm lock; remaining processes spin at 100% CPU and the proxy stops processing traffic until restarted.Relevant System Logs
OS/environment information
FAST_LOCK-ADAPTIVE_WAITbuild; SIP over UDP; carrier-facing proxy with in-dialog ACK handling that callst_on_reply.Additional context
Proposed fix: perform the free/reassign of the (possibly shared) route-ref via an atomic swap under the transaction reply lock, allocating/freeing outside the lock to avoid nesting the reply lock around the shm lock. The lock must be taken only in
REQUEST_ROUTE, becauseFAILURE_ROUTEand (withonreply_avp_mode)ONREPLY_ROUTEalready run underLOCK_REPLIESand call these handlers — taking the non-recursive reply lock there would deadlock? The same three setters (t_on_reply,t_on_negative,t_on_branch) should share the fix. Seems like there have been a number of double-free errors against shm fixed recently... TLS had one for 3.4.18, #3767, etc. Maybe this is more systemic than I'm aware... Or maybe a lock-free fix is to skip the update when the transaction already exists inREQUEST_ROUTE, making settingon_replyon an already-created transaction a no-op?And to note - guarding against calling
t_on_reply()against ACKs did mitigate the issue.Something to note - the orphaned lock was observed causing the attendant process to throw a GPF... which left opensips in an extra sour state. Not any more or less recoverable and I didn't run it down... tm is the root cause.
I'll post a draft PR with a proposed fix...