Summary
On an OpenSIPS SBC/proxy, a stray trailing NUL byte (0x00) is occasionally appended after a proxied 2xx response written to a reused inbound TCP connection. On the TCP byte stream that 0x00 becomes the leading byte before the next SIP message, so a strict RFC 3261 TCP parser at the UAC rejects the following message:
failed to parse: transmission beginning '<0x00>SIP/2.0 200 OK' is not a SIP message
The 200 OK itself is well-formed; only the extra leading 0x00 (which is the trailing byte of the previously-written message on the stream) breaks strict parsers. Lenient parsers that skip leading whitespace/garbage are unaffected, which is likely why this has gone largely unnoticed. Consequence with a strict UAC: it never sees the answer, never ACKs → dead air → CANCEL/481.
Version
opensips 3.5.9 (aarch64/linux) (flags include DISABLE_NAGLE). Confirmed still present in the 3.6 branch by code inspection — the TCP write/relay path (proto_tcp_send, tcp_write_on_socket, tcp_async_add_chunk, tsend_stream_async, tcp_async_write) is functionally unchanged between 3.5.9 and 3.6 tip.
Topology / when it happens
- UAC connects to OpenSIPS over TCP and sends an INVITE.
- OpenSIPS proxies (
t_relay) the call to a downstream leg over UDP (a carrier/gateway).
- The downstream's provisional/final responses are relayed back to the UAC over the same, reused TCP connection.
- The
0x00 appears only on relayed/modified 2xx responses (200-to-INVITE, 200-to-BYE) written to a reused/established TCP connection.
Discriminators observed (packet capture + reassembled TCP stream on the receiving side):
- Locally-generated replies (100 Trying, 503, etc.) are always clean.
- The first response on a freshly-opened TCP connection tends to be clean; the stray byte shows up on later / reused writes.
- It is timing-sensitive / probabilistic (a few per handful of calls) and disappears under
strace on the OpenSIPS workers — consistent with a race or a state-dependent buffer issue rather than a deterministic len+1.
- Because the carrier leg is UDP and the client leg is TCP, the relayed reply is written by a different process than the one that owns/reads the TCP connection (i.e. the cross-process "hand the buffer to the connection owner" path), whereas clean local replies are written in-process.
Ruled out during investigation
modparam("proto_tcp", "tcp_async", 0) — the NUL persists with synchronous writes, so it is not simply the async-chunk path.- CRLF ping/pong (
tcp_crlf_pingpong) — the pong writes CRLF (2 bytes), not a NUL; and the byte appears with no keepalive traffic present.
- Reply build length —
build_res_buf_from_sip_res() returns new_len (the +1 alloc is only the debug NUL and is excluded from returned_len); SEND_PR_BUFFER/send_pr_buffer/proto_tcp_send/tcp_write_on_socket all pass the correct len. So the byte is not an obvious len+1 in the send call.
- Event modules (e.g.
event_kafka) that fire per message — these serialize to a separate evi_build_payload buffer and do not touch the SIP reply buffer.
Reproduction notes
It reproduces reliably on a full production SBC (complete module stack + real network + load). It did not reproduce in an isolated minimal t_relay lab (core + tm/sl/signaling, and again with rr/dialog/topology_hiding), including: fast relay, added ring delay, forced OpenSIPS-side partial writes (tiny client SO_RCVBUF + delayed reads), and a real cross-host VPC client. That environment-sensitivity is itself a clue that this is a state/timing-dependent buffer issue on the cross-process TCP write of a proxied reply.
Why I suspect a NUL-inclusive length somewhere on the per-message path
The symptom is textbook "one byte too long / trailing C-string terminator." The exact dst->len++-after-NUL-terminate anti-pattern was recently fixed in event_rabbitmq (#3828 / #3834). I could not locate the analogous instance on the SIP TCP reply/relay path, but the shape of the corruption is identical. Any pointers to where a proxied reply's buffer could pick up its terminator on the cross-process/reused-connection write path would be very helpful.
Ask
- Guidance on where the stray trailing
0x00 could originate in the cross-process write of a relayed reply (worker → connection-owning process) on a reused TCP connection.
- Whether maintainers can reproduce with a strict-parsing TCP UAC + a UDP downstream leg under load.
Happy to provide packet captures / additional detail.
Summary
On an OpenSIPS SBC/proxy, a stray trailing NUL byte (
0x00) is occasionally appended after a proxied 2xx response written to a reused inbound TCP connection. On the TCP byte stream that0x00becomes the leading byte before the next SIP message, so a strict RFC 3261 TCP parser at the UAC rejects the following message:The 200 OK itself is well-formed; only the extra leading
0x00(which is the trailing byte of the previously-written message on the stream) breaks strict parsers. Lenient parsers that skip leading whitespace/garbage are unaffected, which is likely why this has gone largely unnoticed. Consequence with a strict UAC: it never sees the answer, never ACKs → dead air → CANCEL/481.Version
opensips 3.5.9 (aarch64/linux)(flags includeDISABLE_NAGLE). Confirmed still present in the3.6branch by code inspection — the TCP write/relay path (proto_tcp_send,tcp_write_on_socket,tcp_async_add_chunk,tsend_stream_async,tcp_async_write) is functionally unchanged between 3.5.9 and 3.6 tip.Topology / when it happens
t_relay) the call to a downstream leg over UDP (a carrier/gateway).0x00appears only on relayed/modified 2xx responses (200-to-INVITE, 200-to-BYE) written to a reused/established TCP connection.Discriminators observed (packet capture + reassembled TCP stream on the receiving side):
straceon the OpenSIPS workers — consistent with a race or a state-dependent buffer issue rather than a deterministiclen+1.Ruled out during investigation
modparam("proto_tcp", "tcp_async", 0)— the NUL persists with synchronous writes, so it is not simply the async-chunk path.tcp_crlf_pingpong) — the pong writesCRLF(2 bytes), not a NUL; and the byte appears with no keepalive traffic present.build_res_buf_from_sip_res()returnsnew_len(the+1alloc is only the debug NUL and is excluded fromreturned_len);SEND_PR_BUFFER/send_pr_buffer/proto_tcp_send/tcp_write_on_socketall pass the correctlen. So the byte is not an obviouslen+1in the send call.event_kafka) that fire per message — these serialize to a separateevi_build_payloadbuffer and do not touch the SIP reply buffer.Reproduction notes
It reproduces reliably on a full production SBC (complete module stack + real network + load). It did not reproduce in an isolated minimal
t_relaylab (core + tm/sl/signaling, and again with rr/dialog/topology_hiding), including: fast relay, added ring delay, forced OpenSIPS-side partial writes (tiny clientSO_RCVBUF+ delayed reads), and a real cross-host VPC client. That environment-sensitivity is itself a clue that this is a state/timing-dependent buffer issue on the cross-process TCP write of a proxied reply.Why I suspect a NUL-inclusive length somewhere on the per-message path
The symptom is textbook "one byte too long / trailing C-string terminator." The exact
dst->len++-after-NUL-terminate anti-pattern was recently fixed inevent_rabbitmq(#3828 / #3834). I could not locate the analogous instance on the SIP TCP reply/relay path, but the shape of the corruption is identical. Any pointers to where a proxied reply's buffer could pick up its terminator on the cross-process/reused-connection write path would be very helpful.Ask
0x00could originate in the cross-process write of a relayed reply (worker → connection-owning process) on a reused TCP connection.Happy to provide packet captures / additional detail.