feat(pullsync): delivery batching performance analysis

[janos]

Pullsync Batching Performance Analysis

This PR gives the overview of the performance testing and profiling conducted to evaluate the proposed batched chunk delivery (DeliveryBatch) against the legacy single-message delivery (v2.8.1) in the pullsync protocol. It should not be merged, as it is meant to reproduce the results, but it may be used for potential improvements.

The goal is to identify if the batching of delivery messages in the pullsync protocol can result to a speedup in overall deliveries.

To simulate real-world networking overhead accurately, a benchmark was written that spins up real libp2p nodes locally, multiplexes streams, and transmits chunks between a client and server.

This approach gives a convenient testing, benchmarking and profiling, but the pullsync performance can be measured on the real network as well.

There are two benchmarks:

• BenchmarkDeliveryLibP2P - that uses the complete libp2p stack with real networking
• BenchmarkDeliveryPipe - that just pipes the messages with buffers (no actual networking) which is not the best for real world network conditions but is good for algorithmic changes.

Benchmark

There are to benchmarks, one for he currently released bee v2.8.1 pullsync implementation and one for the proposed batched implementation. Both of them with 25 and 120 chunks delivered. Since the max batch size can be 25 chunks because of the limitation of 128KB payload size of protobuf. There is a possiblity to raise that limit, but with a danger to expose some vulnerability in the network.

Run the libp2p benchmark for 100 iterations tracking memory allocations:

go test -v -run=NONE -bench=BenchmarkDeliveryLibP2P -benchtime=100x -benchmem ./pkg/pullsync/delivery_test.go ./pkg/pullsync/pullsync_test.go

goos: darwin
goarch: arm64
cpu: Apple M1 Pro
BenchmarkDeliveryLibP2P
BenchmarkDeliveryLibP2P/n=25
BenchmarkDeliveryLibP2P/n=25/Batched
BenchmarkDeliveryLibP2P/n=25/Batched-10 	     100	  99077116 ns/op	  601238 B/op	    5915 allocs/op
BenchmarkDeliveryLibP2P/n=25/v2.8.1
BenchmarkDeliveryLibP2P/n=25/v2.8.1-10  	     100	  99083815 ns/op	  507315 B/op	    6076 allocs/op
BenchmarkDeliveryLibP2P/n=120
BenchmarkDeliveryLibP2P/n=120/Batched
BenchmarkDeliveryLibP2P/n=120/Batched-10         	     100	 475344458 ns/op	 2241125 B/op	   27103 allocs/op
BenchmarkDeliveryLibP2P/n=120/v2.8.1
BenchmarkDeliveryLibP2P/n=120/v2.8.1-10          	     100	 475295048 ns/op	 1690773 B/op	   27764 allocs/op
PASS
ok  	command-line-arguments	115.854s

Profiling

Ran Go's built-in pprof tools against both the Batched and v2.8.1 branches of the benchmark independently, to identify any bottlenecks and/or improvements.

Generate CPU and Memory profiles for the Batched implementation:

go test -run=NONE -bench='BenchmarkDeliveryLibP2P/n=120/Batched$' -benchtime=50x -cpuprofile=cpu_batched.prof -memprofile=mem_batched.prof ./pkg/pullsync/delivery_test.go ./pkg/pullsync/pullsync_test.go

Generate CPU and Memory profiles for the legacy v2.8.1 implementation:

go test -run=NONE -bench='BenchmarkDeliveryLibP2P/n=120/v2.8.1$' -benchtime=50x -cpuprofile=cpu_v281.prof -memprofile=mem_v281.prof ./pkg/pullsync/delivery_test.go ./pkg/pullsync/pullsync_test.go

Analyze:

go tool pprof -top cpu_batched.prof | head -n 15
go tool pprof -top cpu_v281.prof | head -n 15
go tool pprof -top mem_batched.prof | head -n 15
go tool pprof -top mem_v281.prof | head -n 15

CPU Profile

Both profiles reveal an identical bottleneck. In both implementations, runtime.pthread_cond_wait, runtime.usleep, and runtime.pthread_cond_signal consume the majority of all execution samples. This indicates the CPU is blocking on libp2p network I/O synchronization than performing protobuf serialization or algorithmic work.

Batched:

      flat  flat%   sum%        cum   cum%
     0.58s 42.96% 42.96%      0.58s 42.96%  runtime.pthread_cond_wait
     0.24s 17.78% 60.74%      0.24s 17.78%  runtime.usleep
     0.22s 16.30% 77.04%      0.22s 16.30%  runtime.pthread_cond_signal
     0.08s  5.93% 82.96%      0.08s  5.93%  syscall.rawsyscalln
     0.05s  3.70% 86.67%      0.05s  3.70%  golang.org/x/crypto/sha3.keccakF1600
     0.05s  3.70% 90.37%      0.05s  3.70%  runtime.kevent

v2.8.1 (Single Message):

      flat  flat%   sum%        cum   cum%
     0.54s 35.06% 35.06%      0.54s 35.06%  runtime.pthread_cond_wait
     0.25s 16.23% 51.30%      0.25s 16.23%  runtime.pthread_cond_signal
     0.20s 12.99% 64.29%      0.20s 12.99%  runtime.usleep
     0.10s  6.49% 70.78%      0.10s  6.49%  syscall.rawsyscalln
     0.09s  5.84% 76.62%      0.09s  5.84%  runtime.memclrNoHeapPointers
     0.08s  5.19% 81.82%      0.08s  5.19%  runtime.kevent
     0.06s  3.90% 85.71%      0.06s  3.90%  golang.org/x/crypto/sha3.keccakF1600

Memory Profile

Batched:

      flat  flat%   sum%        cum   cum%
   27.63MB 22.44% 22.44%    27.63MB 22.44%  github.com/ethersphere/bee/v2/pkg/pullsync/pb.(*Delivery).Unmarshal
   26.50MB 21.53% 43.97%    26.50MB 21.53%  golang.org/x/crypto/sha3.(*state).Sum
   17.63MB 14.32% 58.29%    20.23MB 16.44%  github.com/gogo/protobuf/io.(*varintWriter).WriteMsg
       5MB  4.06% 62.35%        5MB  4.06%  golang.org/x/crypto/sha3.NewLegacyKeccak256 (inline)
    4.42MB  3.59% 65.94%    32.55MB 26.44%  github.com/gogo/protobuf/io.(*varintReader).ReadMsg
    4.23MB  3.43% 69.38%     4.73MB  3.84%  github.com/libp2p/go-buffer-pool.(*BufferPool).Get

v2.8.1 (Single Message):

      flat  flat%   sum%        cum   cum%
   36.16MB 30.47% 30.47%    36.16MB 30.47%  github.com/ethersphere/bee/v2/pkg/pullsync/pb.(*Delivery).Unmarshal
   28.50MB 24.01% 54.48%    28.50MB 24.01%  golang.org/x/crypto/sha3.(*state).Sum
       7MB  5.90% 60.38%        7MB  5.90%  github.com/ethersphere/bee/v2/pkg/bmt.(*goroutineHasher).Write
    4.50MB  3.79% 64.17%    20.01MB 16.85%  github.com/ethersphere/bee-old/pkg/pullsync.(*Syncer).Sync
    4.50MB  3.79% 67.96%     4.50MB  3.79%  golang.org/x/crypto/sha3.NewLegacyKeccak256 (inline)
       3MB  2.53% 70.49%        3MB  2.53%  github.com/ethersphere/bee/v2/pkg/p2p/protobuf.Reader.ReadMsgWithContext

Both the batched and unbatched implementations perform nearly identically over libp2p (~99.07 ms vs ~99.08 ms for n=25 and ~475.34 ms vs ~475.29 ms for n=120).

The separated CPU profiles reveal why this is the case: the top CPU consumers in both implementations are OS-level thread blocking operations (runtime.pthread_cond_wait, runtime.usleep, runtime.kevent). These account for over 60-80% of all CPU samples depending on the run. Because the program is almost entirely bottlenecked by waiting for libp2p stream data to traverse network sockets, any optimizations gained by batching protobuf structures in memory are negligible compared to the network latency.

The memory profile shows that chunk parsing and cryptographic hashing (golang.org/x/crypto/sha3.(*state).Sum and pb.(*Delivery).Unmarshal) dominate memory allocations.

Even if the delimitedReaderMaxSize is raised to 1MB, allowing larger batches, no performance gains are visible.

Conclusion

Introducing a breaking protocol change to batch protobuf messages yields no statistical CPU performance gain over the multiplexer latency, so the simpler, single-message streaming approach is preferred.

I encourage everyone interested to validate the findings.

Checklist

• I have read the coding guide.
• My change requires a documentation update, and I have done it.
• I have added tests to cover my changes.
• I have filled out the description and linked the related issues.

Description

Open API Spec Version Changes (if applicable)

Motivation and Context (Optional)

Related Issue (Optional)

Screenshots (if appropriate):

AI Disclosure

• This PR contains code that has been generated by an LLM.
• I have reviewed the AI generated code thoroughly.
• I possess the technical expertise to responsibly review the code generated in this PR.

[janos]

One comment regarding TCP (and UDP) Dynamics, a standard internet packet (MTU) is about 1500 bytes. A single 4 KB chunk already requires 3 TCP packets. Whether you flush chunks to the OS individually or in a batch, the OS TCP stack will still stream them out in 1500-byte packets at the maximum rate the network allows.