`NavyThread` may hang at startup in `runInEventBaseThreadAndWait` due to a folly keepAlive race

[rowe0x]

Summary

Occasionally the process hangs forever while constructing a NavyThread (via NavyRequestScheduler -> NavyRequestDispatcher). The constructing thread blocks in EventBase::runInEventBaseThreadAndWait(), while the worker ScopedEventBaseThread has already returned from loopForever() and is parked at stop->wait(). Since that loop is gone, the posted callback is never run and the Baton is never posted, so the wait never returns.

It's a startup race and reproduces intermittently when many threads are created concurrently at process start.

Versions

• CacheLib v2024.06.17.00, folly v2024.06.17.00 (relevant code is identical on current main).
• GCC 8, x86_64, Linux. Navy with ioEngine=IoUring, 4 reader + 4 writer threads.

Stack traces (3 threads; addresses anonymized as EB_A/EB_B)

Main thread — blocked in the constructor:

folly::Baton::wait
folly::EventBase::runInEventBaseThreadAndWait (this=EB_A)     EventBase.cpp
facebook::cachelib::navy::NavyThread::NavyThread             NavyThread.cpp
facebook::cachelib::navy::NavyRequestDispatcher::...          (dispatcher i=1)
facebook::cachelib::navy::NavyRequestScheduler::...

navy_reader_1 — drives EB_A, but its loop already exited:

folly::Baton::wait
folly::run (eb=EB_A)     ScopedEventBaseThread.cpp:45   // after loopForever() returned

navy_reader_0 — healthy control (won the race):

epoll_wait
folly::EventBase::loopForever (eb=EB_B)
folly::run (eb=EB_B)     ScopedEventBaseThread.cpp:40   // still inside loopForever

Only 2 of the expected 8 dispatcher threads exist; construction is stuck at dispatcher i=1.

Root cause

NavyThread's constructor:

th_ = std::make_unique<folly::ScopedEventBaseThread>(name.str()); // (1) starts the loop thread
auto& eb = *th_->getEventBase();
fm_ = &folly::fibers::getFiberManager(eb, opts);                  // (2) creates the default VirtualEventBase
eb.runInEventBaseThreadAndWait([this]{ currentNavyThread_ = this; }); // (3) blocks

• ScopedEventBaseThread returns as soon as waitUntilRunning() sees loopTid_ set in loopMainSetup() — before the loop's first applyLoopKeepAlive() activates keepAlive.
• Step (2) creates the default VirtualEventBase, whose constructor takes a keepAlive on the EventBase (loopKeepAliveCount_ += 1).
• The first applyLoopKeepAlive() reads the count and then checks the VEB (a TOCTOU):

auto keepAliveCount = loopKeepAliveCount();          // snapshot
if (auto veb = tryGetVirtualEventBase()) {
  if (veb->keepAliveCount() == 1) --keepAliveCount;  // discount the idle default VEB
}

Losing interleaving:

step	loop thread (navy_reader_1)	main thread (ctor)	count
t1	reads count = 1 (loopForever's own +1; no VEB yet)		1
t2	(preempted between the two statements)	creates VEB: +1	2
t3		publishes VEB pointer	2
t4	sees VEB, keepAliveCount()==1 -> --count -> 0		2

The discount assumes the snapshot already includes the VEB's +1, but here the snapshot was actually loopForever's own token. The loop discounts its own keepAlive to 0, keeps the notification queue as an internal event, event_base_loop reports "ran out of events", and loopForever() returns early. This also explains why i=0 is fine and i=1 hangs.

Suggested fixes

1. CacheLib: add a barrier before getFiberManager, e.g. eb.runInEventBaseThreadAndWait([]{});, so the loop activates keepAlive before the default VEB is created; or add a timeout so the failure is observable instead of a silent hang.
2. folly: make the read-count / check-VEB sequence in applyLoopKeepAlive consistent, or have waitUntilRunning() wait until keepAlive is actually active.