Skip to content

Add back off and retry to GpsReceiver #65

Description

@Michael-O-adtorque

Every so often, Google's pubsub API doesn't respond in 60 seconds, and so this then results in a Symfony\Component\Messenger\Exception\TransportException.

The below is an example from an internal service we use to fetch data from a third party.

I think that you should implement Exponential Back Off on this so that the system will re-attempt it if Google fails to respond.

`

Symfony\Component\Messenger\Exception\TransportException: cURL error 28: Operation timed out after 60000 milliseconds with 0 bytes received (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for https://australia-southeast1-pubsub.googleapis.com/v1/projects/alice-release/subscriptions/analytics.dv360-his...

Issue ID: REDACTED
Project: REDACTED
Date: 12/08/2025, 03:10:13

Tags

  • console.command: messenger:consume
  • console.command.exit_code: 1
  • environment: prod
  • handled: yes
  • level: fatal
  • logger: monolog.console
  • mechanism: generic
  • os: Linux 4.4.0
  • os.name: Linux
  • release: 1.0.0+no-version-set
  • runtime: php 8.3.22
  • runtime.name: php
  • server_name: localhost

Exceptions

Exception 1

Type: GuzzleHttp\Exception\ConnectException
Value: cURL error 28: Operation timed out after 60000 milliseconds with 0 bytes received (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for https://australia-southeast1-pubsub.googleapis.com/v1/projects/[REDACTED]/subscriptions/[REDACTED]:pull?%24alt=json%3Benum-encoding%3Dint

Stacktrace

 GuzzleHttp\Handler\CurlFactory::createRejection in /vendor/guzzlehttp/guzzle/src/Handler/CurlFactory.php [Line 275, column null] (Not in app)
            }
        }

        // Create a connection exception if it was a specific error code.
        $error = isset($connectionErrors[$easy->errno])
            ? new ConnectException($message, $easy->request, null, $ctx)  <-- SUSPECT LINE
            : new RequestException($message, $easy->request, $easy->response, null, $ctx);

        return P\Create::rejectionFor($error);
    }

---
Variable values at the time of the exception:
{
  "ctx": {
    "appconnect_time": "0.0",
    "appconnect_time_us": "0",
    "cainfo": "/etc/ssl/certs/ca-certificates.crt",
    "capath": "/etc/ssl/certs",
    "certinfo": [],
    "connect_time": "0.0",
    "connect_time_us": "0",
    "content_type": "null",
    "curl_version": "7.88.1",
    "download_content_length": "-1.0",
    "effective_method": "POST",
    "errno": "28",
    "error": "Operation timed out after 60000 milliseconds with 0 bytes received",
    "filetime": "-1",
    "header_size": "0",
    "http_code": "0",
    "http_version": "0",
    "local_ip": "169.254.8.1",
    "local_port": "47872",
    "namelookup_time": "4.8E-5",
    "namelookup_time_us": "48",
    "pretransfer_time": "0.000156",
    "pretransfer_time_us": "156",
    "primary_ip": "142.250.67.10",
    "primary_port": "443",
    "protocol": "2",
    "redirect_count": "0",
    "redirect_time": "0.0",
    "redirect_time_us": "0",
    "redirect_url": "<encoding error>",
    "request_size": "1613",
    "scheme": "HTTPS",
    "size_download": "0.0",
    "size_upload": "106.0",
    "speed_download": "0.0",
    "speed_upload": "1.0",
    "ssl_verify_result": "0",
    "ssl_verifyresult": "0",
    "starttransfer_time": "0.0",
    "starttransfer_time_us": "0",
    "total_time": "60.000797",
    "total_time_us": "60000797",
    "upload_content_length": "106.0",
    "url": "https://australia-southeast1-pubsub.googleapis.com/v1/projects/[REDACTED]/subscriptions/[REDACTED]:pull?%24alt=json%3Benum-encoding%3Dint"
  },
  "easy": "Object GuzzleHttp\\Handler\\EasyHandle"
}
------
 GuzzleHttp\Handler\CurlFactory::finishError in /vendor/guzzlehttp/guzzle/src/Handler/CurlFactory.php [Line 205, column null] (Not in app)
        // Retry when nothing is present or when curl failed to rewind.
        if (empty($easy->options['_err_message']) && (!$easy->errno || $easy->errno == 65)) {
            return self::retryFailedRewind($handler, $easy, $ctx);
        }

        return self::createRejection($easy, $ctx);  <-- SUSPECT LINE
    }

    private static function getCurlVersion(): string
    {
        static $curlVersion = null;
---
Variable values at the time of the exception:
{
  "easy": "Object GuzzleHttp\\Handler\\EasyHandle",
  "factory": "Object GuzzleHttp\\Handler\\CurlFactory",
  "handler": "Callable GuzzleHttp\\Promise\\PromiseInterface GuzzleHttp\\Handler\\CurlMultiHandler::__invoke [Psr\\Http\\Message\\RequestInterface request; array options]"
}
------
 GuzzleHttp\Handler\CurlFactory::finish in /vendor/guzzlehttp/guzzle/src/Handler/CurlFactory.php [Line 157, column null] (Not in app)
        if (isset($easy->options['on_stats'])) {
            self::invokeStats($easy);
        }

        if (!$easy->response || $easy->errno) {
            return self::finishError($handler, $easy, $factory);  <-- SUSPECT LINE
        }

        // Return the response if it is present and there is no error.
        $factory->release($easy);

---
Variable values at the time of the exception:
{
  "easy": "Object GuzzleHttp\\Handler\\EasyHandle",
  "factory": "Object GuzzleHttp\\Handler\\CurlFactory",
  "handler": "Callable GuzzleHttp\\Promise\\PromiseInterface GuzzleHttp\\Handler\\CurlMultiHandler::__invoke [Psr\\Http\\Message\\RequestInterface request; array options]"
}
------
 GuzzleHttp\Handler\CurlMultiHandler::processMessages in /vendor/guzzlehttp/guzzle/src/Handler/CurlMultiHandler.php [Line 267, column null] (Not in app)

            $entry = $this->handles[$id];
            unset($this->handles[$id], $this->delays[$id]);
            $entry['easy']->errno = $done['result'];
            $entry['deferred']->resolve(
                CurlFactory::finish($this, $entry['easy'], $this->factory)  <-- SUSPECT LINE
            );
        }
    }

    private function timeToNext(): int
------
 GuzzleHttp\Handler\CurlMultiHandler::tick in /vendor/guzzlehttp/guzzle/src/Handler/CurlMultiHandler.php [Line 180, column null] (Not in app)
        while (\curl_multi_exec($this->_mh, $this->active) === \CURLM_CALL_MULTI_PERFORM) {
            // Prevent busy looping for slow HTTP requests.
            \curl_multi_select($this->_mh, $this->selectTimeout);
        }

        $this->processMessages();  <-- SUSPECT LINE
    }

    /**
     * Runs \curl_multi_exec() inside the event loop, to prevent busy looping
     */
------
 GuzzleHttp\Handler\CurlMultiHandler::execute in /vendor/guzzlehttp/guzzle/src/Handler/CurlMultiHandler.php [Line 206, column null] (Not in app)
        while ($this->handles || !$queue->isEmpty()) {
            // If there are no transfers, then sleep for the next delay
            if (!$this->active && $this->delays) {
                \usleep($this->timeToNext());
            }
            $this->tick();  <-- SUSPECT LINE
        }
    }

    private function addRequest(array $entry): void
    {
------
 GuzzleHttp\Promise\Promise::invokeWaitFn in /vendor/guzzlehttp/promises/src/Promise.php [Line 251, column null] (Not in app)
    private function invokeWaitFn(): void
    {
        try {
            $wfn = $this->waitFn;
            $this->waitFn = null;
            $wfn(true);  <-- SUSPECT LINE
        } catch (\Throwable $reason) {
            if ($this->state === self::PENDING) {
                // The promise has not been resolved yet, so reject the promise
                // with the exception.
                $this->reject($reason);
------
 GuzzleHttp\Promise\Promise::waitIfPending in /vendor/guzzlehttp/promises/src/Promise.php [Line 227, column null] (Not in app)
    private function waitIfPending(): void
    {
        if ($this->state !== self::PENDING) {
            return;
        } elseif ($this->waitFn) {
            $this->invokeWaitFn();  <-- SUSPECT LINE
        } elseif ($this->waitList) {
            $this->invokeWaitList();
        } else {
            // If there's no wait function, then reject the promise.
            $this->reject('Cannot wait on a promise that has '
------
 GuzzleHttp\Promise\Promise::invokeWaitList in /vendor/guzzlehttp/promises/src/Promise.php [Line 272, column null] (Not in app)
        $waitList = $this->waitList;
        $this->waitList = null;

        foreach ($waitList as $result) {
            do {
                $result->waitIfPending();  <-- SUSPECT LINE
                $result = $result->result;
            } while ($result instanceof Promise);

            if ($result instanceof PromiseInterface) {
                $result->wait(false);
------
 GuzzleHttp\Promise\Promise::waitIfPending in /vendor/guzzlehttp/promises/src/Promise.php [Line 229, column null] (Not in app)
        if ($this->state !== self::PENDING) {
            return;
        } elseif ($this->waitFn) {
            $this->invokeWaitFn();
        } elseif ($this->waitList) {
            $this->invokeWaitList();  <-- SUSPECT LINE
        } else {
            // If there's no wait function, then reject the promise.
            $this->reject('Cannot wait on a promise that has '
                .'no internal wait function. You must provide a wait '
                .'function when constructing the promise to be able to '
------
 GuzzleHttp\Promise\Promise::wait in /vendor/guzzlehttp/promises/src/Promise.php [Line 69, column null] (Not in app)
        return $this->then(null, $onRejected);
    }

    public function wait(bool $unwrap = true)
    {
        $this->waitIfPending();  <-- SUSPECT LINE

        if ($this->result instanceof PromiseInterface) {
            return $this->result->wait($unwrap);
        }
        if ($unwrap) {
------
 Google\Cloud\PubSub\V1\Client\SubscriberClient::pull in /vendor/google/cloud-pubsub/src/V1/Client/SubscriberClient.php [Line 713, column null] (Not in app)
     *
     * @throws ApiException Thrown if the API call fails.
     */
    public function pull(PullRequest $request, array $callOptions = []): PullResponse
    {
        return $this->startApiCall('Pull', $request, $callOptions)->wait();  <-- SUSPECT LINE
    }

    /**
     * Seeks an existing subscription to a point in time or to a given snapshot,
     * whichever is provided in the request. Snapshots are used in [Seek]
---
Variable values at the time of the exception:
{
  "callOptions": [],
  "request": "Object Google\\Cloud\\PubSub\\V1\\PullRequest"
}
------
 call_user_func_array in [internal] [Line 0, column null] (Not in app)
---
Variable values at the time of the exception:
{
  "args": [
    "Object Google\\Cloud\\PubSub\\V1\\PullRequest",
    []
  ],
  "callback": "Callable Google\\Cloud\\PubSub\\V1\\PullResponse Google\\Cloud\\PubSub\\V1\\Client\\SubscriberClient::pull [Google\\Cloud\\PubSub\\V1\\PullRequest request; array [callOptions]]"
}
------
 Google\Cloud\Core\RequestHandler::sendRequest in /vendor/google/cloud-core/src/RequestHandler.php [Line 115, column null] (Not in app)
        $allArgs = [$request];
        $allArgs[] = $optionalArgs;

        try {
            $callable = [$clientObj, $method];
            $response = call_user_func_array($callable, $allArgs);  <-- SUSPECT LINE

            return $this->handleResponse($response);
        } catch (ApiException $ex) {
            throw $this->convertToGoogleException($ex);
        } catch (NotFoundException $e) {
---
Variable values at the time of the exception:
{
  "clientClass": "Google\\Cloud\\PubSub\\V1\\Client\\SubscriberClient",
  "method": "pull",
  "optionalArgs": [],
  "request": "Object Google\\Cloud\\PubSub\\V1\\PullRequest"
}
------
 Google\Cloud\PubSub\Subscription::pull in /vendor/google/cloud-pubsub/src/Subscription.php [Line 820, column null] (Not in app)
        $data['subscription'] = $this->name;
        $data['maxMessages'] =  $data['maxMessages'] ?? self::MAX_MESSAGES;
        $request = $this->serializer->decodeMessage(new PullRequest(), $data);

        $messages = [];
        $response = $this->requestHandler->sendRequest(  <-- SUSPECT LINE
            SubscriberClient::class,
            'pull',
            $request,
            $optionalArgs
        );
---
Variable values at the time of the exception:
{
  "options": {
    "maxMessages": "50"
  }
}
------
 PetitPress\GpsMessengerBundle\Transport\GpsReceiver::get in /vendor/petitpress/gps-messenger-bundle/src/Transport/GpsReceiver.php [Line 48, column null] (Not in app)
    public function get(): iterable
    {
        try {
            $messages = $this->pubSubClient
                ->subscription($this->gpsConfiguration->getSubscriptionName())
                ->pull($this->gpsConfiguration->getSubscriptionPullOptions());  <-- SUSPECT LINE

            foreach ($messages as $message) {
                yield $this->createEnvelopeFromPubSubMessage($message);
            }
        } catch (Throwable $exception) {
------

Exception 2

Type: Symfony\Component\Messenger\Exception\TransportException
Value: cURL error 28: Operation timed out after 60000 milliseconds with 0 bytes received (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for https://australia-southeast1-pubsub.googleapis.com/v1/projects/alice-release/subscriptions/analytics.dv360-historical.sub:pull?%24alt=json%3Benum-encoding%3Dint

Stacktrace

 PetitPress\GpsMessengerBundle\Transport\GpsReceiver::get in /vendor/petitpress/gps-messenger-bundle/src/Transport/GpsReceiver.php [Line 54, column null] (Not in app)

            foreach ($messages as $message) {
                yield $this->createEnvelopeFromPubSubMessage($message);
            }
        } catch (Throwable $exception) {
            throw new TransportException($exception->getMessage(), 0, $exception);  <-- SUSPECT LINE
        }
    }

    /**
     * {@inheritdoc}
 Symfony\Component\Messenger\Worker::run in /vendor/symfony/messenger/Worker.php [Line 111, column null] (Not in app)
                    $envelopes = $receiver->getFromQueues($queueNames);
                } else {
                    $envelopes = $receiver->get();
                }

                foreach ($envelopes as $envelope) {  <-- SUSPECT LINE
                    $envelopeHandled = true;

                    if ($receiver instanceof KeepaliveReceiverInterface) {
                        $this->keepalives[$envelope->getMessage()] = [$transportName, $envelope];
                    }
---
Variable values at the time of the exception:
{
  "options": {
    "sleep": "1000000"
  }
}
 Symfony\Component\Messenger\Command\ConsumeMessagesCommand::execute in /vendor/symfony/messenger/Command/ConsumeMessagesCommand.php [Line 260, column null] (Not in app)
        if ($queues = $input->getOption('queues')) {
            $options['queues'] = $queues;
        }

        try {
            $this->worker->run($options);  <-- SUSPECT LINE
        } finally {
            $this->worker = null;
        }

        return 0;
---
Variable values at the time of the exception:
{
  "input": "Object Symfony\\Component\\Console\\Input\\ArgvInput",
  "output": "Object Symfony\\Component\\Console\\Output\\ConsoleOutput"
}
 Symfony\Component\Console\Command\Command::run in /vendor/symfony/console/Command/Command.php [Line 318, column null] (Not in app)

        if ($this->code) {
            return ($this->code)($input, $output);
        }

        return $this->execute($input, $output);  <-- SUSPECT LINE
    }

    /**
     * Supplies suggestions when resolving possible completion options for input (e.g. option or argument).
     */
---
Variable values at the time of the exception:
{
  "input": "Object Symfony\\Component\\Console\\Input\\ArgvInput",
  "output": "Object Symfony\\Component\\Console\\Output\\ConsoleOutput"
}
 Symfony\Component\Console\Application::doRunCommand in /vendor/symfony/console/Application.php [Line 1092, column null] (Not in app)

        try {
            $this->dispatcher->dispatch($event, ConsoleEvents::COMMAND);

            if ($event->commandShouldRun()) {
                $exitCode = $command->run($input, $output);  <-- SUSPECT LINE
            } else {
                $exitCode = ConsoleCommandEvent::RETURN_CODE_DISABLED;
            }
        } catch (\Throwable $e) {
            $event = new ConsoleErrorEvent($input, $output, $e, $command);
---
Variable values at the time of the exception:
{
  "command": "Object Symfony\\Component\\Messenger\\Command\\ConsumeMessagesCommand",
  "input": "Object Symfony\\Component\\Console\\Input\\ArgvInput",
  "output": "Object Symfony\\Component\\Console\\Output\\ConsoleOutput"
}
 Symfony\Bundle\FrameworkBundle\Console\Application::doRunCommand in /vendor/symfony/framework-bundle/Console/Application.php [Line 123, column null] (Not in app)
                $requestStack->push(new CliRequest($command));
            }
        }

        try {
            $returnCode = parent::doRunCommand($command, $input, $output);  <-- SUSPECT LINE
        } finally {
            $requestStack?->pop();
        }

        if ($renderRegistrationErrors && $this->registrationErrors) {
---
Variable values at the time of the exception:
{
  "command": "Object Symfony\\Component\\Messenger\\Command\\ConsumeMessagesCommand",
  "input": "Object Symfony\\Component\\Console\\Input\\ArgvInput",
  "output": "Object Symfony\\Component\\Console\\Output\\ConsoleOutput"
}
 Symfony\Component\Console\Application::doRun in /vendor/symfony/console/Application.php [Line 341, column null] (Not in app)
        if ($command instanceof LazyCommand) {
            $command = $command->getCommand();
        }

        $this->runningCommand = $command;
        $exitCode = $this->doRunCommand($command, $input, $output);  <-- SUSPECT LINE
        $this->runningCommand = null;

        return $exitCode;
    }

---
Variable values at the time of the exception:
{
  "input": "Object Symfony\\Component\\Console\\Input\\ArgvInput",
  "output": "Object Symfony\\Component\\Console\\Output\\ConsoleOutput"
}
 Symfony\Bundle\FrameworkBundle\Console\Application::doRun in /vendor/symfony/framework-bundle/Console/Application.php [Line 77, column null] (Not in app)
            $this->renderRegistrationErrors($input, $output);
        }

        $this->setDispatcher($this->kernel->getContainer()->get('event_dispatcher'));

        return parent::doRun($input, $output);  <-- SUSPECT LINE
    }

    protected function doRunCommand(Command $command, InputInterface $input, OutputInterface $output): int
    {
        $requestStack = null;
---
Variable values at the time of the exception:
{
  "input": "Object Symfony\\Component\\Console\\Input\\ArgvInput",
  "output": "Object Symfony\\Component\\Console\\Output\\ConsoleOutput"
}
 Symfony\Component\Console\Application::run in /vendor/symfony/console/Application.php [Line 192, column null] (Not in app)
        }

        try {
            $this->configureIO($input, $output);

            $exitCode = $this->doRun($input, $output);  <-- SUSPECT LINE
        } catch (\Throwable $e) {
            if ($e instanceof \Exception && !$this->catchExceptions) {
                throw $e;
            }
            if (!$e instanceof \Exception && !$this->catchErrors) {
---
Variable values at the time of the exception:
{
  "input": "Object Symfony\\Component\\Console\\Input\\ArgvInput",
  "output": "Object Symfony\\Component\\Console\\Output\\ConsoleOutput"
}
 Symfony\Component\Runtime\Runner\Symfony\ConsoleApplicationRunner::run in /vendor/symfony/runtime/Runner/Symfony/ConsoleApplicationRunner.php [Line 49, column null] (Not in app)

        if (!$definition->hasOption('no-debug')) {
            $definition->addOption(new InputOption('--no-debug', null, InputOption::VALUE_NONE, 'Switches off debug mode.'));
        }

        return $this->application->run($this->input, $this->output);  <-- SUSPECT LINE
    }
}

 require_once in /vendor/autoload_runtime.php [Line 29, column null] (Not in app)
$app = $app(...$args);

exit(
    $runtime
        ->getRunner($app)
        ->run()  <-- SUSPECT LINE
);

---
Variable values at the time of the exception:
{
  "param0": "/var/www/vendor/autoload_runtime.php"
}
 Unknown function in /bin/console [Line 11, column null] (In app)

if (!is_file(dirname(__DIR__).'/vendor/autoload_runtime.php')) {
    throw new LogicException('Symfony Runtime is missing. Try running "composer require symfony/runtime".');
}

require_once dirname(__DIR__).'/vendor/autoload_runtime.php';  <-- SUSPECT LINE

return function (array $context) {
    $kernel = new Kernel($context['APP_ENV'], (bool) $context['APP_DEBUG']);

    return new Application($kernel);

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions