Tune queue_size defaults per input type and document recommendations

An unset queue_size now defaults to 1000 for local directories (where the
queue holds only file paths, as it historically did) and to 1.2 * n for
archive and s3 streaming (where a whole chunk is materialized in memory),
keeping 20% headroom over the thread pool. The queue_size entry was
dropped from the shipped config.json so these defaults apply out of the
box, and the README gains a "Choosing a queue size" section with tuning
guidance.

Author: lfoppiano

Readme.md

@@ -384,7 +384,6 @@ settings.
 ```json
 {
   "grobid_server": "http://localhost:8070",
-  "queue_size": 1000,
   "sleep_time": 5,
   "timeout": 60,
   "coordinates": [
@@ -403,7 +402,7 @@ settings.
 | Parameter       | Description                                                                                                      | Default                 |
 |-----------------|------------------------------------------------------------------------------------------------------------------|-------------------------|
 | `grobid_server` | GROBID server URL                                                                                                | `http://localhost:8070` |
-| `queue_size`    | Number of files queued per processing chunk. If not set, it follows the concurrency `n` so the thread pool is never starved. **Tune carefully: a large queue size will result in the data being written less frequently, and on the archive/S3 paths a whole chunk is held in memory at once**   | same as `n`             |
+| `queue_size`    | Number of files queued per processing chunk. See [Choosing a queue size](#choosing-a-queue-size). | 1000 for local directories, 1.2 × `n` for archives and S3 |
 | `sleep_time`    | Wait time when server is busy (seconds)                                                                          | 5                       |
 | `timeout`       | Client-side timeout (seconds)                                                                                    | 180                     |
 | `coordinates`   | XML elements for coordinate extraction                                                                           | See above               |
@@ -413,6 +412,24 @@ settings.
 > Since version 0.0.12, the config file is optional. The client will use default localhost settings if no configuration
 > is provided.
 
+### Choosing a queue size
+
+`queue_size` controls how many files are grouped into one processing chunk. It is a memory/durability knob, not a
+concurrency one: parallelism toward the GROBID server is set by `-n`, and each chunk is processed `n` files at a time.
+When `queue_size` is not set, the client picks a sensible default per input type (see the table above); set it
+explicitly only if you need to override that. A few guidelines:
+
+- **Never set it below `n`.** Effective parallelism is `min(n, queue_size)`: a queue smaller than the thread pool
+  leaves workers idle. A bit of headroom above `n` (the default streaming value is 1.2 × `n`) keeps the pool busy.
+- **Results are written per chunk.** Output files land on disk only once a whole chunk has been processed, so a
+  larger queue means results are written less frequently and an interrupted run loses at most one chunk of work
+  (already-written results are skipped on re-run unless `--force` is used).
+- **Local PDF directories:** the queue holds only file paths, so large values are essentially free — the default is
+  1000. Lower it if you want results flushed to disk more often on long runs.
+- **Archives (zip/tar) and S3:** each chunk is read or downloaded *into memory* before processing starts, so peak RAM
+  grows with `queue_size × average file size`. Keep it moderate — the 1.2 × `n` default is a safe floor; going up to
+  a few multiples of `n` (e.g. 2–5 ×) trades memory for slightly better throughput around chunk boundaries.
+
 > [!WARNING]
 > **Citation consolidation and the `timeout` setting.** When `--consolidate_citations` (or `consolidate_citations=True`)
 > is enabled, GROBID queries external services (e.g. CrossRef) to enrich the extracted references. This is considerably

config.json

@@ -1,6 +1,5 @@
 {
   "grobid_server": "http://localhost:8070",
-  "queue_size": 1000,
   "timeout": 180,
   "sleep_time": 5,
   "coordinates": [

grobid_client/grobid_client.py

@@ -16,6 +16,7 @@
 """
 from __future__ import annotations
 
+import math
 import os
 import io
 import json
@@ -163,13 +164,25 @@ def _set_config_params(self, params: dict) -> None:
             if value is not None:
                 self.config[key] = value
 
-    def _effective_queue_size(self, n: int) -> int:
-        """Return the configured queue_size, defaulting to the concurrency n.
+    # Default chunk size when walking a local directory: only file paths are
+    # queued (nothing is pre-loaded), so a large chunk costs next to nothing.
+    LOCAL_DIR_QUEUE_SIZE = 1000
+
+    def _effective_queue_size(self, n: int, local_files: bool = False) -> int:
+        """Return the configured queue_size, or a default derived from n.
 
         A queue smaller than the thread pool leaves workers idle, so when no
-        explicit value is configured the chunk size follows n.
+        explicit value is configured the chunk size follows the concurrency n,
+        with 20% headroom to keep the pool busy around the chunk boundary. For
+        local directories, where the queue holds only file paths, a large
+        fixed default is used instead.
         """
-        return self.config.get("queue_size") or n
+        configured = self.config.get("queue_size")
+        if configured:
+            return configured
+        if local_files:
+            return self.LOCAL_DIR_QUEUE_SIZE
+        return math.ceil(n * 1.2)
 
     def _warn_on_consolidation_timeout(self, consolidate_citations: bool) -> None:
         """Warn when citation consolidation is enabled with a low client timeout.
@@ -869,7 +882,7 @@ def _run_file_batches(
 
         Returns the aggregated (processed, errors, skipped) counts.
         """
-        queue_size = self._effective_queue_size(n)
+        queue_size = self._effective_queue_size(n, local_files=True)
         processed_files_count = 0
         errors_files_count = 0
         skipped_files_count = 0

tests/test_grobid_client.py

@@ -52,14 +52,17 @@ def test_init_default_values(self, mock_configure_logging, mock_test_server):
     @patch('grobid_client.grobid_client.GrobidClient._test_server_connection')
     @patch('grobid_client.grobid_client.GrobidClient._configure_logging')
     def test_effective_queue_size(self, mock_configure_logging, mock_test_server):
-        """Test that an unset queue_size defaults to the concurrency n."""
+        """Test the queue_size defaults: 1.2 * n for streaming, 1000 for local dirs."""
         mock_test_server.return_value = (True, 200)
 
         client = GrobidClient(check_server=False)
-        assert client._effective_queue_size(40) == 40
+        assert client._effective_queue_size(40) == 48
+        assert client._effective_queue_size(10) == 12
+        assert client._effective_queue_size(40, local_files=True) == 1000
 
         client.config['queue_size'] = 100
         assert client._effective_queue_size(40) == 100
+        assert client._effective_queue_size(40, local_files=True) == 100
 
     @patch('grobid_client.grobid_client.GrobidClient._test_server_connection')
     @patch('grobid_client.grobid_client.GrobidClient._configure_logging')