>>> [!note] Migrated issue <!-- Drupal.org comment --> <!-- Migrated from issue #3590771. --> Reported by: [joshuami](https://www.drupal.org/user/434354) Related to !21 >>> <!-- Title: pa:check bulk mode does not time out cleanly on large media libraries --><h3>Problem/Motivation</h3> <p>Running <code>drush pa:check</code> against a large media library, especially over a remote stream wrapper like <code>s3fs</code>, does not time out cleanly. Memory grows monotonically, remote reads have no upper bound on duration, the smalot parser can spin on malformed PDFs, image-heavy scans can decompress 50&ndash;100&times; and trigger uncatchable <code>memory_limit</code> fatals, and a sustained run of read failures (e.g. backend outage) looks identical to a healthy run. Field reproduction on a ~10K-PDF library locked the host VM hard enough to require a manual reboot after only ~750 items.</p> <p>The bulk path needs several related fixes &mdash; bounded I/O and memory, structured failure classification, an outage-aware circuit breaker, queue-mode and incremental-resumption support, and machine-readable output &mdash; coordinated as one change.</p> <h3>Resolution</h3> <h4>Safety nets</h4> <ul> <li><strong>Chunked iteration with memory reclaim.</strong> 50-item chunks; entity cache, static cache, and GC reset between chunks. Memory plateaus across long runs.</li> <li><strong>Bounded I/O.</strong> 30s socket timeout around every read in <code>PdfParserService</code>. 60s <code>pcntl_alarm</code> wall-clock guard around <code>Parser::parseContent()</code>.</li> <li><strong>Three-way failure classification.</strong> <code>_missing_file</code> (orphan), <code>_parse_error</code> (permanent), <code>_io_error</code> (transient). A 5s read-elapsed threshold separates orphan 404s from real backend timeouts.</li> <li><strong>Transient-only circuit breaker.</strong> Only <code>_io_error</code> counts toward the breaker; permanent failures and missing files don't trip it. Configurable via <code>--max-consecutive-failures=N</code> (default 10) and <code>pdfa11y.settings:max_consecutive_io_failures</code>.</li> <li><code>\Throwable</code> <strong>catches</strong> in the analyzer's per-plugin and parser's outer paths so a PHP <code>Error</code> in one plugin is isolated to one STATUS_ERROR row instead of crashing the loop.</li> </ul> <h4>Bulk fitness</h4> <ul> <li><strong>Raw-bytes reuse.</strong> <code>PdfParserService</code> caches the most-recent file's bytes; <code>HeadingStructureCheck</code>'s ObjStm fallback consults the cache before re-reading.</li> <li><strong>Transactional <code>storeResults()</code>.</strong> DELETE + INSERTs in one transaction; parallel readers never see an empty window.</li> <li><strong>Sentinel-aware <code>--missing-only</code>.</strong> A fid counts as checked when any row's <code>check_id</code> is not <code>_io_error</code>. Permanent sentinels (<code>_too_large</code>, <code>_image_payload_too_large</code>, <code>_parse_error</code>, <code>_subprocess_failed</code>, <code>_missing_file</code>) are definitive; only transient <code>_io_error</code> rows retry. Plus <code>--since-fid=N</code> and <code>--status=published|unpublished|any</code>.</li> <li><strong>Queue mode.</strong> <code>pa:check --queue</code> enqueues into <code>pdfa11y_check</code>; <code>drush queue:run pdfa11y_check</code> drains one short-lived worker per item. Recommended pattern for full-library backfills.</li> <li><code>--skip-missing-files</code> + <code>pdfa11y.settings:skip_missing_files</code> for runs where the orphan set is already characterized.</li> </ul> <h4>Out-of-memory guard</h4> <p>Field testing surfaced image-heavy PDFs whose stream decompression exceeds any reasonable memory budget. Three-part defense:</p> <ul> <li><strong>Image-payload pre-flight.</strong> New <code>PdfPreflightService</code> walks the PDF's cross-reference table with bounded <code>fseek</code>/<code>fread</code> windows (~1&ndash;2 MB resident) and estimates decompressed image-payload weight via per-filter multipliers (<code>DCTDecode</code>/<code>JPXDecode</code> &times;10, <code>CCITTFaxDecode</code>/<code>FlateDecode</code>/<code>LZWDecode</code> &times;8, <code>JBIG2Decode</code> &times;5). Files over threshold record <code>_image_payload_too_large</code> without invoking smalot. Configurable via <code>--max-image-bytes=N</code> / <code>pdfa11y.settings:max_image_bytes</code>; default 50 MB.</li> <li><strong>File-size backstop.</strong> Coarse cap on raw source-file size for outliers and PDFs the preflight can't introspect. Configurable via <code>--max-filesize=N</code> / <code>pdfa11y.settings:max_filesize</code>; default 10 MB. Records <code>_too_large</code>.</li> <li><strong>Subprocess isolation (two-layer fork-safety).</strong> Each parse runs in a forked child (<code>pcntl_fork()</code> + <code>stream_socket_pair()</code>); the parent owns all DB writes. <strong>Layer 1</strong>: the child sets <code>memory_limit = parent_limit + subprocess_memory_headroom</code> (default +128 MB) so PHP shutdown can run a <code>register_shutdown_function()</code> handler that <code>SIGKILL</code>s the child before any PDO destructor can write <code>COM_QUIT</code> to the inherited MySQL socket. <strong>Layer 2</strong>: on detected child failure the parent writes the <code>_subprocess_failed</code> sentinel via a freshly-opened DB connection so any in-flight corruption can't propagate. Configurable via <code>--no-subprocess</code> / <code>pdfa11y.settings:use_subprocess_isolation</code> and <code>pdfa11y.settings:subprocess_memory_headroom</code>; transparent no-op without <code>pcntl</code>.</li> </ul> <h4>Operator ergonomics</h4> <ul> <li><code>--format=summary|table|csv|json|quiet</code>. <code>summary</code> (one line per file) is the new bulk default; <code>table</code> remains the single-mid default. CSV/JSON accumulate and flush at end of run.</li> <li><strong>In-memory result context.</strong> <code>AccessibilityCheckResult</code> gains optional readonly <code>fid</code>/<code>mid</code>/<code>uri</code> plus <code>withContext()</code> so CSV/JSON output identifies the file without joining <code>file_managed</code>.</li> <li><strong>Categorized summary line</strong> with explicit buckets: checked, with-issues, missing files, too large, image-payload too large, subprocess failures, permanent parse errors, transient I/O errors, skipped-via-flag.</li> <li><strong>README "Bulk operations" section.</strong> Documents the queue pattern, failure classification, breaker semantics, the calibration table for the OOM-guard thresholds, the subprocess-isolation two-layer model with tuning guidance, and the recommended chunked <code>--missing-only</code> operator flow.</li> </ul> <h3>Field test verification</h3> <p>Validated across three rounds on a real ~8,150-PDF s3fs-backed library on an Acquia ODE (PHP 8.3, 512 M CLI). Final round-3 result, one <code>--limit=1500</code> chunk: 1,478 items checked to a clean summary line, 1 <code>_subprocess_failed</code> row (parent's MySQL connection survived, batch continued), 10&times; drop in subprocess-failure rate vs. round 2 (11 &rarr; 1). No "MySQL server has gone away" or "Packets out of order" cascade. Observed throughput ~25&ndash;30 items/min on this tier; full 10K-PDF coverage in 6&ndash;7 chunks of ~50&ndash;60 min each.</p> <h3>Deferred to follow-up issues</h3> <ul> <li><code>uri</code> column on <code>pdfa11y_results</code> &mdash; schema migration deferred to a follow-up.</li> <li>Distinct editor-facing message for encrypted PDFs (a <code>FAILURE_ENCRYPTED</code> kind with specific remediation copy).</li> <li><code>memory_get_peak_usage(true)</code> in the bulk summary line for ops visibility on long runs.</li> <li>Admin report improvements: expose all stored sentinel + plugin data with drill-down filters.</li> </ul> <h3>Remaining tasks</h3> <ul> <li>None.</li> </ul> <h3>User interface changes</h3> <p>New <code>pa:check</code> options: <code>--max-consecutive-failures</code>, <code>--missing-only</code>, <code>--since-fid</code>, <code>--status</code>, <code>--queue</code>, <code>--format</code>, <code>--skip-missing-files</code>, <code>--max-filesize</code>, <code>--max-image-bytes</code>, <code>--no-subprocess</code>. New default bulk output is one line per file; single-mid output unchanged. New configuration keys: <code>max_consecutive_io_failures</code>, <code>skip_missing_files</code>, <code>max_filesize</code>, <code>max_image_bytes</code>, <code>use_subprocess_isolation</code>, <code>subprocess_memory_headroom</code>.</p> <h3>API changes</h3> <p><code>AccessibilityCheckResult</code> gains optional readonly <code>fid</code>/<code>mid</code>/<code>uri</code> and <code>withContext()</code>. <code>PdfParserService</code> gains public <code>FAILURE_*</code> constants, <code>getLastFailureKind()</code>, <code>getRawBytes()</code>/<code>clearRawBytes()</code>. <code>Pdfa11yAnalyzer::analyze()</code> gains an optional <code>?int $mid</code> parameter; the class adds <code>analyzeIsolated()</code>, <code>canFork()</code>, <code>warmAnalysisCaches()</code>, <code>resolveSubprocessMemoryLimit()</code>, <code>parseMemoryLimitToBytes()</code>, <code>storeResultsOnFreshConnection()</code>, <code>writeResults()</code>, plus <code>tooLargeResult()</code> / <code>imagePayloadTooLargeResult()</code> / <code>subprocessFailedResult()</code> factories. New service <code>pdfa11y.preflight</code> (<code>PdfPreflightService</code>) with <code>getImageStreamBytes(string $fileUri): ?int</code>. New <code>QueueWorker</code> plugin <code>pdfa11y_check</code>. <code>AccessibilityCheckInterface</code> is unchanged. New result <code>check_id</code> sentinels: <code>_missing_file</code>, <code>_io_error</code>, <code>_too_large</code>, <code>_image_payload_too_large</code>, <code>_subprocess_failed</code> (plus existing <code>_parse_error</code>).</p> <h3>Data model changes</h3> <p>None. <code>pdfa11y_results</code> schema unchanged.</p> <h3>Environment</h3> <ul> <li>Reproduces on pdfa11y 1.0.6, Drupal 10 / 11.</li> <li>Most severe with remote stream wrappers (<code>s3fs</code>); local-filesystem libraries hit the memory and parser-spin issues but not the read-timeout issue.</li> <li>Manifests on libraries large enough that one process can't complete the full run before exhausting the container.</li> </ul>