>>> [!note] Migrated issue <!-- Drupal.org comment --> <!-- Migrated from issue #3588772. --> Reported by: [ajv009](https://www.drupal.org/user/3653917) >>> <h3 id="summary-problem-motivation">Problem/Motivation</h3> <p>Acquia Source triggers a Tailwind CSS rebuild plus component compilation after any component upload. With a large bulk upload (e.g., 79 components in one recorded session), this rebuild takes minutes to hours. During the rebuild, the backend CMS API returns HTTP 503. The orchestrator has no post-upload readiness check &mdash; it immediately proceeds to subsequent phases, hits 503s, and retries with no backoff cap.</p> <p>Recorded timeline from a session that uploaded 79 components:</p> <ul> <li>04:05 (2026-03-28): 79 components uploaded successfully.</li> <li>14:38: First 503 on content push &mdash; over 10 hours later.</li> <li>04:27 (2026-03-30): Still 503, ~14 hours after the first 503 &mdash; the next morning.</li> <li>04:50&ndash;04:57: 10+ more 503 retry attempts.</li> </ul> <p>The correct behavior is to poll a fast-recovery signal (the OAuth token endpoint) every 60s and wait for HTTP 200 before proceeding to subsequent phases. Instead, the agent retried content push 10+ times across 14+ hours with no delay policy calibrated to the rebuild duration. The fix has two parts: (1) post-upload readiness poll instead of immediate proceed, and (2) batched uploads (10&ndash;15 components at a time with waits between batches) to keep individual rebuild cycles shorter.</p> <h4 id="summary-steps-reproduce">Steps to reproduce</h4> <ol> <li>Run a migrate-site session that uploads a large component batch (40+ components) to an Acquia Source instance.</li> <li>Observe that Phase 4 reports completion immediately after upload acknowledgement.</li> <li>Observe Phase 5/6/7 attempting content push and receiving HTTP 503 from the CMS API.</li> <li>Observe the agent retrying with no cap and no calibrated backoff &mdash; accumulating retries over many hours.</li> </ol> <h3 id="summary-proposed-resolution">Proposed resolution</h3> <p><strong>Step 1 &mdash; Add Phase 4 Completion readiness check to <code>SKILL.md</code>:</strong></p> <pre>PHASE 4 COMPLETION &mdash; Post-Upload Readiness Check After component upload, DO NOT immediately proceed to Phase 5/6/7. Poll the OAuth token endpoint every 60s for up to 20 minutes: curl -s -o /dev/null -w "%{http_code}" -X POST "$CANVAS_SITE_URL/oauth/token" \ -d "grant_type=client_credentials&amp;client_id=$CLIENT_ID&amp;client_secret=$CLIENT_SECRET" When response is 200, the Tailwind rebuild is complete and the CMS is ready. Log: "CMS ready after Xmin Ys". If not ready after 20 min, log warning and continue (it may still be rebuilding).</pre><p><strong>Step 2 &mdash; Implement <code>scripts/wait-for-cms.sh</code>:</strong></p> <pre>#!/usr/bin/env bash # wait-for-cms.sh &mdash; poll CMS readiness after bulk component upload CMS_URL="${1:-$CANVAS_SITE_URL}" MAX_WAIT=1200 # 20 minutes INTERVAL=60 elapsed=0 echo "Waiting for CMS to be ready after upload..." while [ $elapsed -lt $MAX_WAIT ]; do STATUS=$(curl -s -o /dev/null -w "%{http_code}" -X POST "$CMS_URL/oauth/token" \ -d "grant_type=client_credentials&amp;client_id=${CANVAS_CLIENT_ID}&amp;client_secret=${CANVAS_CLIENT_SECRET}") if [ "$STATUS" = "200" ]; then echo "CMS ready after ${elapsed}s"; exit 0 fi echo "CMS not ready (HTTP $STATUS), waiting ${INTERVAL}s... (${elapsed}/${MAX_WAIT}s elapsed)" sleep $INTERVAL elapsed=$((elapsed + INTERVAL)) done echo "WARNING: CMS not ready after ${MAX_WAIT}s. Tailwind rebuild may still be in progress." exit 1</pre><p><strong>Step 3 &mdash; Add 503 retry cap to upload-verifier and content-composer agents:</strong></p> <pre>503 RETRY POLICY (for content push after upload): Maximum 5 retries with 3-minute gaps. If still 503 after 5 retries: stop and report "CMS rebuilding &mdash; come back in 15 min." DO NOT retry 10+ times over multiple hours. Use the post-upload readiness check (wait-for-cms.sh) instead of reactive retries.</pre><p><strong>Step 4 &mdash; Batched-upload preventive measure:</strong> for large migrations (40+ components), upload in batches of 10&ndash;15 components with 2-minute waits between batches to keep individual Tailwind rebuild cycles shorter. Add this guidance to <code>SKILL.md</code> or <code>component-builder.md</code>.</p> <h3 id="summary-remaining-tasks">Remaining tasks</h3> <ul> <li>Add PHASE 4 COMPLETION readiness-check block to <code>.claude/skills/migrate-site/SKILL.md</code>.</li> <li>Add post-upload readiness step to <code>.claude/agents/upload-verifier.md</code> as a required final action before reporting Phase 4 complete.</li> <li>Create <code>scripts/wait-for-cms.sh</code> (executable).</li> <li>Add 503 RETRY POLICY block to <code>upload-verifier.md</code> and <code>content-composer.md</code>: max 5 retries, 3-min gaps, then user-facing "CMS rebuilding" message.</li> <li>Add batched-upload guidance for migrations with 40+ components.</li> <li>Verify on a staging instance: a 10+ component upload triggers the OAuth poll; Phase 4 complete is not signaled until 200 is observed; simulated 503 from CMS produces max 5 retries with 3-min gaps then a clear user message.</li> </ul> <h3 id="summary-ui-changes">User interface changes</h3> <p>None.</p> <h3 id="summary-api-changes">API changes</h3> <p>None introduced. The fix consumes existing endpoints: <code>POST /oauth/token</code> (used as the readiness signal because it recovers faster than the main API) and the existing component upload/content-push endpoints.</p> <h3 id="summary-data-model-changes">Data model changes</h3> <p>New skill artifact: <code>scripts/wait-for-cms.sh</code>. New log line in Phase 4 completion: "CMS ready after Xmin Ys".</p> <h3 id="summary-internal-references">Internal references</h3> <ul> <li>Source markdown: <code>RESEARCH-2026-04-28/99-final/ISSUES/Sprint-2-Determinism/04-P1-3-bulk-upload-503-tailwind-rebuild.md</code> (id: P1-3)</li> <li>Session evidence (full timeline, 14+ hours of retries): <code>RESEARCH-2026-04-28/02-edu-site-mockup-ai/fc6fc3db/issues-found.md</code> (ISSUE-2)</li> <li>Recommended verbatim text: <code>RESEARCH-2026-04-28/99-final/migrate-site-skill-recommendations.md</code> ("Add: Post-Upload Readiness Wait" section)</li> <li>Pattern: <code>RESEARCH-2026-04-28/07-cross-cutting/patterns.md</code> &sect; B1 (broader 503 pattern on Acquia instances; this is the POST-bulk-upload case, while 2-03 is the PATCH case)</li> <li>Anti-pattern: <code>RESEARCH-2026-04-28/07-cross-cutting/session-anti-patterns.md</code> &sect; AP1 (Retry Storm &mdash; 10+ retries over 14 hours is textbook)</li> <li>Cross-reference: <code>RESEARCH-2026-04-28/99-final/consolidated-issues.md</code> (P1-3 entry)</li> </ul> <h3 id="summary-open-questions">Open questions</h3> <ul> <li>Is the OAuth token endpoint available on all Acquia Source instances? Verify <code>CANVAS_CLIENT_ID</code> and <code>CANVAS_CLIENT_SECRET</code> are in the migration environment before relying on the poll.</li> <li>The 14+ hour 503 block was for 79 components. For smaller migrations (5&ndash;15 components) the rebuild may be only a few minutes. Should the 20-minute poll timeout scale with upload batch size?</li> <li>Does Acquia Source have a webhook/event signaling "rebuild complete"? If so, polling could be replaced with a more reliable signal.</li> <li>Are content-push 503s and component-upload 503s on the same endpoint, or different paths with potentially different 503 semantics?</li> </ul>