>>> [!note] Migrated issue <!-- Drupal.org comment --> <!-- Migrated from issue #2670036. --> Reported by: [hestenet](https://www.drupal.org/user/54034) >>> <p>Some recent instability that may have lead to the outages the weekend of 2016-02-14 are likely due to limitations in php 5.3. Upgrading the php version on the Drupal.org webnodes is primarily limited by the remaining Drupal 6 sites: qa.drupal.org which will be statically archived shortly, and groups.drupal.org which will receive some module updates to ensure it runs on 5.4.</p> <p>"Due to this same limitation, attempts to create references to any contained data, nested or otherwise, will fail silently. So $var = &amp;$object['foo'] will not throw an error, and $var will be populated with the contents of $object['foo'], but that data will be passed by value, not reference. For more information on the PHP limitation, see the note in the official PHP documentation at&middot; <a href="http://php.net/manual/arrayaccess.offsetget.php">http://php.net/manual/arrayaccess.offsetget.php</a> on ArrayAccess::offsetGet()"</p> <p>Once we have tested php 5.4 in our staging environment we'll upgrade a single webnode and put it in rotation, and if that works well upgrade all the remaining webnodes. </p> <p><strong>Post Mortem</strong><br> Following a series of outages caused by PHP instability, we began investigating the root causes. The first two outages on Feb 1, 2016 and Feb 8, 2016 were caused by what we believe to be an unrelated issue to the following three outages. Debugging the issue lead us to believe there was something which caused the garbage collection in combination with APC to error out and poison the php-fpm workers, eventually exhausting the database connections and causing the site to go offline until the poisoned php-fpm pool was restarted. These outages happened on:</p> <ul> <li>Feb 13, 2016 morning</li> <li>Feb 13, 2016 evening</li> <li>Feb 20, 2016 morning</li> </ul> <p>On February 9, 2016, an update to PHP 5.3.3 distributed with CentOS 6 was pushed to the web servers. This update included a single change to PHP's garbage collection (<a href="https://rhn.redhat.com/errata/RHBA-2016-0141.html">https://rhn.redhat.com/errata/RHBA-2016-0141.html</a>). We believe this change introduced a regression which we were hitting under load on Drupal.org.</p> <p>To resolve the issue, we determined upgrading to PHP 5.4 (already on our roadmap) presented the path forward with the most benefit.</p>