Skip to content

Issue #3189622: Implement a Generic Caching Stream Wrapper that Can Decorate Other Remote Stream Wrappers

This MR implements a stream wrapper that can decorate other remote stream wrappers, to provide seamless local caching.

When used, each target resource is transferred into local, temporary storage the first time it is accessed. From then on, I/O-intensive operations like seeking and reading are performed on the local, cached copy of the resource. Subsequent requests for the same resource will re-use the cached copy of the resource if it is available, to avoid unnecessary requests to the remote system.

The new stream wrapper is implemented to generically decorate/wrap anything that implements \Drupal\remote_stream_wrapper\StreamWrapper\RemoteStreamWrapperInterface, including HttpStreamWrapper. Since stream wrappers get instantiated by PHP and therefore aren't as flexible as true Symfony-managed services, we can't actually use service decoration in *.services.yml nor can we inject services via the constructor. Stream wrappers also must provide a constructor that takes no arguments.

So, to use this class to wrap another stream wrapper protocol, you must:

  1. Define a subclass of this class that specifies the protocol of the "real" stream wrapper via a call to the parent constructor.
  2. Declare the subclass as a stream wrapper in *.services.yml but under a different protocol scheme than the wrapped protocol. This new protocol scheme is what you will use any time you want caching behavior.

For example, assume you had a custom FTP-based backend service called "Initech" for which you've written a custom FTP stream wrapper that implements RemoteStreamWrapperInterface and retrieves payroll reports over FTP using credentials configured on the site. Also assume that your custom stream wrapper is registered to handle the initech scheme. Now, you want to add caching to requests through that stream wrapper.

You can accomplish this feat without making changes to your existing stream wrapper. You just need to adjust your service definitions and then define a subclass of this decorator class that specifies the appropriate protocol to wrap.

The service definitions in *.services.yml would need to look like this:

stream_wrapper.initech.direct:
  class: Drupal\initech\StreamWrapper\InitechStream
  tags:
    - { name: 'stream_wrapper', scheme: 'initech-direct' }

stream_wrapper.initech:
  class: Drupal\initech\StreamWrapper\CachingInitechStream
  tags:
    - { name: 'stream_wrapper', scheme: 'initech' }

And then the subclass of the decorator would look like this:

<?php
namespace Drupal\initech\StreamWrapper;

class CachingInitechStream extends CachingStreamWrapperBase {
  public function __construct() {
    parent::__construct('initech', 'initech-direct');
  }
}

For completeness, this assumes that your original Initech stream wrapper is defined like this:

<?php
namespace Drupal\initech\StreamWrapper;

class InitechStream implements RemoteStreamWrapperInterface {
  // ... implementation omitted ... /
}

With these definitions in place, now all requests for the initech:// scheme should automatically be cached into the temporary file system under temporary://remote_stream_cache/initech/.

As a bonus, all files cached this way will automatically be garbage collected by Drupal during cron runs, usually every four hours. The actual frequency is controlled by the temporary_maximum_age setting of the system.file config, which can be set by admins via the "Delete temporary files after" setting on the "File system" settings page. The cached files also get cleaned up on Drupal cache clears.

Edited by Guy Elsmore-Paddock

Merge request reports