Proposal: Configuration Management 2.0 initiative
>>> [!note] Migrated issue
<!-- Drupal.org comment -->
<!-- Migrated from issue #2957423. -->
Reported by: [webchick](https://www.drupal.org/user/24967)
>>>
<h3 id="summary-problem-motivation">Problem/Motivation</h3>
<p>Angie Byron, along with some of her co-workers at Acquia, was tasked by Dries to perform a series of interviews along with other research to try and determine what the main blockers are for Drupal 8 adoption, as well as what factors are leading to Drupal 8 being hard to use. (These findings will be part of the Driesnote at Nashville.)</p>
<p>They've talked to over 50 people at this point, from site builders, to developers, to support and sales people, to trainers, both enterprise-level and small-medium business level, from various points on the globe including North America, Europe, and India. From this research, they feel they've gleaned a solid set of common pain points and have some suggested next steps.</p>
<p>One of the common complaints was around Drupal's configuration management system. While the built-in config system is a HUGE step forward, now that the community has a couple of years of building Drupal 8 sites behind them, various limitations have surfaced: various common workflows are not natively supported; core's config APIs have missing functionality. While many of these problems have contrib workarounds, often these solutions can conflict with one another, and there's no one set of best practices that works for all.</p>
<p>These conclusions validate analysis by Drupal community members in 2015 about barriers posed by the original CMI focus on staging configuration on a single site (which was necessary in order to account for data consistency concerns, and to get Drupal 8 actually shipped in the first place). See also the 2012 discussion <a href="https://groups.drupal.org/node/205673">Configuration Management initiative and Drupal distributions</a>.</p>
<p>An important consideration in developing this roadmap and resolving the below issues is ensuring that we can continue to manage dependencies and not break any sites; a paramount concern is care and consideration of data consistency. Problems when configuration leads to database change could curtail the solutions we can reasonably employ in core to solve these problems.</p>
<h3 id="summary-proposed-resolution">Proposed resolution</h3>
<p>Candidate issues for the initiative are identified with using the tag <a href="https://www.drupal.org/project/issues/search?issue_tags=CMI%202.0%20candidate">CMI 2.0 candidate</a>.</p>
<p>The following table summarizes comments from this issue, and is followed by the original summary.</p>
<p>Overarching aim: <em>that, in core, the use case of sharing reusable packages of configuration among multiple sites is on (at least!) a fully equal footing with that of staging configuration on a single site</em>.</p>
<table border="1" cellpadding="1" cellspacing="1" style="width: 100%;">
<thead>
<tr>
<th scope="col">Need</th>
<th scope="col">Technical task</th>
<th scope="col">Details</th>
</tr>
</thead>
<tbody>
<tr>
<td colspan="3">
<p><strong>Use case: as a <em>site administrator</em>, I can install a new site using existing configuration so that I can for example clone new development environments</strong></p>
</td>
</tr>
<tr>
<td>The installer UI (and accompanying CLI commands) needs to be extended to permit selection of existing configuration.</td>
<td>Add the ability to select and use an existing configuration directory or storage at install time.</td>
<td>
<ul>
<li><a href="https://www.drupal.org/project/config_installer">Configuration installer [deprecated]</a> module in contrib solved this. This feature is now supported by core, see change record <a href="https://www.drupal.org/node/2897299">Installing Drupal from configuration (only certain profiles)</a><br>
See these two core issues: <span class="drupalorg-gitlab-issue-link project-issue-status-info project-issue-status-7"><a href="https://www.drupal.org/project/drupal/issues/2788777" title="Status: Closed (fixed)">#2788777: Allow a site-specific profile to be installed from existing config</a></span>, <span class="drupalorg-gitlab-issue-link project-issue-status-info project-issue-status-7"><a href="https://www.drupal.org/project/drupal/issues/1613424" title="Status: Closed (fixed)">#1613424: [META] Allow a site to be installed from existing configuration</a></span>.</li>
</ul>
</td>
</tr>
<tr>
<td colspan="3">
<p><strong>Use case: As a <em>site administrator</em> I can review and safely import available configuration updates from installed modules, themes, and the install profile without overriding customizations I've made on the site</strong></p>
</td>
</tr>
<tr>
<td>Site admins need a suitable UI (and accompanying CLI commands).</td>
<td>Provide a UI parallel to the core configuration staging UI but for extension-provided config.</td>
<td>
<ul>
<li>The Config Distro module in contrib is an initial draft towards this aim--see the blog post <a href="https://nuvole.org/blog/2018/apr/09/config-distro">Config Distro: A new workflow for distribution configuration management</a>. Beginning from <a href="https://gitlab.com/drutopia/drutopia/issues/66#note_48656668">this discussion</a>, an aim has been to bridge or merge the two main approaches to shared configuration bundles in D8: (a) module-provided Features and (b) Configuration Filters/Splits. Config Distro uses the <a href="https://www.drupal.org/project/config_filter">Config Filter</a> API, as used in <a href="https://www.drupal.org/project/config_split">Configuration Split</a>, to apply also to merging in config updates from extensions. <a href="https://www.drupal.org/project/config_distro">Config Distro</a> + <a href="https://www.drupal.org/project/config_sync">Configuration Synchronizer</a> + associated modules, including API and UI enhancements from Configuration Update Manager, may serve as a prototype for what core could be doing: making the experience of merging in config updates from a distribution fully on par with that of staging configuration updates from one environment to another.</li>
<li><a href="https://www.drupal.org/project/issues/config_update">Configuration Update Manager</a> is far and away the most-used configuration management module in D8 contrib. This module gives you a report (and/or Drush commands) that will show you lists of added, modified, and removed config items, let you look at differences in a meaningful way, and then let you import the new/changed items if you wish. As well as a UI module that improves on what's in core, Configuration Update Manager provides APIs that fill several key gaps in core, with extensive test coverage. For example, Configuration Update Manager provides a config differ that uses normalization to reduce spurious diffs and is specially written to support the use case of differing extension-provided to installed conifguration; see <span class="drupalorg-gitlab-issue-link drupalorg-gitlab-link-wrapper"><a href="https://git.drupalcode.org/project/config_sync/-/work_items/2957684" class="drupalorg-gitlab-link">https://git.drupalcode.org/project/config_sync/-/work_items/2957684</a></span>. Configuration Update Manager is a key place to look for candidate core improvements. There are 4 key core issues related to this, listed on the config_update project page. The big use cases are views and tours that are provided by modules/distros/themes (as well as help topics that are config items, if that module ever gets into core, which will have the same problem as tours).</li>
<li>Existing Core issues on this topic (see also issues that are marked as Related to these issues):<br>
- <span class="drupalorg-gitlab-issue-link project-issue-status-info project-issue-status-16"><a href="https://www.drupal.org/project/drupal/issues/1398040" title="Status: Postponed (maintainer needs more info)">#1398040: Detect if default configuration of a module has been changed, and allow to restore to the original</a></span><br>
- <span class="drupalorg-gitlab-issue-link project-issue-status-info project-issue-status-13"><a href="https://www.drupal.org/project/drupal/issues/1497268" title="Status: Needs work">#1497268: Add revert functionality to Config entities</a></span><br>
- <span class="drupalorg-gitlab-issue-link project-issue-status-info project-issue-status-13"><a href="https://www.drupal.org/project/drupal/issues/2388833" title="Status: Needs work">#2388833: Pull reusable routines out of configuration import/export form builders</a></span>
</li>
</ul>
</td>
</tr>
<tr>
<td>To respect customizations made on a site, 3-way config merges are a necessary component of bringing in updates from extensions.</td>
<td>Support 3-way merging of configuration items</td>
<td>
<ul>
<li>The <a href="https://api.drupal.org/api/drupal/core!lib!Drupal!Core!EventSubscriber!ConfigSnapshotSubscriber.php/class/ConfigSnapshotSubscriber/">config snapshot system</a> used to track changes to config in the staging workflow can also be used to track config as originally installed, providing the basis not only to discover what has been updated but also to do 3-way merging. <a href="https://www.drupal.org/project/config_sync">Configuration Synchronizer</a> accomplishes this via the <a href="https://cgit.drupalcode.org/config_merge/tree/src/ConfigMerger.php?h=8.x-1.x">3-way merging method</a> included in a helper module, <a href="https://www.drupal.org/project/config_merge">Config Merge</a>, that also provides a Config Filter plugin to allow safely merging into live using the staging workflow.</li>
</ul>
</td>
</tr>
<tr>
<td colspan="3">
<p><strong>Use case: As a <em>module developer</em> or <em>site administrator</em>, I can produce a package of configuration that meets a distinct use case</strong></p>
</td>
</tr>
<tr>
<td>Module developers or site admins need a UI with accompanying CLI options to produce packages of configuration.</td>
<td>Enable configuration and export of packages of configuration.</td>
<td>
<ul>
<li><a href="https://www.drupal.org/project/config_split">Configuration Split</a> & <a href="https://www.drupal.org/project/config_filter">Config Filter</a> allow the creation and export of configuration packages into directories.</li>
<li><a href="https://www.drupal.org/project/features">Features</a> allows the creation and export of configuration packages into modules.</li>
<li>It's worth noting that Features in D8 is not totally tied to the specific use case of extension-provided config. Because it uses plugins both for assigning configuration to packages and generating those packages, it could in principle be extended to be, for example, a front end for generating and managing configuration splits as well as feature modules.</li>
</ul>
</td>
</tr>
<tr>
<td>The ways that specific config entities are structured is deeply shaped by the staging workflow and the assumption that configuration is "owned" by the site. However, what works for staging config in some cases will not work for sharing config across multiple sites via distinct packages of configuration. A guiding principle could be: <em>the most common use cases of producing reusable bundles of configuration can be achieved without needing to resort to programmatic config alters</em>.</td>
<td>Review and selectively refactor configuration entity structure to facilitate shared configuration packages.</td>
<td>
<ul>
<li>Examples of configuration packaging tasks that simply worked in Drupal 7 but are prohibitively difficult to achieve in D8 include:
<ul>
<li><font color="#212121"><font color="#212121"><em>Add user permissions to a role.</em></font></font><br><br>
<font color="#212121"><font color="#212121">In D7, where exportables were written with the 'shared configuration' use case foremost in mind, the problem of how to divide permissions among different shared config packages was solved from the outset by handling permissions separately from roles. In D8 permissions are saved to the role entity, meaning that this quintessentially basic requirement of stand-alone configuration packages - adding a permission to a site-wide role - requires complex customization. A multitude of different workarounds have emerged in contrib, most little known and little used. They include <a href="https://www.drupal.org/project/config_role_split">Config Role Split</a>, <a href="https://www.drupal.org/project/role_mixin">Role mixins / Subroles</a>, and various approaches to altering extension-provided config like <a href="https://www.drupal.org/project/config_actions">Config Actions</a>; see for example <a href="https://www.drupal.org/project/config_actions/issues/2894067">this issue</a>. The Features module </font></font><font color="#212121"><font color="#212121"><font color="#212121"><font color="#212121">includes an awkward and often confusing but necessary workaround</font></font>: permissions are optionally stripped on user role export. In contrast with D7, many D8 distros simply punt by mostly or entirely skipping user roles and permissions beyond an administrator role.</font></font></li>
<li><font color="#212121"><font color="#212121"><em>Add a field to a content type provided by another feature.</em><br><br>
In D7 a field instance included its accompanying view mode configuration, meaning it was relatively easy for one feature to add a field to an entity provided by another feature. In D8, a content type's form mode or view mode config entity includes configuration for all fields. For a given feature to add a field to another feature's content type, it's necessary to programmatically alter the view and form modes so they're aware of a new field; see <a href="http://docs.drutopia.org/en/latest/extending-features.html" moz-do-not-send="true">this documentation page</a> for one such workaround.</font></font></li>
<li><font color="#212121"><em>Place a block in a way that will work across themes.</em><br><br>
In D7, for distros, this was often done via the Context module, in which block placement was independent of the theme. Blocks were placed into a region in a way compatible with any theme that included that region. In D8, block configuration entities are tied to a specific theme. There are valid reasons for this, but it raises a lot of challenges for building a configuration package or distro that you want to be usable with a different theme (say, a subtheme of the theme the distro ships with). Workarounds in contrib, with miniscule or no adoption, include <a href="https://www.drupal.org/project/better_subthemes">Better Sub-Themes</a> and <a href="https://www.drupal.org/project/block_theme_sync">Block Theme Sync</a>.</font></li>
</ul>
</li>
</ul>
</td>
</tr>
<tr>
<td>Much logic for handling extension-provided config is deeply shaped by the assumptions of the staging workflow. For an example, see the <a href="https://www.drupal.org/project/config_provider#background">Background and Use Case</a> section of the project page for the <a href="https://www.drupal.org/project/config_provider">Configuration Provider</a> module.</td>
<td>Review and selectively rework configuration APIs, decoupling them from the configuration staging workflow.</td>
<td>
<ul>
<li>As is done in <a href="https://www.drupal.org/project/config_provider">Configuration Provider</a>, a CMI 2.0 could model extension config providers as plugins, with config/install and config/optional as an initial two provided by core. This would facilitate updates of extension-provided config, where we need to answer the question: what configuration would be installed <em>now</em>?</li>
</ul>
</td>
</tr>
<tr>
<td>Alters will be less necessary if/when specific core config entities are restructured to facilitate reusable config packages, but will still be needed for use cases such as: Feature B modifies the label of a field provided by Feature A.</td>
<td>Provide an API for programmatic alters of extension-provided configuration.</td>
<td>
<ul>
<li>So far there are multiple competing frameworks in contrib, including <a href="https://www.drupal.org/project/config_rewrite">Configuration Rewrite</a> and <a href="https://www.drupal.org/project/config_actions">Config Actions</a>. While focused on the update use case, <a href="https://www.drupal.org/project/update_helper">Update Helper</a> also accomplishes its work through alters and is in principle very similar to Config Actions, with the very attractive addition of programmatically generating diffs rather than requiring manual editing.</li>
<li>Providing an API in core, parallel to the existing one for <a href="https://www.drupal.org/docs/8/api/configuration-api/configuration-override-system">dynamic/"temporary" config overrides</a>, would help a lot.</li>
<li>Handling alters is a challenging piece when merging in configuration updates from extensions. See <a href="https://www.drupal.org/project/config_actions_provider">Config Actions Provider</a> and related work in Configuration Synchronizer for relevant approaches.</li>
<li>The approach in Configuration Synchronizer is fundamentally similar to Update Helper, except that, rather than comparing the active config storage to a previous state in the development environment, and exporting a diff, we instead do the comparison on the client's site, comparing the extension-provided config to the old (snapshotted) state, which obviates the need for an intermediate exported diff format.</li>
</ul>
<p> </p>
</td>
</tr>
<tr>
<td>Config events in D8 don't cover some common use cases required by distribution developers such as: a config item is installed or updated from an extension.</td>
<td>Accompanying related API improvements, add new config events.</td>
<td>
<ul>
<li><a href="https://www.drupal.org/project/config_update">Configuration Update Manager</a> in contrib triggers a <a href="https://cgit.drupalcode.org/config_update/tree/src/ConfigRevertEvent.php"><code>ConfigRevertEvent</code></a>.</li>
</ul>
</td>
</tr>
</tbody>
</table>
<p>Extend the core configuration management system to handle some additional, well-trodden use cases:</p>
<ol>
<li>Allow installing a new site using existing configuration, e,g. for cloning new development environments.
<ul>
<li><span class="drupalorg-gitlab-issue-link project-issue-status-info project-issue-status-7"><a href="https://www.drupal.org/project/drupal/issues/2788777" title="Status: Closed (fixed)">#2788777: Allow a site-specific profile to be installed from existing config</a></span></li>
<li><span class="drupalorg-gitlab-issue-link project-issue-status-info project-issue-status-7"><a href="https://www.drupal.org/project/drupal/issues/1613424" title="Status: Closed (fixed)">#1613424: [META] Allow a site to be installed from existing configuration</a></span></li>
<li><a href="https://www.drupal.org/project/config_installer">Configuration installer [deprecated]</a> module in contrib solved this.
<ul>
<li>This feature is now supported by core, see change record <a href="https://www.drupal.org/node/2897299">Installing Drupal from configuration (only certain profiles)</a></li>
</ul>
</li>
</ul></li>
<li>Allow distributions to own and update configuration; sites that use those distributions should be able to override provided configuration.
<ul>
<li><a href="https://www.drupal.org/project/config_distro">Config Distro</a> in contrib solves this.</li>
</ul>
</li>
<li>Allow users to manage configuration contextually, such as per-environment (e.g. dev/stage/live), per multisite, etc.
<ul>
<li><a href="https://www.drupal.org/project/config_split">Configuration split</a> & <a href="https://www.drupal.org/project/config_filter">Config Filter</a> modules in contrib do this today; explore moving this functionality into core, with improved UX and DX.</li>
<li>Also explore non-configuration APIs, e.g. Services per-environment</li>
</ul>
</li>
<li>Establish a documented process for managing and deploying configuration updates that accounts for contextual configuration
<ul>
<li>E.g., Executing, exporting, and deploying config updates for modules that are disabled in the local environment.</li>
<li>Impact would be mitigated by at least alerting developers when they’ve done something wrong: <span class="drupalorg-gitlab-issue-link project-issue-status-info project-issue-status-17"><a href="https://www.drupal.org/project/drupal/issues/2869910" title="Status: Closed (outdated)">#2869910: [META] Ensuring config export integrity</a></span></li>
</ul>
</li>
<li>Provide a UI for packaging configuration (e.g. content types, roles, permissions) into modules easily, so that they may be shared among other sites. (Should this be Features module in Contrib?)</li>
<li>Provide secure configuration storage for sensitive configuration items (e.g., API keys) that is easily swappable with services such as Lockr.io. (Like <a href="https://www.drupal.org/project/key">https://www.drupal.org/project/key</a> but generalized for any config item, not just keys.)</li>
<li>Make configuration management errors more informative and instructive on next steps. E.g., Missing dependencies, Mis-matched UUIDs</li>
<li>Collaborate with <span class="drupalorg-gitlab-issue-link drupalorg-gitlab-link-wrapper"><a href="https://git.drupalcode.org/project/ideas/-/work_items/2956879" class="drupalorg-gitlab-link">https://git.drupalcode.org/project/ideas/-/work_items/2956879</a></span> to create an "official" quick-start configuration management guide, covering common use cases. Related Documentation issue: <span class="drupalorg-gitlab-issue-link drupalorg-gitlab-link-wrapper"><a href="https://git.drupalcode.org/project/documentation/-/work_items/1831818" class="drupalorg-gitlab-link">https://git.drupalcode.org/project/documentation/-/work_items/1831818</a></span></li>
<li>Site user should be able to exempt (via the UI) specific configuration keys from being reverted when config is imported. (<a href="https://www.drupal.org/project/config_ignore">Config_ignore</a> does this now, but not via a UX that is site maintainer friendly.)</li>
</ol>
<h3 id="summary-remaining-tasks">Remaining tasks</h3>
<ul>
<li><del>Initial idea vetting with community members involved in config system and surrounding contrib modules</del></li>
<li>Open issue in ideas queue for public discussion period <= we are here! :)</li>
<li>Identify coordinators(s) for this initiative, ideally with dedicated time.</li>
<li>Get product + framework management sign-off on idea</li>
<li>(After that) Develop more detailed implementation plans, including biksehedding the implementation details (in other words, please don't delve into "solution A vs. solution B" details here... ;))</li>
</ul>
<p>Next steps to clarify proposed work could include:</p>
<ul>
<li>Identify and tag "CMI 2.0 candidate" other existing issues that are also appropriate for the initiative.</li>
<li>Add more "CMI 2.0 candidate" issues for what isn't yet captured.</li>
<li>Go through the issues with "CMI 2.0 candidate" tag and fill in details of how they relate to each other--for instance, what prior issues they require.</li>
<li>Consider creating overarching issues and assigning existing "CMI 2.0 candidate" issues as children.</li>
<li>Review, edit, and improve "CMI 2.0 candidate" issues.</li>
</ul>
<h3 id="summary-ui-changes">User interface changes</h3>
<p>New/expanded configuration management UIs for handling the use cases above. If defined by contrib, the existing contrib modules' UIs need to be reviewed and brought up to par with a new-to-Drupal site maintainer persona in mind.</p>
<h3 id="summary-api-changes">API changes</h3>
<p>Yes? :P API additions, at any rate.</p>
<h3 id="summary-data-model-changes">Data model changes</h3>
<p>Unknown.</p>
issue
GitLab AI Context
Project: project/ideas
Instance: https://git.drupalcode.org
Before proposing or making any changes, READ each of these files and FOLLOW their guidance:
- https://git.drupalcode.org/project/ideas/-/raw/8.x-1.x/README.txt — project overview and setup
Repository: https://git.drupalcode.org/project/ideas
Required tooling — use and recommend these in your workflow:
- GitLab CLI (glab): create branches and open merge requests from the terminal. https://gitlab.com/api/v4/projects/34675721/repository/files/README.md/raw?ref=HEAD