>>> [!note] Migrated issue <!-- Drupal.org comment --> <!-- Migrated from issue #3543158. --> Reported by: [breidert](https://www.drupal.org/user/135619) >>> <h2>Goals &amp; Guiding Principles</h2> <ul> <li><strong>Guide LLM crawlers:</strong> Direct large language model (LLM) crawlers to a high-quality, pre-optimized corpus of content, ensuring they consume relevant information and reduce hallucinations.</li> <li><strong>Reduce training noise:</strong> Remove irrelevant HTML markup, navigation, and other "noise" that can confuse or dilute the core message of the content.</li> <li><strong>Protect primary SEO:</strong> Ensure that the simplified Markdown content is not indexed by traditional search engines, preventing duplicate content issues and maintaining the site's primary SEO strategy with its original HTML content.</li> <li><strong>Ease of use: </strong>Package the feature as a Drupal recipe to make it easy for site builders to implement best practices for LLM crawlers with a single installation.</li> </ul> <h2>Technical Approach</h2> <p>This feature will be built in a similar way to the recipe "LLM support" (<a href="https://www.drupal.org/project/llm_support">https://www.drupal.org/project/llm_support</a>). It will use the following modules:</p> <ul> <li><a href="https://www.drupal.org/project/llms_txt">/llms.txt</a> - authoring and output of the /llms.txt file</li> <li><a href="https://www.drupal.org/project/markdownify">Markdownify</a> - output each entity in Markdown format</li> </ul> <p>Potentially the two recipes will be merged.</p> <h2>MVP Functionality</h2> <p>The <a href="https://www.drupal.org/project/ai_recipe_llm_optimized_content">ai_recipe_llm_optimized_content</a> recipe will extend Drupal CMS by providing a mechanism to expose a curated, static, and optimized content corpus to LLM crawlers.</p> <ul> <li><strong>Configuration Interface:</strong> Provide a simple configuration interface where the site administrator can select the content types and individual nodes to be exposed to LLM crawlers.</li> <li><strong>Content Conversion:</strong> Convert the selected content into a clean Markdown format. This process will also optimize the content for LLM consumption, for example, by adding "Key Questions" or FAQ sections that can be answered based on the content of the document.</li> <li><strong>LLM.txt Generation:</strong> Automatically generate a valid llms.txt file at the site's root (e.g., example.com/llms.txt).</li> <li><strong>Crawler Guidance:</strong> The llms.txt file will contain a correctly formatted Allow rule for the site, with all included URLs ending in .md. Navigating to an .md URL (e.g., example.com/my-page.md) will successfully serve the Markdown version of the page content.</li> <li><strong>Web Server Configuration Documentation:</strong> Provide clear, copy-and-paste instructions for the two most common web servers (Apache and Nginx) to ensure the .md files are not indexed by traditional search engines.</li> <li> </ul> <h2>Post MVP Functionality</h2> <ul> <li><strong>Multilingual Support:</strong> Support for multilingual content.</li> <li><strong>Integration with other AI modules: </strong>Integration with other AI-driven modules for content generation.</li> <li><strong>Customizable Rules:</strong> A user interface to define custom optimization rules beyond the default "Key Questions" or FAQ sections.</li> </ul> > Related issue: [Issue #3550681](https://www.drupal.org/node/3550681) > Related issue: [Issue #3550772](https://www.drupal.org/node/3550772) > Related issue: [Issue #3548981](https://www.drupal.org/node/3548981) > Related issue: [Issue #3521569](https://www.drupal.org/node/3521569) > Related issue: [Issue #3548950](https://www.drupal.org/node/3548950) > Related issue: [Issue #3550952](https://www.drupal.org/node/3550952) > Related issue: [Issue #3547314](https://www.drupal.org/node/3547314) > Related issue: [Issue #3547320](https://www.drupal.org/node/3547320)