Change default fallback for token counting to cl100k_base
>>> [!note] Migrated issue <!-- Drupal.org comment --> <!-- Migrated from issue #3471721. --> Reported by: [scott_euser](https://www.drupal.org/user/3267594) Related to !56 >>> <h3 id="summary-problem-motivation">Problem/Motivation</h3> <p>Currently we fall back to gpt 3.5 turbo for counting when tiktoken-php does not have the provided model, yet this is confusing for developers as it implies openai is used in some errors yet openai may not actually be used.<br> cl100k_base is what gpt 3.5 uses anyways so change it to be the fallback.</p> <h4 id="summary-steps-reproduce">Steps to reproduce</h4> <p>Use a model that is not supported by tiktoken-php</p> <h3 id="summary-proposed-resolution">Proposed resolution</h3> <p>Fallback to cl100k_base</p> <h3 id="summary-remaining-tasks">Remaining tasks</h3> <p>MR</p> <h3 id="summary-ui-changes">User interface changes</h3> <p>N/A</p> <h3 id="summary-api-changes">API changes</h3> <p>N/A</p> <h3 id="summary-data-model-changes">Data model changes</h3> <p>N/A</p>
issue