Change default fallback for token counting to cl100k_base
>>> [!note] Migrated issue
<!-- Drupal.org comment -->
<!-- Migrated from issue #3471721. -->
Reported by: [scott_euser](https://www.drupal.org/user/3267594)
Related to !56
>>>
<h3 id="summary-problem-motivation">Problem/Motivation</h3>
<p>Currently we fall back to gpt 3.5 turbo for counting when tiktoken-php does not have the provided model, yet this is confusing for developers as it implies openai is used in some errors yet openai may not actually be used.<br>
cl100k_base is what gpt 3.5 uses anyways so change it to be the fallback.</p>
<h4 id="summary-steps-reproduce">Steps to reproduce</h4>
<p>Use a model that is not supported by tiktoken-php</p>
<h3 id="summary-proposed-resolution">Proposed resolution</h3>
<p>Fallback to cl100k_base</p>
<h3 id="summary-remaining-tasks">Remaining tasks</h3>
<p>MR</p>
<h3 id="summary-ui-changes">User interface changes</h3>
<p>N/A</p>
<h3 id="summary-api-changes">API changes</h3>
<p>N/A</p>
<h3 id="summary-data-model-changes">Data model changes</h3>
<p>N/A</p>
issue