Improve context rendering, token handling, and prompt formatting
Follow-up to:
#3582504
_**NOTE: Before working on this issue, verify it's still relevant and makes sense.**_
**Description:**
Improve how selected context is ordered, formatted, compressed, and fit into token budgets. This work should make injected context easier for models to use, reduce wasted tokens, and prepare for model-specific rendering strategies.
**Includes:**
- `1.1` Lost-in-the-middle aware ordering
- `1.2` Model-aware token counting
- `1.3` Smart token overflow handling
- `1.4` Structured delimiter format
- `6.1` Compression for repeat injections
- `6.2` Per-item injection prefix
- `6.3` Prompt caching hints
**Scope:**
- Add configurable context ordering strategies.
- Improve token counting when model information is available.
- Add configurable overflow behavior.
- Support structured context output formats.
- Add repeat-injection compression options.
- Allow per-item injection prefixes.
- Explore prompt caching metadata/hints where supported by upstream AI APIs.
**Acceptance criteria:**
- Existing rendering behavior remains the default unless intentionally changed.
- New settings have config schema and defaults.
- Rendering behavior is covered by tests.
- Documentation explains each strategy and when to use it.
[x] AI Assisted Issue
issue