Output definitions cannot say "always present, may be null"

Problem/Motivation

An output definition has one flag, required, and the JSON Schema normalizer reads it as two things:

if (!$object->isRequired()) {
  // ...
  $schema['type'] = [$schema['type'], 'null'];
}

and, on the parent map:

if ($property->isRequired()) {
  $schema['required'][] = $property_name;
}

So required: TRUE means "the key is present and not null", and required: FALSE means "the key may be missing and may be null". JSON Schema keeps these apart: required is key presence, and null in type is nullability.

The shape that falls between them is common: a key that is always sent and holds null when there is no value. A photo's EXIF date, an entity's optional field, a timestamp that has not happened yet:

new MapOutputDefinition('Image', 'A photo.', property_definitions: [
  'date' => new OutputDefinition('timestamp', 'Date', 'When the photo was taken.', required: FALSE),
]);
{
  "type": "object",
  "properties": {"date": {"type": ["integer", "null"]}}
}

The tool always sends date. The schema says it may be missing. That passes validation, but it promises less than the tool delivers. A client that trusts the schema writes code for a missing key that never goes missing, and a strict structured-output consumer can't rely on the key.

Declaring it required: TRUE doesn't help. The schema drops the null branch, and core's NotNull rejects the null the tool sends.

Method tools on #3518120 (closed) hit this for every nullable member that isn't marked omit_when_null:

#[ToolProperty(description: 'When the photo was taken, from its EXIF data.', data_type: 'timestamp')]
public function getDate(): ?int

Proposed resolution

Add a nullable flag to output definitions, separate from required. It is a plain boolean that defaults to FALSE; NULL means absence. The four combinations:

required nullable result schema
TRUE FALSE (default) key always present, never null T, listed in required
TRUE TRUE key always present, null when there is no value [T, "null"], listed in required
FALSE FALSE (default) an output holding NULL is left out of the result T, key may be missing
FALSE TRUE key sent with null when there is no value, may also be missing [T, "null"]
new OutputDefinition('timestamp', 'Date', 'When the photo was taken.', required: TRUE, nullable: TRUE);
{
  "type": "object",
  "properties": {"date": {"type": ["integer", "null"]}},
  "required": ["date"]
}

The third row changes today's behavior: an optional output that never declared the flag currently advertises [T, "null"] and sends the null. That is a change to the published schema of every optional output, so it lands before rc1 rather than after.

The normalizer reads required for the parent's required list and nullable for the null branch. Validation leaves out core's NotNull for a nullable definition. Lists (#3583040 (closed)) and maps (#3583069 (closed)) already work this way: EffectiveDataDefinition does not mark them required, so typed data adds no NotNull, and the value walkers reject NULL themselves through validateListNotNull(). A nullable definition skips that check.

On #3518120 (closed), the describer then maps a method-tool member like this:

member required nullable type
non-nullable yes no T
nullable yes yes [T, null]
nullable + omit_when_null no no T

Inputs get the same flag in a follow-up issue. The default keeps today's behavior, an optional input may be sent as null; nullable: TRUE means null is a value, so getInputValue() returns the NULL instead of the default.

API changes

New nullable parameter and isNullable() / setNullable() on output definitions, default FALSE. An optional output that never declared the flag stops advertising null and is left out of the result when it holds NULL.

Remaining tasks

  • Flip the default to FALSE and leave an optional NULL output out of the
  • Decide the nested rule and apply it at every depth: a null in an optional, non-nullable map property is left out like a top-level one; a null list item cannot be left out without shifting the positions the path contract (#3583065 (closed)) relies on, so it is a violation unless the item definition is nullable. result; setOutputValue() already stores the key, so array_key_exists() tells "set to NULL" from "never set" for the required cases.
  • Teach the normalizers and output validation to read the flag.
  • Map omit_when_null and nullable types to the two flags in the method-tool describer (#3518120 (closed) landed first).
  • Remove the @todo on TypedMethodDescriber::memberIsRequired().
  • File the follow-up for the input flag.
Edited by Michael Lander