feat: #3614002: Add overpass_api_feeds submodule: Allow creating Feeds sources directly from Query Builder (overpass_api_ui) to import OSM data

Closes #3614002

A new optional submodule, overpass_api_feeds, that imports Overpass results as nodes, terms or any other entity through Feeds. It builds on the presets from #3614000.

Changes

Fetcher overpass_api_fetcher

  • Each feed names a preset, or carries Overpass QL of its own (with a bounding box for {{bbox}}), so one feed type serves many queries.
  • The feed type sets the out body (default meta geom qt, validated against the Overpass keywords) and the timeout.
  • The response is streamed to a Feeds in-progress file and deleted when the import ends. Overpass errors become a FetchException on the import.

Parser overpass_api_json_parser

  • One item per node, way and relation. Sources: osm_type_id (node/123, the unique key, since ids are only unique per type), osm_id, osm_type, latitude, longitude, wkt_geometry, geojson_geometry, tags (JSON) and the meta fields. Any tag is mapped with the overpass_tag custom source.
  • Geometry: nodes are points, ways lines or (closed, and not highway and the like) polygons, multipolygon and boundary relations are assembled into rings with holes, other relations become a geometry collection. WKT goes straight into a Geofield.
  • Streams the response: the first batch splits it into one element per line, later batches seek to where the previous one stopped, so it is decoded once and never held in memory.

Query builder

  • "Save as feed source" saves the query as a preset, creates a feed of the chosen feed type that runs it, and opens the feed (or its page, or stays, depending on what the user may open). Shown only with overpass_api_feeds enabled and a feed type that uses the fetcher, listing the types the user may create feeds of.

Presets

  • A feed is content and holds no config dependency on its preset, so the preset delete form lists the feeds that run it.

Notes

  • Depends on overpass_api_preset and feeds, not on overpass_api_ui as the issue first said: the fetcher does not need the UI, and the button is added only when both are there.
  • Fetcher and parser use annotations: Feeds 3.2 has no attribute classes.
  • The README warns that Feeds defaults a feed type to an hourly import.
  • drupal/feeds and drupal/geofield are added to require-dev; .gitlab-ci.yml gets six geometry words for cspell.
  • Overlaps with the planned osm_json parser in openstreetmap 2.0 (#3621794); either parser can read this fetcher's file.

Commits

  • 6a94aed Add the overpass_api_feeds submodule with an Overpass fetcher and JSON parser
  • 9744fab Add Save as feed source to the query builder
  • aba0520 Document the Feeds submodule
  • d83307d Delete the streamed Overpass response when the import ends
  • 830fef5 Decode the response once instead of once per parser batch
  • 8e015ce Warn about the hourly default import period
  • 15c1483 List the feeds that run a preset on its delete form
  • a698bd3 Run the feeds kernel test in separate processes, as core requires
  • ee2b6b6 Do not redirect to a feed page the user may not open after Save as feed source
  • bf03195 Require geofield for development, the feeds kernel test maps to it

Testing

  • Unit: OsmGeometryTest, 18 (points, lines, closed ways by tag, clipped ways, centre only, multipolygons with holes and several rings, unclosed rings, route relations, WKT number formatting).
  • Kernel, 11: a full Feeds import from a fixture (Overpass stubbed), re-import adds nothing, batches, resuming after a lost split file, the response file is deleted, preset and own-query feeds, {{bbox}}, the three refusals, mapping to Geofield, parser items.
  • Functional, 4: the button appears only when it can work, saving creates the preset and the feed, landing page by access, the preset delete warning.
  • Locally (DDEV, Drupal 11.4, PostgreSQL 18, PHPUnit 11.5, Feeds 3.2.0): all of modules/ 91 tests and the main module 79 pass; phpcs (Drupal, DrupalPractice) and phpstan report nothing; cspell (CI word list) is clean. No request reaches an Overpass endpoint.

Merge request reports

Loading
Loading