Loading
feat: #3614002: Add overpass_api_feeds submodule: Allow creating Feeds sources directly from Query Builder (overpass_api_ui) to import OSM data
Closes #3614002
A new optional submodule, overpass_api_feeds, that imports Overpass results as nodes, terms or any other entity through Feeds. It builds on the presets from #3614000.
Changes
Fetcher overpass_api_fetcher
- Each feed names a preset, or carries Overpass QL of its own (with a bounding box for
{{bbox}}), so one feed type serves many queries. - The feed type sets the
outbody (defaultmeta geom qt, validated against the Overpass keywords) and the timeout. - The response is streamed to a Feeds in-progress file and deleted when the import ends. Overpass errors become a
FetchExceptionon the import.
Parser overpass_api_json_parser
- One item per node, way and relation. Sources:
osm_type_id(node/123, the unique key, since ids are only unique per type),osm_id,osm_type,latitude,longitude,wkt_geometry,geojson_geometry,tags(JSON) and themetafields. Any tag is mapped with theoverpass_tagcustom source. - Geometry: nodes are points, ways lines or (closed, and not
highwayand the like) polygons, multipolygon and boundary relations are assembled into rings with holes, other relations become a geometry collection. WKT goes straight into a Geofield. - Streams the response: the first batch splits it into one element per line, later batches seek to where the previous one stopped, so it is decoded once and never held in memory.
Query builder
- "Save as feed source" saves the query as a preset, creates a feed of the chosen feed type that runs it, and opens the feed (or its page, or stays, depending on what the user may open). Shown only with
overpass_api_feedsenabled and a feed type that uses the fetcher, listing the types the user may create feeds of.
Presets
- A feed is content and holds no config dependency on its preset, so the preset delete form lists the feeds that run it.
Notes
- Depends on
overpass_api_presetandfeeds, not onoverpass_api_uias the issue first said: the fetcher does not need the UI, and the button is added only when both are there. - Fetcher and parser use annotations: Feeds 3.2 has no attribute classes.
- The README warns that Feeds defaults a feed type to an hourly import.
drupal/feedsanddrupal/geofieldare added to require-dev;.gitlab-ci.ymlgets six geometry words for cspell.- Overlaps with the planned
osm_jsonparser in openstreetmap 2.0 (#3621794); either parser can read this fetcher's file.
Commits
6a94aedAdd the overpass_api_feeds submodule with an Overpass fetcher and JSON parser9744fabAdd Save as feed source to the query builderaba0520Document the Feeds submoduled83307dDelete the streamed Overpass response when the import ends830fef5Decode the response once instead of once per parser batch8e015ceWarn about the hourly default import period15c1483List the feeds that run a preset on its delete forma698bd3Run the feeds kernel test in separate processes, as core requiresee2b6b6Do not redirect to a feed page the user may not open after Save as feed sourcebf03195Require geofield for development, the feeds kernel test maps to it
Testing
- Unit:
OsmGeometryTest, 18 (points, lines, closed ways by tag, clipped ways, centre only, multipolygons with holes and several rings, unclosed rings, route relations, WKT number formatting). - Kernel, 11: a full Feeds import from a fixture (Overpass stubbed), re-import adds nothing, batches, resuming after a lost split file, the response file is deleted, preset and own-query feeds,
{{bbox}}, the three refusals, mapping to Geofield, parser items. - Functional, 4: the button appears only when it can work, saving creates the preset and the feed, landing page by access, the preset delete warning.
- Locally (DDEV, Drupal 11.4, PostgreSQL 18, PHPUnit 11.5, Feeds 3.2.0): all of
modules/91 tests and the main module 79 pass; phpcs (Drupal, DrupalPractice) and phpstan report nothing; cspell (CI word list) is clean. No request reaches an Overpass endpoint.