Commit 5d14cc1a authored by Youri van Koppen's avatar Youri van Koppen
Browse files

Issue #2907721 by MegaChriz, carolpettirossi, unstatu, PunamShelke, Anybody,...

Issue #2907721 by MegaChriz, carolpettirossi, unstatu, PunamShelke, Anybody, danharper, trickfun, webdrips, bharata: Added Feeds Log module for tracking imports extensively (including items that failed to import).
parent 5c688d8f
Loading
Loading
Loading
Loading

modules/log/README.md

0 → 100644
+83 −0
Original line number Diff line number Diff line
Feeds Log
=========
This module gives you more insight in what happens during an import.

It covers the following information:
- Which source data was used for the import?
- Per imported item, how did the source item look like after parsing?
- Which entities got created, updated or deleted? And which did fail to import?

**Beware! This module logs a lot of data. If you regularly run large imports, a
lot of data can get written to the filesystem and to the database.**


## Files logged on the filesystem
There are two types of files logged on the filesystem:
 * Fetched source data
 * Processed items

### Fetched source data
For each import, fetched data is logged on the filesystem. By default this is
in a folder called `private://feeds/logs/[import_id]/source/`.

### Processed items
Each processed item is logged on the filesystem. By default this is in a folder
called `private://feeds/logs/[import_id]/items/`.


## Configuration
There are three levels of configuration:
 * Global settings at /admin/config/content/feeds_log;
 * Settings per feed type;
 * Settings per feed.

### Global settings
Here you can configure the following:
 * How long logs should be kept;
 * Where on the filesystem to store fetched sources and processed items;
 * Overload protection: if an import ever happens to get in an (infinite) loop,
   Feeds Log stops after a certain amount of iterations with logging as else the
   filesystem could fill up with logged sources or processed items. You can
   configure the maximum allowed amount of logs per feed in a certain
   timeframe.

### Settings per feed type
Logging can be enabled per feed type. It is enabled by default. Additionally,
you can configure what kind of data you are interested in:
 * Logging can be enabled/disabled per operation (created, updated, failed,
   etc.)
 * Per operation you can configure whether or not you are interested in the
   processed item. Processed items can potentially take up a lot of space on the
   filesystem.
 * The fetched source. The fetched source can be big, so just as processed items
   it can take up a lot of space on the filesystem.

### Settings per feed
Logging can be enabled/disabled per feed, but logging on the feed type must be
enabled in order to enable logging for a feed. By default, logging is enabled
for the feed.
Feeds Log can automatically disable logging for a feed if an import gets in an
infinite loop. This is an effect of *overload protection* (see 'Global
settings' above). You then have to manually enable logging again if you still
want things logged for this feed.


## Data model

### Feeds Import Log entity type
On each import an entity of type "feeds_import_log" gets created.
This entity type acts as the container for a series of log entries. On this
entity type are stored:
 * Import ID
 * Feed ID
 * Import start time
 * Import end time
 * The user under which the import is ran (can be different from who triggered
   it)
 * Path to logged source files

### Database table "feeds_import_log_entry"
This table contains the individual log entries. Each entry represents an
operation on a processed item. It tells if a certain entity was created, updated
or cleaned or if an item failed to import. It contains a reference to the Feeds
Import Log entity and a reference to the Feed entity.
+4 −0
Original line number Diff line number Diff line
age_limit: 604800
stampede:
  max_amount: 100
  age: 1800
+657 −0
Original line number Diff line number Diff line
langcode: en
status: true
dependencies:
  module:
    - feeds_log
id: feeds_import_logs
label: 'Feeds import logs'
module: views
description: ''
tag: ''
base_table: feeds_import_log
base_field: import_id
display:
  default:
    id: default
    display_title: Master
    display_plugin: default
    position: 0
    display_options:
      title: Logs
      fields:
        import_id:
          id: import_id
          table: feeds_import_log
          field: import_id
          relationship: none
          group_type: group
          admin_label: ''
          entity_type: feeds_import_log
          entity_field: import_id
          plugin_id: field
          label: ID
          exclude: false
          alter:
            alter_text: false
            text: ''
            make_link: false
            path: ''
            absolute: false
            external: false
            replace_spaces: false
            path_case: none
            trim_whitespace: false
            alt: ''
            rel: ''
            link_class: ''
            prefix: ''
            suffix: ''
            target: ''
            nl2br: false
            max_length: 0
            word_boundary: true
            ellipsis: true
            more_link: false
            more_link_text: ''
            more_link_path: ''
            strip_tags: false
            trim: false
            preserve_tags: ''
            html: false
          element_type: ''
          element_class: ''
          element_label_type: ''
          element_label_class: ''
          element_label_colon: true
          element_wrapper_type: ''
          element_wrapper_class: ''
          element_default_classes: true
          empty: ''
          hide_empty: false
          empty_zero: false
          hide_alter_empty: true
          click_sort_column: value
          type: number_integer
          settings:
            thousand_separator: ''
            prefix_suffix: true
          group_column: value
          group_columns: {  }
          group_rows: true
          delta_limit: 0
          delta_offset: 0
          delta_reversed: false
          delta_first_last: false
          multi_type: separator
          separator: ', '
          field_api_classes: false
        start:
          id: start
          table: feeds_import_log
          field: start
          relationship: none
          group_type: group
          admin_label: ''
          entity_type: feeds_import_log
          entity_field: start
          plugin_id: field
          label: 'Import start time'
          exclude: false
          alter:
            alter_text: false
            text: ''
            make_link: false
            path: ''
            absolute: false
            external: false
            replace_spaces: false
            path_case: none
            trim_whitespace: false
            alt: ''
            rel: ''
            link_class: ''
            prefix: ''
            suffix: ''
            target: ''
            nl2br: false
            max_length: 0
            word_boundary: true
            ellipsis: true
            more_link: false
            more_link_text: ''
            more_link_path: ''
            strip_tags: false
            trim: false
            preserve_tags: ''
            html: false
          element_type: ''
          element_class: ''
          element_label_type: ''
          element_label_class: ''
          element_label_colon: true
          element_wrapper_type: ''
          element_wrapper_class: ''
          element_default_classes: true
          empty: ''
          hide_empty: false
          empty_zero: false
          hide_alter_empty: true
          click_sort_column: value
          type: timestamp
          settings:
            date_format: medium
            custom_date_format: ''
            timezone: ''
          group_column: value
          group_columns: {  }
          group_rows: true
          delta_limit: 0
          delta_offset: 0
          delta_reversed: false
          delta_first_last: false
          multi_type: separator
          separator: ', '
          field_api_classes: false
        end:
          id: end
          table: feeds_import_log
          field: end
          relationship: none
          group_type: group
          admin_label: ''
          entity_type: feeds_import_log
          entity_field: end
          plugin_id: field
          label: 'Import finish time'
          exclude: false
          alter:
            alter_text: false
            text: ''
            make_link: false
            path: ''
            absolute: false
            external: false
            replace_spaces: false
            path_case: none
            trim_whitespace: false
            alt: ''
            rel: ''
            link_class: ''
            prefix: ''
            suffix: ''
            target: ''
            nl2br: false
            max_length: 0
            word_boundary: true
            ellipsis: true
            more_link: false
            more_link_text: ''
            more_link_path: ''
            strip_tags: false
            trim: false
            preserve_tags: ''
            html: false
          element_type: ''
          element_class: ''
          element_label_type: ''
          element_label_class: ''
          element_label_colon: true
          element_wrapper_type: ''
          element_wrapper_class: ''
          element_default_classes: true
          empty: ''
          hide_empty: false
          empty_zero: false
          hide_alter_empty: true
          click_sort_column: value
          type: timestamp
          settings:
            date_format: medium
            custom_date_format: ''
            timezone: ''
          group_column: value
          group_columns: {  }
          group_rows: true
          delta_limit: 0
          delta_offset: 0
          delta_reversed: false
          delta_first_last: false
          multi_type: separator
          separator: ', '
          field_api_classes: false
        uid:
          id: uid
          table: feeds_import_log
          field: uid
          relationship: none
          group_type: group
          admin_label: ''
          entity_type: feeds_import_log
          entity_field: uid
          plugin_id: field
          label: User
          exclude: false
          alter:
            alter_text: false
            text: ''
            make_link: false
            path: ''
            absolute: false
            external: false
            replace_spaces: false
            path_case: none
            trim_whitespace: false
            alt: ''
            rel: ''
            link_class: ''
            prefix: ''
            suffix: ''
            target: ''
            nl2br: false
            max_length: 0
            word_boundary: true
            ellipsis: true
            more_link: false
            more_link_text: ''
            more_link_path: ''
            strip_tags: false
            trim: false
            preserve_tags: ''
            html: false
          element_type: ''
          element_class: ''
          element_label_type: ''
          element_label_class: ''
          element_label_colon: true
          element_wrapper_type: ''
          element_wrapper_class: ''
          element_default_classes: true
          empty: ''
          hide_empty: false
          empty_zero: false
          hide_alter_empty: true
          click_sort_column: target_id
          type: entity_reference_label
          settings:
            link: true
          group_column: target_id
          group_columns: {  }
          group_rows: true
          delta_limit: 0
          delta_offset: 0
          delta_reversed: false
          delta_first_last: false
          multi_type: separator
          separator: ', '
          field_api_classes: false
        sources_value:
          id: sources_value
          table: feeds_import_log__sources
          field: sources_value
          relationship: none
          group_type: group
          admin_label: ''
          entity_type: feeds_import_log
          entity_field: sources
          plugin_id: field
          label: Sources
          exclude: false
          alter:
            alter_text: false
            text: ''
            make_link: false
            path: ''
            absolute: false
            external: false
            replace_spaces: false
            path_case: none
            trim_whitespace: false
            alt: ''
            rel: ''
            link_class: ''
            prefix: ''
            suffix: ''
            target: ''
            nl2br: false
            max_length: 0
            word_boundary: true
            ellipsis: true
            more_link: false
            more_link_text: ''
            more_link_path: ''
            strip_tags: false
            trim: false
            preserve_tags: ''
            html: false
          element_type: ''
          element_class: ''
          element_label_type: ''
          element_label_class: ''
          element_label_colon: true
          element_wrapper_type: ''
          element_wrapper_class: ''
          element_default_classes: true
          empty: ''
          hide_empty: false
          empty_zero: false
          hide_alter_empty: true
          click_sort_column: value
          type: feeds_log_file_uri_link
          settings: {  }
          group_column: value
          group_columns: {  }
          group_rows: true
          delta_limit: 0
          delta_offset: 0
          delta_reversed: false
          delta_first_last: false
          multi_type: ul
          separator: ', '
          field_api_classes: false
        lid:
          id: lid
          table: feeds_import_log_entry
          field: lid
          relationship: none
          group_type: count
          admin_label: ''
          plugin_id: numeric
          label: 'Number of entries'
          exclude: false
          alter:
            alter_text: false
            text: ''
            make_link: false
            path: ''
            absolute: false
            external: false
            replace_spaces: false
            path_case: none
            trim_whitespace: false
            alt: ''
            rel: ''
            link_class: ''
            prefix: ''
            suffix: ''
            target: ''
            nl2br: false
            max_length: 0
            word_boundary: true
            ellipsis: true
            more_link: false
            more_link_text: ''
            more_link_path: ''
            strip_tags: false
            trim: false
            preserve_tags: ''
            html: false
          element_type: ''
          element_class: ''
          element_label_type: ''
          element_label_class: ''
          element_label_colon: true
          element_wrapper_type: ''
          element_wrapper_class: ''
          element_default_classes: true
          empty: ''
          hide_empty: false
          empty_zero: false
          hide_alter_empty: true
          set_precision: false
          precision: 0
          decimal: .
          separator: ','
          format_plural: false
          format_plural_string: !!binary MQNAY291bnQ=
          prefix: ''
          suffix: ''
        operations:
          id: operations
          table: feeds_import_log
          field: operations
          relationship: none
          group_type: group
          admin_label: ''
          entity_type: null
          entity_field: null
          plugin_id: entity_operations
          label: ''
          exclude: false
          alter:
            alter_text: false
            text: ''
            make_link: false
            path: ''
            absolute: false
            external: false
            replace_spaces: false
            path_case: none
            trim_whitespace: false
            alt: ''
            rel: ''
            link_class: ''
            prefix: ''
            suffix: ''
            target: ''
            nl2br: false
            max_length: 0
            word_boundary: true
            ellipsis: true
            more_link: false
            more_link_text: ''
            more_link_path: ''
            strip_tags: false
            trim: false
            preserve_tags: ''
            html: false
          element_type: ''
          element_class: ''
          element_label_type: ''
          element_label_class: ''
          element_label_colon: true
          element_wrapper_type: ''
          element_wrapper_class: ''
          element_default_classes: true
          empty: ''
          hide_empty: false
          empty_zero: false
          hide_alter_empty: true
          destination: false
      pager:
        type: mini
        options:
          offset: 0
          items_per_page: 30
          total_pages: null
          id: 0
          tags:
            next: ››
            previous: ‹‹
          expose:
            items_per_page: false
            items_per_page_label: 'Items per page'
            items_per_page_options: '5, 10, 25, 50'
            items_per_page_options_all: false
            items_per_page_options_all_label: '- All -'
            offset: false
            offset_label: Offset
      exposed_form:
        type: basic
        options:
          submit_button: Apply
          reset_button: false
          reset_button_label: Reset
          exposed_sorts_label: 'Sort by'
          expose_sort_order: true
          sort_asc_label: Asc
          sort_desc_label: Desc
      access:
        type: feeds_log_access
        options: {  }
      cache:
        type: tag
        options: {  }
      empty:
        area_text_custom:
          id: area_text_custom
          table: views
          field: area_text_custom
          relationship: none
          group_type: group
          admin_label: ''
          plugin_id: text_custom
          empty: true
          content: 'There are no logged imports yet.'
          tokenize: false
      sorts: {  }
      arguments:
        feed:
          id: feed
          table: feeds_import_log
          field: feed
          relationship: none
          group_type: group
          admin_label: ''
          entity_type: feeds_import_log
          entity_field: feed
          plugin_id: numeric
          default_action: 'not found'
          exception:
            value: all
            title_enable: false
            title: All
          title_enable: false
          title: ''
          default_argument_type: fixed
          default_argument_options:
            argument: ''
          default_argument_skip_url: false
          summary_options:
            base_path: ''
            count: true
            override: false
            items_per_page: 25
          summary:
            sort_order: asc
            number_of_records: 0
            format: default_summary
          specify_validation: true
          validate:
            type: none
            fail: 'not found'
          validate_options: {  }
          break_phrase: false
          not: false
      filters: {  }
      style:
        type: table
        options:
          grouping: {  }
          row_class: ''
          default_row_class: true
          columns:
            import_id: import_id
            start: start
            end: end
            uid: uid
            sources_value: sources_value
            operations: operations
          default: start
          info:
            import_id:
              sortable: false
              default_sort_order: asc
              align: ''
              separator: ''
              empty_column: false
              responsive: ''
            start:
              sortable: true
              default_sort_order: desc
              align: ''
              separator: ''
              empty_column: false
              responsive: ''
            end:
              sortable: true
              default_sort_order: desc
              align: ''
              separator: ''
              empty_column: false
              responsive: ''
            uid:
              sortable: false
              default_sort_order: asc
              align: ''
              separator: ''
              empty_column: false
              responsive: ''
            sources_value:
              align: ''
              separator: ''
              empty_column: false
              responsive: ''
            operations:
              align: ''
              separator: ''
              empty_column: false
              responsive: ''
          override: true
          sticky: false
          summary: ''
          empty_table: false
          caption: ''
          description: ''
      row:
        type: 'entity:feeds_import_log'
      query:
        type: views_query
        options:
          query_comment: ''
          disable_sql_rewrite: false
          distinct: false
          replica: false
          query_tags: {  }
      relationships: {  }
      group_by: true
      header: {  }
      footer: {  }
      display_extenders: {  }
    cache_metadata:
      max-age: -1
      contexts:
        - 'languages:language_content'
        - 'languages:language_interface'
        - url
        - url.query_args
      tags: {  }
  page_1:
    id: page_1
    display_title: Page
    display_plugin: page
    position: 1
    display_options:
      display_extenders: {  }
      path: feed/%feeds_feed/log
      menu:
        type: none
        title: Overview
        description: ''
        weight: 0
        expanded: false
        menu_name: main
        parent: ''
        context: '0'
      tab_options:
        type: tab
        title: Logs
        description: ''
        weight: 0
    cache_metadata:
      max-age: -1
      contexts:
        - 'languages:language_content'
        - 'languages:language_interface'
        - url
        - url.query_args
      tags: {  }
+44 −0
Original line number Diff line number Diff line
# Schema for the configuration files of the feeds_log module.

feeds_log.settings:
  type: config_object
  label: 'Feeds logging settings'
  mapping:
    age_limit:
      type: integer
      label: 'How long to keep logged imports'
    log_dir:
      type: string
      label: 'Feeds log directory'
    stampede:
      type: mapping
      label: 'Settings to prevent an overload of import logs'
      mapping:
        max_amount:
          type: integer
          label: 'The maximum number of allowed logs for a single feed within a certain timeframe'
        age:
          type: integer
          label: 'How long should be looked back for detecting an import log overload'

# Third party settings for feed types.
feeds.feed_type.*.third_party.feeds_log:
  type: mapping
  label: 'Feeds Log settings'
  mapping:
    status:
      type: boolean
    operations:
      type: sequence
      sequence:
        type: string
    items:
      type: sequence
      sequence:
        type: string
    source:
      type: boolean 

views.access.feeds_log_access:
  type: mapping
  label: 'Feeds Log access'
+9 −0
Original line number Diff line number Diff line
name: 'Feeds Log'
type: module
description: 'Detailed logging and reporting for Feeds imports.'
package: Feeds
core_version_requirement: ^9.3 || ^10
configure: feeds_log.settings
dependencies:
  - 'drupal:views'
  - 'feeds:feeds'
Loading