aggregator.module

<?php
/* $Id$ */

/**
 * Implementation of hook_help().
 */
function aggregator_help($section) {
  switch ($section) {
    case 'admin/help#aggregator':
      return t('
      <p>Thousands of web sites, especially news sites and weblogs, syndicate their most recent site content for others to display. The syndicated content always includes titles, also known as headlines, for the newest published stories. Each headline acts as a direct link to the stories on the remote site. Along with the headline, most sites typically provide either the first few paragraphs of the story or a short summary. Many individuals use client-based news aggregators on their personal computer to aggregate content, such as <a href="http://www.disobey.com/amphetadesk/">Amphetadesk</a>.</p>
      <p>Drupal also has a news aggregator built in as a standard feature. With it, you can subscribe to feeds from other sites and display their content for your site users. Simply enable the aggregator module in site administration and enter the feeds that you choose.</p>
      <h3>What do I need to subscribe to a feed?</h3>
      <p>The standard method of syndication is using the XML-based <a href="http://blogs.law.harvard.edu/tech/rss">Rich Site Summary</a> (RSS). To syndicate a site\'s content, obtain the full URL of the RSS page providing syndication. Common file tags for RSS pages are .rss, .xml and .rdf. Example: <a href="http://slashdot.org/slashdot.rdf">Slashdot RSS</a>.</p>
      <p>Most weblog sites that offer syndication will have an obvious link on the main page. Often you need only look for a XML syndication button, such as the one Drupal uses for site syndication.</p>
      <p>But some sites do not make their RSS feeds as easy to find. Or maybe you want to find a number of feeds on a given topic, without extensively searching the web. In that case, try an RSS syndication directory such as <a href="http://www.syndic8.com/">Syndic8</a>.</p>
      <p>To learn much more about RSS, read Mark Pilgrim\'s <a href="http://www.xml.com/pub/a/2002/12/18/dive-into-xml.html">What is RSS</a> and WebReference.com\'s <a href="http://www.webreference.com/authoring/languages/xml/rss/1/">The Evolution of RSS</a>.</p>
      <p>NOTE: Enable your site\'s XML syndication button by turning on the Syndicate block in block management.</p>
      <h3>Adding news feeds</h3>
      <p>To subscribe to an RSS feed on another site, use the <a href="%admin-news">aggregation page</a>.</p>
      <p>Once there, click the <a href="%new-feed">new feed</a> tab.  Drupal will then ask for the following:</p>
      <ul>
       <li><strong>Title</strong> -- The text entered here will be used in your news aggregator, within the administration configuration section, and as title for the news feed block. As a general rule, use the web site name from which the feed originates.</li>
       <li><strong>URL</strong> -- Here you\'ll enter the fully-qualified URL for the feed for the site you want to subscribe to.</li>
       <li><strong>Update interval</strong> -- The update interval is how often Drupal will automatically access the RSS URL for the site for fresh content. The 1 hour default is typically the minimum you will want to use. Accessing another site\'s RSS page more frequently can be considered impolite. After all, it does require the other site\'s server handle your requests. To use this feature cron.php must be called regularly, otherwise, you\'ll have to manually update feeds one at a time within the news aggregation administration section by using <a href="%update-items">update items</a>.</li>
       <li><strong>Latest items block</strong> -- The number of items selected here will determine how many of the latest items from the feed will appear in a block which may be enabled and placed in the <a href="%block">blocks</a> administration page.</li>
       <li><strong>Automatically file items</strong> -- As items are received from a feed they will be put in the checked categories.</li>
      </ul>
      <p>Once you submit your new feed, check to see if it is working properly. Select <a href="%update-items">update items</a> on the <a href="%admin-news">aggregation page</a>. If you do not see any items listed for that feed, edit the feed and make sure that the URL was entered correctly.</p>
      <h3>Adding categories</h3>
      <p>News items can be filed into categories. To create a category, start at the <a href="%admin-news">aggregation page</a>.</p>
      <p>Once there, select <a href="%new-category">new category</a> from the menu. Drupal will then ask for the following:</p>
      <ul>
       <li><strong>Title</strong> -- The title will be used in the <i>news by topics</i> listing in your news aggregator and with the customized block created for the bundle.</li>
       <li><strong>Description</strong> -- A short description of the category to tell users more details about what they might find filed in the category.</li>
       <li><strong>Latest items block</strong> -- The number of items selected here will determine how many of the latest items from the category will appear in a block which may be enabled and placed in the <a href="%block">blocks</a> administration page.</li>
      </ul>
      <h3>Using the news aggregator</h3>
      <p>The news aggregator has a number of ways that it displays your subscribed content:</p>
      <ul>
       <li><strong><a href="%news-aggregator">News aggregator</a></strong> (latest news) -- Displays all incoming content in the order received.</li>
       <li><strong><a href="%sources">Sources</a></strong> -- Organizes incoming content by feed, displaying feed titles which link to a page of the latest items from the feed and item titles which link to the originating post.</li>
       <li><strong><a href="%categories">Categories</a></strong> -- Organizes incoming content by category, displaying category titles which link to a page of the latest items from the category and item titles which link to the originating post.</li>
      </ul>
      <p>Pages which show the latest items in either a source, category, or of all items contain the following information for each item:
      <ul>
       <li>The title of the original post.</li>
       <li>The categories which the item is filed in, which are links to view the category page with items from that category only.</li>
       <li>A description, the first few paragraphs or summary of the originating post (if any).</li>
       <li>The name of the source, which acts as a link to an individual feed page, listing information about that feed and incoming content for that feed only. This is not shown on feed pages.</li>
      </ul>
      <p>Additionally, users with the administer news feeds permission will see a link to categorize the news items. Clicking this will add a right hand column to select the categories which each news item is in.</p>
      <h3>Technical details</h3>
      <p>Drupal automatically generates an OPML feed file that is available by selecting the XML icon on the News Sources page.</p>
      <p>When fetching feeds Drupal supports conditional GETs, this reduces the bandwidth usage for feeds that have not been updated since the last check.</p>
      <p>If a feed is permanently moved to a new location Drupal will automatically update the feed URL to the new address.</p>', array('%block' => url('admin/block'), '%admin-news' => url('admin/aggregator'), '%new-feed' => url('admin/aggregator/add/feed'), '%new-category' => url('admin/aggregator/add/category'), '%update-items' => url('admin/aggregator'), '%news-aggregator' => url('aggregator'), '%sources' => url('aggregator/sources'), '%categories' => url('aggregator/categories')));
    case 'admin/modules#description':
      return t('Used to aggregate syndicated content (RSS and RDF).');
    case 'admin/aggregator':
      return t('Several web sites, especially news related sites, syndicate parts of their site\'s content for other web sites to display. Usually, the syndicated content includes the latest headlines with a direct link to that story on the remote site. Some syndicated content also includes a description of the headline. The standard method of syndication is using the XML based Rich Site Summary (RSS) or Resource Description Framework (RDF).  To get a feed to work you <strong>must</strong> have <a href=\"%cron\">cron</a> working. To display the feed or category in a block you must decide how many items to show by editing the feed or block and turn on the <a href="%block">feed\'s block</a>.', array('%block' => url('admin/block'), '%cron' => url('admin/help', NULL, 'cron')));
    case 'admin/aggregator/add/feed':
      return t('Add a site that has an RSS/RDF feed. The URL is the full path to the RSS feed file. For the feed to update automatically you must run "cron.php". If you already have a feed with the URL you are planning to use, the system will not accept another feed with the same URL.');
    case 'admin/aggregator/add/category':
      return t('Categories provide a generalized way of creating composite news pages or blocks. They allow you, for example, to combine various sport-related items into one category called <i>Sports</i>. News items can be added to a category automatically by setting a feed to automatically place items in a category or by using the categorize items link in any listing of news items.');
    case 'admin/aggregator/configure':
      return t('These settings control the display of aggregated content.');
  }
}

/**
 * Menu callback; displays the aggregator-specific information from admin/help.
 */
function aggregator_help_page() {
  print theme('page', aggregator_help('admin/help#aggregator'));
}

function aggregator_configure() {
  if ($_POST) {
    system_settings_save();
  }

  $output = '';
  $number = drupal_map_assoc(array(5, 10, 15, 20, 25, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100));
  $items = array(0 => t('none'), 3 => t('3 items'), 5 => t('5 items'), 10 => t('10 items'), 15 => t('15 items'), 20 => t('20 items'), 25 => t('25 items'));

  $output .= form_select(t('Items shown in sources and categories pages'), 'aggregator_summary_items', variable_get('aggregator_summary_items', 3), $items, t('The number of items which will be shown with each feed or category in the feed and category summary pages.'));
  $output .= form_radios(t('Category selection type'), 'aggregator_category_selector', variable_get('aggregator_category_selector', 'check'), array('check' => t('checkboxes'), 'select' => t('multiple selector')), t('The type of category selection widget which is shown on categorization pages. Checkboxes are easier to use; a multiple selector is good for working with large numbers of categories.'));

  print theme('page', system_settings_form($output));
}

/**
 * Implementation of hook_perm().
 */
function aggregator_perm() {
  return array('administer news feeds', 'access news feeds');
}

/**
 * Implementation of hook_link().
 */
function aggregator_link($type) {
  if ($type == 'page' && user_access('access news feeds')) {
    return array(l(t('news feeds'), 'aggregator', array('title' => t('Read the latest news from syndicated web sites.'))));
  }
}

/**
 * Implementation of hook_menu().
 */
function aggregator_menu() {
  $items = array();

  $edit = user_access('administer news feeds');
  $view = user_access('access news feeds');

  $items[] = array('path' => 'admin/aggregator', 'title' => t('aggregator'),
    'callback' => 'aggregator_admin_overview', 'access' => $edit);
  $items[] = array('path' => 'admin/aggregator/edit/feed', 'title' => t('edit feed'),
    'callback' => 'aggregator_admin_edit_feed', 'access' => $edit,
    'type' => MENU_CALLBACK);
  $items[] = array('path' => 'admin/aggregator/edit/category', 'title' => t('edit category'),
    'callback' => 'aggregator_admin_edit_category', 'access' => $edit,
    'type' => MENU_CALLBACK);
  $items[] = array('path' => 'admin/aggregator/remove', 'title' => t('remove items'),
    'callback' => 'aggregator_admin_remove_feed', 'access' => $edit,
    'type' => MENU_CALLBACK);
  $items[] = array('path' => 'admin/aggregator/update', 'title' => t('update items'),
    'callback' => 'aggregator_admin_refresh_feed', 'access' => $edit,
    'type' => MENU_CALLBACK);

  $items[] = array('path' => 'admin/aggregator/list', 'title' => t('list'),
    'type' => MENU_DEFAULT_LOCAL_TASK, 'weight' => -10);
  $items[] = array('path' => 'admin/aggregator/add/feed', 'title' => t('add feed'),
    'callback' => 'aggregator_admin_edit_feed', 'access' => $edit,
    'type' => MENU_LOCAL_TASK);
  $items[] = array('path' => 'admin/aggregator/add/category', 'title' => t('add category'),
    'callback' => 'aggregator_admin_edit_category', 'access' => $edit,
    'type' => MENU_LOCAL_TASK);
  $items[] = array('path' => 'admin/aggregator/configure', 'title' => t('configure'),
    'callback' => 'aggregator_configure', 'access' => $edit,
    'type' => MENU_LOCAL_TASK);

  $items[] = array('path' => 'aggregator', 'title' => t('news aggregator'),
    'callback' => 'aggregator_page_last', 'access' => $view,
    'weight' => 5);
  $items[] = array('path' => 'aggregator/sources', 'title' => t('sources'),
    'callback' => 'aggregator_page_sources', 'access' => $view);
  $items[] = array('path' => 'aggregator/categories', 'title' => t('categories'),
    'callback' => 'aggregator_page_categories', 'access' => $view,
    'type' => MENU_ITEM_GROUPING);

  // To reduce the number of SQL queries, we don't query the database when
  // not on an aggregator page.
  // If caching of the menu is implemented, this check should be removed
  // so that DHTML menu presentation can be used correctly.
  if (arg(0) == 'aggregator') {
    // Sources:
    $result = db_query('SELECT title, fid FROM {aggregator_feed} ORDER BY title');
    while ($feed = db_fetch_object($result)) {
      $items[] = array('path' => 'aggregator/sources/'. $feed->fid, 'title' => $feed->title,
        'callback' => 'aggregator_page_source', 'access' => $view);
      $items[] = array('path' => 'aggregator/sources/'. $feed->fid .'/view', 'title' => t('view'),
        'type' => MENU_DEFAULT_LOCAL_TASK, 'weight' => -10);
      $items[] = array('path' => 'aggregator/sources/'. $feed->fid .'/categorize', 'title' => t('categorize'),
        'callback' => 'aggregator_page_source', 'access' => $edit,
        'type' => MENU_LOCAL_TASK);
      $items[] = array('path' => 'aggregator/sources/'. $feed->fid .'/configure', 'title' => t('configure'),
        'callback' => 'aggregator_edit', 'access' => $edit,
        'type' => MENU_LOCAL_TASK,
        'weight' => 1);
    }

    // Categories:
    $result = db_query('SELECT title, cid FROM {aggregator_category} ORDER BY title');
    while ($category = db_fetch_object($result)) {
      $items[] = array('path' => 'aggregator/categories/'. $category->cid, 'title' => $category->title,
        'callback' => 'aggregator_page_category', 'access' => $view);
      $items[] = array('path' => 'aggregator/categories/'. $category->cid .'/view', 'title' => t('view'),
        'type' => MENU_DEFAULT_LOCAL_TASK, 'weight' => -10);
      $items[] = array('path' => 'aggregator/categories/'. $category->cid .'/categorize', 'title' => t('categorize'),
        'callback' => 'aggregator_page_category', 'access' => $edit,
        'type' => MENU_LOCAL_TASK);
      $items[] = array('path' => 'aggregator/categories/'. $category->cid .'/configure', 'title' => t('configure'),
        'callback' => 'aggregator_edit', 'access' => $edit,
        'type' => MENU_LOCAL_TASK,
        'weight' => 1);
    }
  }

  $items[] = array('path' => 'aggregator/opml', 'title' => t('opml'),
    'callback' => 'aggregator_page_opml', 'access' => $view,
    'type' => MENU_CALLBACK);

  return $items;
}

/**
 * Implementation of hook_cron().
 *
 * Checks news feeds for updates once their refresh interval has elapsed.
 */
function aggregator_cron() {
  $result = db_query('SELECT * FROM {aggregator_feed} WHERE checked + refresh < %d', time());
  while ($feed = db_fetch_array($result)) {
    aggregator_refresh($feed);
  }
}

/**
 * Implementation of hook_block().
 *
 * Generates blocks for the latest news items in each category and feed.
 */
function aggregator_block($op, $delta) {
  if (user_access('access news feeds')) {
    if ($op == 'list') {
      $result = db_query('SELECT cid, title FROM {aggregator_category} WHERE block != 0 ORDER BY title');
      while ($category = db_fetch_object($result)) {
        $block['category:'. $category->cid]['info'] = t('%title category latest items', array('%title' => $category->title));
      }
      $result = db_query('SELECT fid, title FROM {aggregator_feed} WHERE block != 0 ORDER BY fid');
      while ($feed = db_fetch_object($result)) {
        $block['feed:'. $feed->fid]['info'] = t('%title feed latest items', array('%title' => $feed->title));
      }
    }
    else {
      list($type, $id) = split(':', $delta);
      switch ($type) {
        case 'feed':
          $feed = db_fetch_object(db_query('SELECT fid, title, block FROM {aggregator_feed} WHERE fid = %d', $id));
          $block['subject'] = $feed->title;
          $result = db_query_range('SELECT * FROM {aggregator_item} WHERE fid = %d ORDER BY timestamp DESC, iid DESC', $feed->fid, 0, $feed->block);
          $block['content'] = '<div class="more-link">'. l(t('more'), 'aggregator/sources/'. $feed->fid, array('title' => t('View this feed\'s recent news.'))) .'</div>';
          break;
        case 'category':
          $category = db_fetch_object(db_query('SELECT cid, title, block FROM {aggregator_category} WHERE cid = %d', $id));
          $block['subject'] = $category->title;
          $result = db_query_range('SELECT i.* FROM {aggregator_category_item} ci LEFT JOIN {aggregator_item} i ON ci.iid = i.iid WHERE ci.cid = %d ORDER BY i.timestamp DESC, i.iid DESC', $category->cid, 0, $category->block);
          $block['content'] = '<div class="more-link">'. l(t('more'), 'aggregator/categories/'. $category->cid, array('title' => t('View this category\'s recent news.'))) .'</div>';
          break;
      }
      $items = array();
      while ($item = db_fetch_object($result)) {
        $items[] = theme('aggregator_block_item', $item);
      }
      $block['content'] = theme('item_list', $items) . $block['content'];
    }
    return $block;
  }
}

function aggregator_remove($feed) {
  $result = db_query('SELECT iid FROM {aggregator_item} WHERE fid = %d', $feed['fid']);
  while ($item = db_fetch_object($result)) {
    $items[] = "iid = $item->iid";
  }
  if ($items) {
    db_query('DELETE FROM {aggregator_category_item} WHERE '. implode(' OR ', $items));
  }
  db_query('DELETE FROM {aggregator_item} WHERE fid = %d', $feed['fid']);
  db_query("UPDATE {aggregator_feed} SET checked = 0, etag = '', modified = 0 WHERE fid = %d", $feed['fid']);
  drupal_set_message(t('removed news items from \'%site\'.', array('%site' => $feed['title'])));
}

/**
 * Call-back function used by the XML parser.
 */
function aggregator_element_start($parser, $name, $attributes) {
  global $item, $element, $tag;

  switch ($name) {
    case 'IMAGE':
    case 'TEXTINPUT':
      $element = $name;
      break;
    case 'ITEM':
      $element = $name;
      $item += 1;
  }

  $tag = $name;
}

/**
 * Call-back function used by the XML parser.
 */
function aggregator_element_end($parser, $name) {
  global $element;

  switch ($name) {
    case 'IMAGE':
    case 'TEXTINPUT':
    case 'ITEM':
      $element = '';
  }
}

/**
 * Call-back function used by the XML parser.
 */
function aggregator_element_data($parser, $data) {
  global $channel, $element, $items, $item, $image, $tag;

  switch ($element) {
    case 'ITEM':
      $items[$item][$tag] .= $data;
      break;
    case 'IMAGE':
      $image[$tag] .= $data;
      break;
    case 'TEXTINPUT':
      // The sub-element is not supported. However, we must recognize
      // it or its contents will end up in the item array.
      break;
    default:
      $channel[$tag] .= $data;
  }
}

/**
 * Checks a news feed for new items.
 */
function aggregator_refresh($feed) {
  global $channel, $image;

  // Generate conditional GET headers.
  $headers = array();
  if ($feed['etag']) {
    $headers['If-None-Match'] = $feed['etag'];
  }
  if ($feed['modified']) {
    $headers['If-Modified-Since'] = gmdate('D, d M Y H:i:s', $feed['modified']) .' GMT';
  }

  // Request feed.
  $result = drupal_http_request($feed['url'], $headers);

  // Process HTTP response code.
  switch ($result->code) {
    case 304:
      db_query('UPDATE {aggregator_feed} SET checked = %d WHERE fid = %d', time(), $feed['fid']);
      drupal_set_message(t('no new syndicated content from "%site".', array('%site' => $feed['title'])));
      break;
    case 301:
      $feed['url'] = $result->redirect_url;
      watchdog('special', t('aggregator: updated URL for feed "%title" to %url', array('%title' => $feed[title], '%url' => $feed[url])));
    case 200:
    case 302:
    case 307:
      // Filter the input data:
     if (aggregator_parse_feed($result->data, $feed)) {

        if ($result->headers['Last-Modified']) {
          $modified = strtotime($result->headers['Last-Modified']);
        }

        /*
        ** Prepare the image data (if any):
        */

        foreach ($image as $key => $value) {
          $image[$key] = trim($value);
        }

        if ($image['LINK'] && $image['URL'] && $image['TITLE']) {
          $image = '<a href="'. $image['LINK'] .'"><img src="'. $image['URL'] .'" alt="'. $image['TITLE'] .'" /></a>';
        }
        else {
          $image = NULL;
        }

        /*
        ** Update the feed data:
        */

        db_query("UPDATE {aggregator_feed} SET url = '%s', checked = %d, link = '%s', description = '%s', image = '%s', etag = '%s', modified = %d WHERE fid = %d", $feed['url'], time(), strip_tags($channel['LINK']), strip_tags($channel['DESCRIPTION']), $image, $result->headers['ETag'], $modified, $feed['fid']);

        /*
        ** Clear the cache:
        */

        cache_clear_all();

        watchdog('regular', t('aggregator: syndicated content from "%site"', array('%site' => $feed[title])));
        $message = t('syndicated content from "%site".', array('%site' => $feed['title']));
        drupal_set_message($message);
      }
      break;
    default:
      watchdog('error', t('aggregator: failed to parse RSS feed "%site": %error', array('%site' => $feed['title'], '%error' => $result->code .' '. $result->error)));
      $message = t('failed to parse RSS feed "%site": %error.', array('%site' => $feed['title'], '%error' => $result->code .' '. $result->error));
      drupal_set_message($message);
  }
}

function aggregator_parse_feed(&$data, $feed) {
  global $items, $image, $channel;

  // Unset the global variables before we use them:
  unset($GLOBALS['element'], $GLOBALS['item'], $GLOBALS['tag']);
  $items = array();
  $image = array();
  $channel = array();

  // parse the data:
  $xml_parser = drupal_xml_parser_create($data);
  xml_set_element_handler($xml_parser, 'aggregator_element_start', 'aggregator_element_end');
  xml_set_character_data_handler($xml_parser, 'aggregator_element_data');

  if (!xml_parse($xml_parser, $data, 1)) {
    watchdog('error', t('aggregator: failed to parse RSS feed "%site": %error at line %line.', array('%site' => $feed['title'], '%error' => xml_error_string(xml_get_error_code($xml_parser)), '%line' => xml_get_current_line_number($xml_parser))));
    $message = t('failed to parse RSS feed "%site": %error at line %line.', array('%site' => $feed['title'], '%error' => xml_error_string(xml_get_error_code($xml_parser)), '%line' => xml_get_current_line_number($xml_parser)));
    drupal_set_message($message, 'error');
    return 0;
  }
  xml_parser_free($xml_parser);

  // initialize the translation table:
  $tt = array_flip(get_html_translation_table(HTML_SPECIALCHARS));
  $tt['&apos;'] = "'";

  /*
  ** We reverse the array such that we store the first item last,
  ** and the last item first.  In the database, the newest item
  ** should be at the top.
  */

  $items = array_reverse($items);

  foreach ($items as $item) {
    unset($title, $link, $author, $description);

    // Prepare the item:
    foreach ($item as $key => $value) {
      $item[$key] = filter_default(strtr(trim($value), $tt));
    }

    /*
    ** Resolve the item's title.  If no title is found, we use
    ** up to 40 characters of the description ending at a word
    ** boundary but not splitting potential entities.
    */

    if ($item['TITLE']) {
      $title = $item['TITLE'];
    }
    else {
      $title = preg_replace('/^(.*)[^\w;&].*?$/', "\\1", truncate_utf8($item['DESCRIPTION'], 40));
    }

    /*
    ** Resolve the items link.
    */

    if ($item['LINK']) {
      $link = $item['LINK'];
    }
    elseif ($item['GUID'] && (strncmp($item['GUID'], 'http://', 7) == 0)) {
      $link = $item['GUID'];
    }
    else {
      $link = $feed['link'];
    }

    /*
    ** Try to resolve and parse the item's publication date.  If no
    ** date is found, we use the current date instead.
    */

    if ($item['PUBDATE']) $date = $item['PUBDATE'];                        // RSS 2.0
    else if ($item['DC:DATE']) $date = $item['DC:DATE'];                   // Dublin core
    else if ($item['DCTERMS:ISSUED']) $date = $item['DCTERMS:ISSUED'];     // Dublin core
    else if ($item['DCTERMS:CREATED']) $date = $item['DCTERMS:CREATED'];   // Dublin core
    else if ($item['DCTERMS:MODIFIED']) $date = $item['DCTERMS:MODIFIED']; // Dublin core
    else $date = 'now';

    $timestamp = strtotime($date);  // strtotime() returns -1 on failure

    if ($timestamp < 0) {
      /*
      ** The Dublin core's default data format uses ISO 8601 which can't
      ** be parsed directly using PHP's strtotime().  It is not the only
      ** valid format so this might fail nonetheless ...
      */
      list($year, $month, $day, $hour, $minute, $second) = sscanf($date, '%4d-%2d-%2dT%2d:%2d:%2d');
      $timestamp = strtotime("$year-$month-$day $hour:$minute:$second");
    }

    if ($timestamp < 0) {
      $timestamp = time();
    }

    /*
    ** Save this item.  Try to avoid duplicate entries as much as
    ** possible.  If we find a duplicate entry, we resolve it and
    ** pass along it's ID such that we can update it if needed.
    */

    if ($link && $link != $feed['link'] && $link != $feed['url']) {
      $entry = db_fetch_object(db_query("SELECT iid FROM {aggregator_item} WHERE fid = %d AND link = '%s'", $feed['fid'], $link));
    }
    else {
      $entry = db_fetch_object(db_query("SELECT iid FROM {aggregator_item} WHERE fid = %d AND title = '%s'", $feed['fid'], $title));
    }

    if (!valid_input_data($item['DESCRIPTION'])) {
      drupal_set_message(t('failed to parse entry from "%site" feed: suspicious input data.', array('%site' => $feed['title'])), 'error');
    }
    else {
      aggregator_save_item(array('iid' => $entry->iid, 'fid' => $feed['fid'], 'timestamp' => $timestamp, 'title' => $title, 'link' => $link, 'author' => $item['AUTHOR'], 'description' => $item['DESCRIPTION']));
    }
  }

  /*
  ** Remove all items that are older than 3 months:
  */

  $age = time() - 1209600; // 3 month
  $result = db_query('SELECT iid FROM {aggregator_item} WHERE fid = %d AND timestamp < %d', $feed['fid'], $age);

  if (db_num_rows($result)) {
    $items = array();
    while ($item = db_fetch_object($result)) {
      $items[] = $item->iid;
    }
    db_query('DELETE FROM {aggregator_category_item} WHERE iid IN ('. implode(', ', $items) .')');
    db_query('DELETE FROM {aggregator_item} WHERE fid = %d AND timestamp < %d', $feed['fid'], $age);
  }

  return 1;
}

function aggregator_save_item($edit) {
  if ($edit['iid'] && $edit['title']) {
    db_query("UPDATE {aggregator_item} SET title = '%s', link = '%s', author = '%s', description = '%s' WHERE iid = %d", $edit['title'], $edit['link'], $edit['author'], $edit['description'], $edit['iid']);
  }
  else if ($edit['iid']) {
    db_query('DELETE FROM {aggregator_item} WHERE iid = %d', $edit['iid']);
    db_query('DELETE FROM {aggregator_category_item} WHERE iid = %d', $edit['iid']);
  }
  else if ($edit['title'] && $edit['link']) {
    $next_id = db_next_id('{aggregator_item}_iid');
    db_query("INSERT INTO {aggregator_item} (iid, fid, title, link, author, description, timestamp) VALUES (%d, %d, '%s', '%s', '%s', '%s', %d)", $next_id, $edit['fid'], $edit['title'], $edit['link'], $edit['author'], $edit['description'], $edit['timestamp']);
    // file the items in the categories indicated by the feed
    $categories = db_query('SELECT cid FROM {aggregator_category_feed} WHERE fid = %d', $edit['fid']);
    while ($category = db_fetch_object($categories)) {
      db_query('INSERT INTO {aggregator_category_item} (cid, iid) VALUES (%d, %d)', $category->cid, $next_id);
    }
  }
}

function aggregator_form_category($edit = array()) {
  $block_items = array(0 => t('no block'), 3 => t('3 items'), 5 => t('5 items'), 10 => t('10 items'), 15 => t('15 items'), 20 => t('20 items'), 25 => t('25 items'));

  $form = form_textfield(t('Title'), 'title', $edit['title'], 50, 64);
  $form .= form_textarea(t('Description'), 'description', $edit['description'], 60, 5);
  $form .= form_select(t('Latest items block'), 'block', $edit['block'], $block_items, t('If enabled, a block containing the latest items in this category will be available for placement on the <a href="%url">block configuration</a> page.', array('%url' => url('admin/block'))));
  $form .= form_submit(t('Submit'));

  if ($edit['cid']) {
    $form .= form_submit(t('Delete'));
    $form .= form_hidden('cid', $edit['cid']);
  }

  return form($form);
}

function aggregator_save_category($edit) {
  if ($edit['cid'] && $edit['title']) {
    db_query('UPDATE {aggregator_category} SET title = \'%s\', description = \'%s\', block = %d WHERE cid = %d', $edit['title'], $edit['description'], $edit['block'], $edit['cid']);
  }
  else if ($edit['cid']) {
    db_query('DELETE FROM {aggregator_category} WHERE cid = %d', $edit['cid']);
  }
  else if ($edit['title']) {
    // a single unique id for bundles and feeds, to use in blocks
    $next_id = db_next_id('{aggregator_category}_cid');
    db_query('INSERT INTO {aggregator_category} (cid, title, description, block) VALUES (%d, \'%s\', \'%s\', %d)', $next_id, $edit['title'], $edit['description'], $edit['block']);
  }
}

function aggregator_form_feed($edit = array()) {
  $period = drupal_map_assoc(array(900, 1800, 3600, 7200, 10800, 21600, 32400, 43200, 64800, 86400, 172800, 259200, 604800, 1209600, 2419200), 'format_interval');
  $block_items = array(0 => t('no block'), 3 => t('3 items'), 5 => t('5 items'), 10 => t('10 items'), 15 => t('15 items'), 20 => t('20 items'), 25 => t('25 items'));

  if ($edit['refresh'] == '') {
    $edit['refresh'] = 3600;
  }

  $form .= form_textfield(t('Title'), 'title', $edit['title'], 50, 64, t('The name of the feed; typically the name of the web site you syndicate content from.'));
  $form .= form_textfield(t('URL'), 'url', $edit['url'], 50, 255, t('The fully-qualified URL of the feed.'));
  $form .= form_select(t('Update interval'), 'refresh', $edit['refresh'], $period, t('The refresh interval indicating how often you want to update this feed.  Requires crontab.'));
  $form .= form_select(t('Latest items block'), 'block', $edit['block'], $block_items, t('If enabled, a block containing the latest items from this feed will be available for placement on the <a href="%url">block configuration</a> page.', array('%url' => url('admin/block'))));
  $categories = db_query('SELECT c.cid, c.title, f.fid FROM {aggregator_category} c LEFT JOIN {aggregator_category_feed} f ON c.cid = f.cid AND f.fid = %d ORDER BY title', $edit['fid']);
  while ($category = db_fetch_object($categories)) {
    $checkboxes .= form_checkbox($category->title, "category][$category->cid", 1, $category->fid ? 1 : 0);
  }
  if ($checkboxes) {
    $form .= form_group(t('Automatically file items'), $checkboxes, t('New items in this feed will be automatically filed in the the checked categories as they are received.'));
  }

  $form .= form_submit(t('Submit'));

  if ($edit['fid']) {
    $form .= form_submit(t('Delete'));
    $form .= form_hidden('fid', $edit['fid']);
  }

  return form($form);
}

function aggregator_save_feed($edit) {
  if ($edit['fid']) {
    // an existing feed is being modified, delete the category listings
    db_query('DELETE FROM {aggregator_category_feed} WHERE fid = %d', $edit['fid']);
  }
  if ($edit['fid'] && $edit['title']) {
    db_query('UPDATE {aggregator_feed} SET title = \'%s\', url = \'%s\', refresh = %d, block = %d WHERE fid = %d', $edit['title'], $edit['url'], $edit['refresh'], $edit['block'], $edit['fid']);
  }
  else if ($edit['fid']) {
    $result = db_query('SELECT iid FROM {aggregator_item} WHERE fid = %d', $edit['fid']);
    while ($item = db_fetch_object($result)) {
      $items[] = "iid = $item->iid";
    }
    if ($items) {
      db_query('DELETE FROM {aggregator_category_item} WHERE '. implode(' OR ', $items));
    }
    db_query('DELETE FROM {aggregator_feed} WHERE fid = %d', $edit['fid']);
    db_query('DELETE FROM {aggregator_item} WHERE fid = %d', $edit['fid']);
  }
  else if ($edit['title']) {
    // a single unique id for bundles and feeds, to use in blocks
    $edit['fid'] = db_next_id('{aggregator_feed}_fid');
    db_query('INSERT INTO {aggregator_feed} (fid, title, url, refresh, block) VALUES (%d, \'%s\', \'%s\', %d, %d)', $edit['fid'], $edit['title'], $edit['url'], $edit['refresh'], $edit['block']);
  }
  if ($edit['title']) {
    // the feed is being saved, save the categories as well
    if ($edit['category']) {
      foreach ($edit['category'] as $cid => $checked) {
        if ($checked) {
          db_query('INSERT INTO {aggregator_category_feed} (fid, cid) VALUES (%d, %d)', $edit['fid'], $cid);
        }
      }
    }
  }
}

function aggregator_get_feed($fid) {
  return db_fetch_array(db_query('SELECT * FROM {aggregator_feed} WHERE fid = %d', $fid));
}

function aggregator_get_category($cid) {
  return db_fetch_array(db_query('SELECT * FROM {aggregator_category} WHERE cid = %d', $cid));
}

function aggregator_view() {
  $result = db_query('SELECT f.*, COUNT(i.iid) AS items FROM {aggregator_feed} f LEFT JOIN {aggregator_item} i ON f.fid = i.fid GROUP BY f.fid, f.title, f.url, f.refresh, f.checked, f.link, f.description, f.etag, f.modified, f.image ORDER BY f.title');

  $output .= '<h3>'. t('Feed overview') .'</h3>';

  $header = array(t('title'), t('items'), t('last update'), t('next update'), array('data' => t('operations'), 'colspan' => 3));
  $rows = array();
  while ($feed = db_fetch_object($result)) {
    $rows[] = array(l($feed->title, "aggregator/sources/$feed->fid"), format_plural($feed->items, '1 item', '%count items'), ($feed->checked ? t('%time ago', array('%time' => format_interval(time() - $feed->checked))) : t('never')), ($feed->checked ? t('%time left', array('%time' => format_interval($feed->checked + $feed->refresh - time()))) : t('never')), l(t('edit feed'), "admin/aggregator/edit/feed/$feed->fid"), l(t('remove items'), "admin/aggregator/remove/$feed->fid"), l(t('update items'), "admin/aggregator/update/$feed->fid"));
  }
  $output .= theme('table', $header, $rows);

  $result = db_query('SELECT c.cid, c.title, count(ci.iid) as items FROM {aggregator_category} c LEFT JOIN {aggregator_category_item} ci ON c.cid = ci.cid GROUP BY c.cid ORDER BY title');

  $output .= '<h3>'. t('Category overview') .'</h3>';

  $header = array(t('title'), t('items'), t('operations'));
  $rows = array();
  while ($category = db_fetch_object($result)) {
    $rows[] = array(l($category->title, "aggregator/categories/$category->cid"), format_plural($category->items, '1 item', '%count items'), l(t('edit category'), "admin/aggregator/edit/category/$category->cid"));
  }
  $output .= theme('table', $header, $rows);

  return $output;
}

function aggregator_edit() {

  if ($_POST['op'] == t('Submit')) {
    if (arg(1) == 'categories') {
      aggregator_save_category($_POST['edit']);
      drupal_set_message(t('The category has been updated.'));
    }
    else {
      aggregator_save_feed($_POST['edit']);
      drupal_set_message(t('The feed has been updated.'));
    }

    drupal_goto($_GET['q']);
  }
  else if ($_POST['op'] == t('Delete')) {
    // Unset the title:
    unset($_POST['edit']['title']);

    if (arg(1) == 'categories') {
      aggregator_save_category($_POST['edit']);
      drupal_set_message(t('The category has been deleted.'));
    }
    else {
      aggregator_save_feed($_POST['edit']);
      drupal_set_message(t('The feed has been deleted.'));
    }

    drupal_goto('aggregator/'. arg(1));
  }

  if (arg(1) == 'categories') {
    $output = aggregator_form_category(aggregator_get_category(arg(2)));
  }
  else {
    $output = aggregator_form_feed(aggregator_get_feed(arg(2)));
  }

  print theme('page', $output);
}


/**
 * Menu callback; displays the category edit form, or saves changes and
 * redirects to the overview page.
 */
function aggregator_admin_edit_category($category = 0) {
  $edit = $_POST['edit'];
  $op = $_POST['op'];

  switch ($op) {
    case t('Delete'):
      $edit['title'] = 0;
      // Fall through:
    case t('Submit'):
      aggregator_save_category($edit);
      drupal_set_message($edit['title'] ? t('The category has been updated.') : t('The category has been deleted.'));
      drupal_goto('admin/aggregator');
      break;
    default:
      if ($category) {
        $output = aggregator_form_category(aggregator_get_category($category));
      }
      else {
        $output = aggregator_form_category();
      }
  }
  print theme('page', $output);
}

/**
 * Menu callback; displays the feed edit form.
 *
 * After editing, saves changes and redirects to the overview page.
 */
function aggregator_admin_edit_feed($feed = 0) {
  $edit = $_POST['edit'];
  $op = $_POST['op'];

  switch ($op) {
    case t('Delete'):
      $edit['title'] = 0;
      // Fall through:
    case t('Submit'):
      aggregator_save_feed($edit);
      drupal_set_message($edit['title'] ? t('The feed has been updated.') : t('The feed has been deleted.'));
      drupal_goto('admin/aggregator');
      break;
    default:
      if ($feed) {
        $output = aggregator_form_feed(aggregator_get_feed($feed));
      }
      else {
        $output = aggregator_form_feed();
      }
  }
  print theme('page', $output);
}

/**
 * Menu callback; removes all items from a feed, then redirects to the overview page.
 */
function aggregator_admin_remove_feed($feed) {
  aggregator_remove(aggregator_get_feed($feed));
  drupal_goto('admin/aggregator');
}

/**
 * Menu callback; refreshes a feed, then redirects to the overview page.
 */
function aggregator_admin_refresh_feed($feed) {
  aggregator_refresh(aggregator_get_feed($feed));
  drupal_goto('admin/aggregator');
}

/**
 * Menu callback; displays the aggregator administration page.
 */
function aggregator_admin_overview() {
  print theme('page', aggregator_view());
}

/**
 * Menu callback; displays the most recent items gathered from any feed.
 */
function aggregator_page_last() {
  _aggregator_page_list('SELECT i.*, f.title AS ftitle, f.link AS flink FROM {aggregator_item} i INNER JOIN {aggregator_feed} f ON i.fid = f.fid ORDER BY i.timestamp DESC, i.iid DESC', arg(1));
}

/**
 * Menu callback; displays all the items captured from a particular feed.
 */
function aggregator_page_source() {
  $feed = db_fetch_object(db_query('SELECT * FROM {aggregator_feed} WHERE fid = %d', arg(2)));
  $info = theme('aggregator_feed', $feed);

  _aggregator_page_list('SELECT * FROM {aggregator_item} WHERE fid = '. $feed->fid .' ORDER BY timestamp DESC, iid DESC', arg(3), "<div class=\"feed\">$info</div>");
}

/**
 * Menu callback; displays all the items aggregated in a particular category.
 */
function aggregator_page_category() {
  $category = db_fetch_object(db_query('SELECT cid, title FROM {aggregator_category} WHERE cid = %d', arg(2)));

  _aggregator_page_list('SELECT i.*, f.title AS ftitle, f.link AS flink FROM {aggregator_category_item} c LEFT JOIN {aggregator_item} i ON c.iid = i.iid LEFT JOIN {aggregator_feed} f ON i.fid = f.fid WHERE cid = '. $category->cid .' ORDER BY timestamp DESC, iid DESC', arg(3));
}

/**
 * Prints an aggregator page listing a number of feed items. Various
 * menu callbacks use this function to print their feeds.
 */
function _aggregator_page_list($sql, $op, $header = '') {
  if (user_access('administer news feeds') && $op == 'categorize') {
    if ($edit = $_POST['edit']) {
      foreach ($edit['categories'] as $iid => $selection) {
        db_query('DELETE FROM {aggregator_category_item} WHERE iid = %d', $iid);
        foreach ($selection as $cid) {
          if ($cid) {
            db_query('INSERT INTO {aggregator_category_item} (cid, iid) VALUES (%d, %d)', $cid, $iid);
          }
        }
      }
      drupal_set_message(t('The categories have been saved.'));
      drupal_goto($_GET['q']);
    }
    else {
      $categorize = true;
    }
  }

  $output = '<div id="aggregator">';
  if ($header) {
    $output .= $header;
  }
  if ($links) {
    $output .= theme('links', $links);
  }

  $result = pager_query($sql, 20);

  $rows = array();
  $categories = array();
  while ($item = db_fetch_object($result)) {
    if ($categorize) {
      $categories_result = db_query('SELECT c.cid, c.title, ci.iid FROM {aggregator_category} c LEFT JOIN {aggregator_category_item} ci ON c.cid = ci.cid AND ci.iid = %d', $item->iid);
      if (variable_get('aggregator_category_selector', 'check') == 'select') {
        $selected = array();
        while ($category = db_fetch_object($categories_result)) {
          if (!$done) {
            $categories[$category->cid] = check_form($category->title);
          }
          if ($category->iid) {
            $selected[] = $category->cid;
          }
        }
        $done = true;
        $form = form_select(NULL, 'categories]['. $item->iid, $selected, $categories, NULL, 'size="10"', true);
      }
      else {
        $form = '';
        while ($category = db_fetch_object($categories_result)) {
          $form .= form_checkbox(check_form($category->title), 'categories]['. $item->iid .'][', $category->cid, !is_null($category->iid));
        }
      }
      $rows[] = array(theme('aggregator_page_item', $item), array('data' => $form, 'class' => 'categorize-item'));
    }
    else {
      $output .= theme('aggregator_page_item', $item);
    }
  }
  if ($categorize) {
    $output .= form(theme('table', array('', t('categorize')), $rows) . form_submit(t('Save categories')));
  }
  $output .= '</div>';

  if ($pager = theme('pager', NULL, 20, 0)) {
    $output .= $pager;
  }

  print theme('page', $output);
}

/**
 * Menu callback; displays all the feeds used by the aggregator.
 */
function aggregator_page_sources() {
  $result = db_query('SELECT f.fid, f.title, f.description, f.image, MAX(i.timestamp) AS last FROM {aggregator_feed} f LEFT JOIN {aggregator_item} i ON f.fid = i.fid GROUP BY fid');
  $output = "<div id=\"aggregator\">\n";
  while ($feed = db_fetch_object($result)) {
    $output .= "<h2>$feed->title</h2>\n";

    // Most recent items:
    $list = array();
    if (variable_get('aggregator_summary_items', 3)) {
      $items = db_query_range('SELECT i.title, i.timestamp, i.link FROM {aggregator_item} i WHERE i.fid = %d ORDER BY i.timestamp DESC', $feed->fid, 0, variable_get('aggregator_summary_items', 3));
      while ($item = db_fetch_object($items)) {
        $list[] = theme('aggregator_summary_item', $item);
      }
    }
    $output .= theme('item_list', $list);
    $output .= '<div class="links">'. theme('links', array(l(t('more'), 'aggregator/sources/'. $feed->fid))) ."</div>\n";
  }
  $output .= theme('xml_icon', url('aggregator/opml'));
  $output .= '</div>';
  print theme('page', $output);
}

/**
 * Menu callback; generates an OPML representation of all feeds.
 */
function aggregator_page_opml() {
  $result = db_query('SELECT * FROM {aggregator_feed} ORDER BY title');

  $output = "<?xml version=\"1.0\" encoding=\"utf-8\"?>\n";
  $output .= "<opml version=\"1.1\">\n";
  $output .= "<head>\n";
  $output .= '<title>'. drupal_specialchars(variable_get('site_name', 'Drupal')) ."</title>\n";
  $output .= '<dateModified>'. gmdate('r') ."</dateModified>\n";
  $output .= "</head>\n";
  $output .= "<body>\n";

  while ($feed = db