Parser runs out of memory on large codebases

Migrated issue

Reported by: rickvug

PHP Fatal error:  Allowed memory size of 536870912 bytes exhausted (tried to allocate 162 bytes) in /vagrant/repos/apisite/includes/file.inc on line 960

I've successfully setup API and successfully parsed Drupal 7 core as a test. However when I try to parse the code of a D7 distribution with many contrib modules cron continually runs out of memory. It does not get to the point where the files are added to the job queue for parsing. My assumption at this point is that all of the time is spent in api_scan_directories(), which needs to build the entire array of file names in a single pass.

Any ideas on how to work around the problem or improve api_scan_directories()?