We added about 30000 products to our site and now the generator is not working. It starts crawling but stops part way through and prints an array below the crawling info. I then have to delete the files in the Data folder for it to run again. It seems to be stopping at just under 12,000 Kb memory usage, but our php memory limit has been upped to 32 mb for another script we run.
You can see our PHP setting at [external links are visible to admins only]
Here is what is printed on the crawling page (I cut the middle out because it was too huge):
Links depth: 3
Current page: prod/frames-picture-miscellaneous/5240764
Pages added to sitemap: 4742
Pages scanned: 5280 (237,213.7 Kb)
Pages left: 28390 (+ 2743 queued for the next depth level)
Time passed: 26:08
Time left: 140:32
Memory usage: 11,971.4 Kb
array ( 'url_ind' => 3040, 'urls_list' => array ( 0 => 'sitewide/images/prod/0-postitnotes-postitpads-valuepriced34postit_lg.jpg', 1 => 'sitewide/images/prod/0-postitnotes-postitpads-234x3postitnote_lg.jpg', 2 => 'postitnotes/bicstickynotes/bicstickynote3x3', 3 => 'postitnotes/bicstickynotes/bicstickynote2-1_2x3', 4 => 'adhesivenotes/bicpadsgeneralinformation', 5 => 'postitnotes/postitpads/www.imprintitems.com/adhesivenotes/postitnotesgeneralinformation', 6 => 'postitnotes/bicstickynotes/3inchhalfbiccube', 7 => 'postitnotes/bicstickynotes/3inchbiccube', 8 => 'postitnotes/bicstickynotes/2-3/4biccube', 9 => 'postitnotes/bicstickynotes/bicstickynote8x3', 10 =>
**********************************A BUNCH (more than I could copy) MORE LINES WERE HERE*****************************************************************
1 => array ( 0 => 'postitnotes/postitpads/www.imprintitems.com/adhesivenotes/postitnotesgeneralinformation', 1 => 'postitnotes/bicstickynotes', ), 2 => array ( 0 => '[external links are visible to admins only]', 1 => 'blog/', ), 3 => array ( 0 => 'blog/wp-admin/www.imprintitems.com/flyswatters.htm', 1 => 'blog/', ), 4 => array ( 0 => '
sales@imprintitems.com', 1 => 'bobbleheads_Custom_photos.htm', ), 5 => array ( 0 => 'pictureframes/pictureframes', 1 => 'pictureframes/pictureframes/woodpictureframes', ), 6 => array ( 0 => 'lapelpins/christian/christianflag', 1 => 'lapelpin3.htm', ), 7 => array ( 0 => 'lapelpins/christian/TENCOMMANDMENTSPIN', 1 => 'lapelpin3.htm', ), ), 'nt' => 178, 'tsize' => 242906838, 'pn' => 5280, 'links_level' => 3, 'ctime' => 1568.2806270123, 'time' => 1191864822, )
