site map eats memory + creats array on crawling page
« on: August 28, 2007, 05:51:15 PM »
Script problems

when i first loaded the script it worked great it crawled all 49K plus pages without a problem. it didnt show any info about memory the first time i ran the script.now however it seems to eat memory like  alzheimer's  and screw up and create the array in the crawling page

the memory problem has been sorted but I dont understand how it can not have a problem on first install, but when trying to run it a second time has a problem. I have redownloaded and reinstalled the script
example of what im getting on the crewling page:

Links depth: 5
Current page: index.php?Title=PNY+flash+memory+card+-+512+MB+-+miniSD&pid=0000081282
Pages added to sitemap: 36680
Pages scanned: 36680 (6,788.5 Kb)
Pages left: 3733 (+ 5004 queued for the next depth level)
Time passed: 27:03
Time left: 2:45
Memory usage: 32,248.0 Kb
array ( 'url_ind' => 6268, 'urls_list' => array ( 0 => 'index.php?Title=Joe+Browns+Combat+Jacket&pid=0000054306', 1 => 'index.php?Title=Joe+Browns+Cote+D%27Azur+Cami&pid=0000054307', 2 => 'index.php?Title=Joe+Browns+Crazy+Crinkle+Cami&pid=0000054308', 3 => 'index.php?Title=Joe+Browns+Crochet+Dress&pid=0000054309', 4 => 'index.php?Title=Joe+Browns+Crop+Crush+Shirt&pid=0000054310', 5 => 'index.php?Title=Joe+Browns+Crop+Summer+Shirt&pid=0000054311', 6 => 'index.php?Title=Joe+Browns+Cross+Pebble+Necklace&pid=0000054312', 7 => 'index.php?Title=Joe+Browns+Crossover+Tee&pid=0000054313', 8 => 'index.php?Title=Joe+Browns+Crown+Belt&pid=0000054314', 9 => 'index.php?Title=Joe+Browns+Crucifix+T-


I would like to get it back to the state when i first installed it where it just did what it was supposed to do . any help would be greatly appriciated.
Re: site map eats memory + creats array on crawling page
« Reply #1 on: August 28, 2007, 09:31:09 PM »
Hello,
Quote
now however it seems to eat memory like  alzheimer's  and screw up and create the array in the crawling page
did you change anything at that point?
Re: site map eats memory + creats array on crawling page
« Reply #2 on: August 28, 2007, 09:35:33 PM »
no, the only thing that changed was i added more pages to the site
Re: site map eats memory + creats array on crawling page
« Reply #3 on: August 28, 2007, 09:53:16 PM »
ive also reinstalled the script from scratch and still the same, the crawl_dump.log seems to be writern to, should there be any other files in the data directory ? theres the placeholder file and dump_log
Re: site map eats memory + creats array on crawling page
« Reply #4 on: August 29, 2007, 10:43:44 PM »
Hello,

crawl_dump.log is a temporary file that sitemap generator creates and updated during crawling. Please let me know your generator URL in private message so that I can check it.
Re: site map eats memory + creats array on crawling page
« Reply #5 on: August 31, 2007, 10:22:04 AM »
tried to send a PM to admin dont know if it worked as nothing in outbox