Fatal Error Message

[orders6]

I get the following fata error message when trying to run a crawl.  Any ideas?

Links depth: 4
Current page: 2100_Lab_%20Meter_Specs.pdf
Pages added to sitemap: 80
Pages scanned: 180 (807.7 Kb)
Pages left: 278 (+ 52 queued for the next depth level)
Time passed: 0:00
Time left: 0:00
Memory usage: 670.7 Kb

Fatal error: Allowed memory size of 33554432 bytes exhausted (tried to allocate 13527364 bytes) in /home/httpd/vhosts/astisensor.com/httpdocs/Sitemap_Generator/pages/class.http.inc.php(2) : eval()'d code(1) : eval()'d code(1) : eval()'d code on line 59

[admin]

Hello!

As discussed here:
http://www.xml-sitemaps.com/forum/index.php/topic,130.0.html
http://www.xml-sitemaps.com/forum/index.php/topic,122.0.html

You should either:
- increase the memory_limit setting in php.ini
- limit the number of pages included in sitemap at Configuration page
- exclude a part of you pages from indexing with "Do not parse URLs" option

Also disabling the "HTML sitemap generation" and "Changelog calculation" reduces the memory usage a bit.

[orders6]

I forgot to remove the pdf files from getting parsed.  That solved the problem.  I only have not further question.  I have an extensive graphical set of imagemaps that are used to allow users to easy and graphically navigate to find their location with the terminal link ending in an html file with the appropriate contact information.  The starting page is located at [external links are visible to admins only] if you are interested to view it in action.  code from this starting page is posted below.  Is there any way that the href linkages contained within the image map could be indexed in the sitemap?  A sample of the code is given below.  This would be a terrific addition!!

<table cellpadding="0" cellspacing="0" align="center">
<tr>
<TD VALIGN=TOP>

<img SRC="worldaa.gif" BORDER="0" width="377" height="244" usemap="#worldaa">
<map name="worldaa">
<area shape="polygon" alt="North America " coords="159,32,117,36,48,49,11,65,58,53,50,68,38,87,45,105,48,107,27,116,34,129,67,129,73,121,85,109,94,114,129,103,128,100,83,97,81,86,101,74,114,68,115,53,129,53,147,44,156,37,157,32" href="na.htm">
<area shape="polygon" alt="South  America" coords="96,119,75,120,71,138,71,143,79,156,85,159,90,185,97,199,102,206,111,210,111,210,115,205,115,205,104,200,106,188,114,170,118,167,121,155,124,143,122,139,111,136,107,127,98,125,96,119" href="sa.htm">
<area shape="polygon" alt="Europe" coords="205,39,196,38,177,44,147,47,154,63,155,81,159,83,169,79,174,79,177,83,178,83,187,83,187,83,190,82,190,76,193,70,210,74,207,65,207,62,216,62,216,53,213,46,206,38,206,38" href="eu.htm">
<area shape="polygon" alt="Africa" coords="158,83,147,96,144,113,144,113,153,126,175,137,180,150,177,158,186,185,195,182,217,170,227,152,215,137,224,128,225,118,216,118,213,112,209,106,201,91,189,88,185,90,177,86,174,83,167,83" href="af.htm">
<area shape="polygon" alt="Middle East" coords="241,74,213,72,212,75,209,77,195,73,191,83,204,83,201,87,217,116,244,100,244,85,240,74" href="me.htm">
<area shape="polygon" alt="Asia"  coords="210,35,210,35,219,56,220,62,210,65,213,68,244,72,246,98,246,98,245,104,259,128,279,131,279,131,288,145,289,145,306,150,322,145,330,147,331,137,331,137,312,124,311,84,303,59,293,50,264,44,231,37,231,37,209,35" href="as.htm">
<area shape="polygon" alt="Australia/Oceania" coords="329,151,315,149,315,149,306,156,293,165,282,186,304,195,311,206,348,190,373,158,376,127,333,129,332,146,329,150,329,150" href="au.htm">
<area shape="rect" coords="3,95,12,108" href="na.htm">
<area shape="default" nohref></map></td>
  </tr>
</table>

[admin]

Hi,

that's great and thank you for the map area point.
Please try to redownload the generator (it's slightly modified) and let me know if it works fine for you now.

PS: however, it still better to include the "normal" links at your pages additinally to image maps to ensure they're crawled successfully by search engine bots.

[orders6]

New version worked perfect with the image maps href links.  Thank you for the feature addition!

[sales184]

sh-2.05$
1 | 66 | 61.9 | 0:03 | 3:32 | 1 | 616.1 Kb | 1 | 0 | 616
20 | 47 | 1,116.9 | 0:12 | 0:29 | 1 | 678.8 Kb | 16 | 446 | 62
60 | 7 | 2,114.1 | 0:26 | 0:03 | 1 | 720.4 Kb | 44 | 563 | 42
80 | 514 | 2,787.4 | 0:36 | 3:53 | 2 | 783.9 Kb | 63 | 169 | 63

...... ..... ......

3100 | 454 | 71,124.2 | 17:50 | 2:36 | 3 | 3,656.6 Kb | 1956 | 3951 | 18
3120 | 434 | 71,675.9 | 17:58 | 2:30 | 3 | 3,681.7 Kb | 1971 | 4021 | 25
3140 | 414 | 72,058.1 | 18:03 | 2:22 | 3 | 3,706.1 Kb | 1981 | 4044 | 25
3180 | 374 | 72,706.7 | 18:10 | 2:08 | 3 | 3,689.5 Kb | 1998 | 4070 | -17
3240 | 314 | 73,940.6 | 18:35 | 1:48 | 3 | 3,791.3 Kb | 2033 | 4125 | 102
Allowed memory size of 8388608 bytes exhausted (tried to allocate 40 bytes)

I've deselected html generation and also calculate changelog is off.  I also changed the 'xs_sm_size' => '2000' from 50000.

Shouldn't runcrawl do the sitemap in chunks?  Seems like it is trying to do the whole crawl in memory at once?

=======================
Update:

I increased memory from 8MB to 32MB in php.ini.  Hopefully that does it!

memory_limit = 32M      ; Maximum amount of memory a script may consume

[admin]

Hello,

yes, increasing memory_limit should help.

[aka.vexx]

Hey,

I just installed the script today and it gives me the following error after let's say 10 min of activity

Allowed memory size of 67108864 bytes exhausted (tried to allocate 15993278 bytes) in /home/fileclus/public_html/generator/pages/class.utils.inc.php(2) : eval()'d code on line 8

[aka.vexx]

momory_limit of 64M here, no ror, html and text sitemaps enabled...still after a while it gives me the out of memory message....php5, apache 2.2....quad core 4 gig system runnign redhat ent 5

[aka.vexx]

update...raised memory_limit to 128M from 64...script runs longer but still the same error...also I have disabled html, text, logging and ror sitemaps...nothing

played with var_export and serialize options...same thing...so..basically i disabled everything, tried everything...doesn't work..now what?

[aka.vexx]

update...raised memory_limit to 128M from 64...script runs longer but still the same error...also I have disabled html, text, logging and ror sitemaps...nothing

played with var_export and serialize options...same thing...so..basically i disabled everything, tried everything...doesn't work..now what?

[aka.vexx]

another update...i have raised it to 256Mb...the script runs for about 1h without erros...but in the end..the same error...

[admin]

Hello,

how many URLs do you have crawled at that point?