php error on running the generator.

[abhishek2]

Hi,
I have installed the generator as per the installation guide. when i run the Crawling url using the terminal i got the following error:-

PHP Warning:  fopen(/var/www/vhosts/mysite.com/httpdocs/generator/data/progress/07/0771e86d7cfe6418678b5df2318e35d6.txt): failed to open stream: No such file or directory in /var/www/vhosts/mysite.com/httpdocs/generator/pages/class.utils.inc.php on line 102

pls provide solution asap

[XML-Sitemaps Support]

Hello,

please check that you have "use temporary files to store crawling progress " setting disabled in generator configuration.

[abhishek2]

Hi,
Thanks... I'm not getting that error anymore. now I'm getting the new error

PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 13190440 bytes) in /var/www/vhosts/mysite.com/httpdocs/generator/pages/class.utils.inc.php on line 102

I read through posts related to the same error but could not get the solution. In most of the posts its mentioned as follows:

"Please try to increase memory_limit and max_execution_time settings in php configuration"

but in my case there is no option for me to increase the limit.

my site has more than 1.8 million pages which are dynamically created with depth of 2 or 3 and all are main pages.

I think by setting "use temporary files to store crawling progress" enabled would have minimized script memory usage. but after enabling it, script throws previous error.

following is the final break status:

Updated on 2015-04-26 06:59:19 (517 seconds ago) ,
Time elapsed: 2:33:51,
Pages crawled: 1398 (1396 added in sitemap),
Queued: 20581,
Depth level: 2
Current page: mysite-link/mysite-link/mysite-link/mysite-link/1985 (8.3)

pls provide solution asap 

[XML-Sitemaps Support]

Hello,

it looks like your server configuration doesn't allow to run the script long enough to create full sitemap. Please try to increase memory_limit and max_execution_time settings in php configuration at your host (php.ini file) or contact hosting support regarding this.
 

[XML-Sitemaps Support]

Hello,

with website of this size the best option is to create a limited sitemap - with "Maximum depth" or "Maximume URLs" option limited so that it would gather about 200-300,000 URLs, which would be main pages representing "roadmap" sitemap for search engines.

[abhishek2]

Hi,
Thanks for the help, it worked.