Sitemap Generator Forum

Hello,

my site has around 3000+ pages.
When we launch the crawling script, it stops at 150-200 pages.

We have tried to increase memory limit.
Webhoster says its limited to 70M - is this not enough ?

We have tried to increase max_execution_time settings
Webhoster says CPU execution is limited to 40seconds

We also tried CRON JOB
Webhoster says 1 CRON JOB is limited to 6 minutes

The webhoster mentioned that if i need more than the limits above they would make me an offer for a managed server. Now, that would mean, in order to run the XML sitemap script i would have to pay 100$ per month more for webhosting - that doesn't make sense, am sure there is a way to solve this.

Can you advise of what we could do to have the crawling run without interruptions ?
I thought of having the Cron Job activate the crawling script every week, when crawler is activated it should simply complete its job. Any special settings maybe that can make it work smoothly ?

Your answers to solve this will be highly appreciated, hope to hear from you soon, bye bye.

Hello,

please try to setup a cron task for sitemap generator - 6 minutes should be enough to create sitemap for your site.

Hi Oleg,

thanks for your response, but as mentioned above, i already tried it with cron job without success. You mentioned here that for 1000 pages it can take 16minutes, well i have above 3000+ so its most likely going to take above an hour: https://www.xml-sitemaps.com/forum/index.php/topic,4012.msg14437.html#msg14437

What other solution do you propose ?

I just launched 13 cron jobs yesterday for 2 hours, each 10mins. No success, nothing was generated.
I then tried to open /runcrawl.php directly in my browser to see if process gets started, there it says:
"This tool can be executed in command line mode only"

Does this mean that it won't work with cron jobs?

Please respond.

Well my webhoster says there is no PHP-CLI installed for shared webserver, i would need to get my own server. I mean that wouldn't make sense, just in order to have a sitemap script running properly to get a server...
I have other scripts that run well with cron jobs.

What can you suggest to get this working ?

Hello,

please let me know your generator URL/login in private message to check this.

Hi Oleg,

we are trying out this:
https://www.xml-sitemaps.com/forum/index.php/topic,3467.msg12811.html#msg12811

maybe this will solve our problem. But i still need to set up around 13 cron jobs, each on around 20mins intervals.

Hi Oleg, sent you a PM to have a look.

Hi Oleg,

This scenario happen to me last week where i hosted the websites at hostgator. The problem I'm facing is similar to yours which is the background process can't exceed 90 seconds.

What i've tried is that create a batch job to clear the temporary crawling data file every 4th, 9th, 14th,19th ... 59th minutes every hour. And then I've created a cron job to run every 5 munites to crawl the a site for 3 hours. which pretty much not so practical at all.

At last I put away the shared hosting at hostgator and go for a VPS server which I have more control of the hosting. Now the cron jobs runs very smooth.

For your information I have 35+ website (*expected to be 300+ websites within 2 months) with each of the website have average of 15,000 pages. Pretty much the VPS can handle the requests without any problem even though I run the cron job everyday (*now it's running once a week each website to give room for more websites).

Hope this help.

Hi,

we have also tried with like 20 cron jobs each set on 5 mins... But that has been a timely issue, of testing, trial and error. At the moment it wouldn't make sense for us yet to get a fully managed server. So what we will try now is: Sitemap Generator v4.0 and i hope this solves the problem. Sitemap generation isn't that simple after all - I can understand now why Google appreciates it so much when we the the structure of a big website is presented to them on a plate...

Bad News: we are having this problem with version 4.0:
https://www.xml-sitemaps.com/forum/index.php/board,3.html

I hope you can help us out Oleg, we are more than a week on this trying to make it work...

Hi,

we have also tried with like 20 cron jobs each set on 5 mins... But that has been a timely issue, of testing, trial and error. At the moment it wouldn't make sense for us yet to get a fully managed server. So what we will try now is: Sitemap Generator v4.0 and i hope this solves the problem. Sitemap generation isn't that simple after all - I can understand now why Google appreciates it so much when we the the structure of a big website is presented to them on a plate...

Fully manage server is costy, but for VPS you can get it around USD 25 /month which is kinda okay is you like full control of your hosting. No limtation on memory, cpu usage and also background process.

Hi, but VPS is not the right solution for me as I do not have the time to control my server - i have so many databases and scripts, cmses etc. running on my webhosting package. I'm sure there is a way to make this script work without the need to get an extra server

So back to my problem:

@Oleg:

Bad News: we are having this problem with version 4.0:
https://www.xml-sitemaps.com/forum/index.php/board,3.html

I hope you can help us out Oleg, we are more than a week on this trying to make it work...

Up! This issue is still pending, response required. Pls help.

Please try to re-download generator package, the issue should be corrected:
https://www.xml-sitemaps.com/forum/index.php/topic,802.msg14749.html#msg14749