Sitemap Generator Forum

When I run manually crawling through web interface there are always in the chngelog listed New and Removed urls,  but when it is done by cronthe log alwas shows "job done" but ZERO for both NEW and Removed urls.

Manually run crawling (web interface) New URLs 22 / Removed 196

12    2007-12-21 09:02    500    721.70s    15.32    22    196    4

Cronjob: New URLs 0 / Removed 0

13    2007-12-21 12:02    500    737.14s    15.32    0    0    4

Any idea why?

Hello,

don't you have the limit for maximum number of URLs to index in configuration?

Yes, I do and it is 500, but I can run manually crawling through web interface every 1 hour and there will a number of New and Removed urls but cron every 4 hours shows ZERO for both NEW and Removed urls.

So, what is the diference between running manually crawling through web interface and cronjob?

Hello,

don't you have the limit for maximum number of URLs to index in configuration?

Well, I though it is good idea to keep the limit down to 500. I run busy website (news, classifieds and so) and cron does the job every 4 hours (which seem to me OK)

The website has thousands links.

Should I put ZERO for Maximum pages or there is a difference when the job is done by cron and web interface (do not have any problem with web interface, works perfect.)

Hello,

in this case Sitemap Generator indexes the *first spotted* pages, which may differ from time to time in case of dynamic website.

Yes, I would suggest to remove the limit (or increase it) and there is no need to re-generate sitemap every 4 hours, daily cron job should be more than enough.