XML Sitemaps Generator

Author Topic: Question  (Read 17379 times)

keny

  • Approved member
  • *
  • Posts: 1
Question
« on: May 18, 2008, 08:38:46 PM »
Hi, i would like to buy this script but first i need to know if i can set a cron job to crawl 50 000 pages a day and resume them for next day until the crawl end (i have about 10 millions different pages on my web site)

And what for the new pages ? can i make a another job that will only crawl page that have been created for the last day ?

Bassicly my site change each minute whit about 5-25 news pages.

Thanks

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10624
Re: Question
« Reply #1 on: May 19, 2008, 09:54:34 PM »
Hello,

sitemap generator cannot find "new" pages unless the whole site is recrawled.
You can setup a "maximum script running time" in configuration and setup a cron job, so it will work for say 30 minutes every time, then interrupt and resume the next day.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

 

SMF 2.0.12 | SMF © 2014, Simple Machines
XHTML RSS WAP2