XML Sitemaps Generator

Author Topic: 15 million post  (Read 6511 times)

tony3433

  • Registered Customer
  • Approved member
  • *
  • Posts: 4
15 million post
« on: October 21, 2014, 02:04:23 PM »
Hello,

Just about to start using this powerful script although i bought it over 2yrs ago. My questions..

My new website is going to have about 15-18million post. which hints and tips will you offer me  to get the best out of this sofware considering this post size. Any particulat settings to help me get the best including server settings-e.g -max-excution, etc. The site will have 40 categories each category with a post of 300k.

Regards
Tony

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10625
Re: 15 million post
« Reply #1 on: October 22, 2014, 05:07:31 PM »
Hello,

with website of this size the best option is to create a limited sitemap - with "Maximum depth" or "Maximume URLs" option limited so that it would gather about 200-300,000 URLs, which would be main pages representing "roadmap" sitemap for search engines.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

tony3433

  • Registered Customer
  • Approved member
  • *
  • Posts: 4
Re: 15 million post
« Reply #2 on: October 22, 2014, 09:01:22 PM »
Thanks for the advise. Please which section of the script  below should i complete? and what should i put.?

  • Other Sitemap Types (click to expand)
  • Sitemap Entry Attributes (click to expand)
  • Miscellaneous Settings (click to expand)
  • Narrow Indexed Pages Set (click to expand)
  • Crawler Limitations, Finetune (click to expand)
  • Advanced Settings (click to expand)



Thanks

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10625
Re: 15 million post
« Reply #3 on: October 23, 2014, 08:20:35 AM »
Hello,

I'd recommend to keep all setting in default state, except for "Maximum pages" setting which will be limited in this case.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

tony3433

  • Registered Customer
  • Approved member
  • *
  • Posts: 4
Re: 15 million post
« Reply #4 on: October 23, 2014, 06:21:19 PM »
Thanks

tony3433

  • Registered Customer
  • Approved member
  • *
  • Posts: 4
Re: 15 million post
« Reply #5 on: October 23, 2014, 06:26:36 PM »
is it possible to have a url one can use with cron services  instead of the /usr/bin/php /home/site/public_html/generator/runcrawl.php   .  i use setcron for all my cron jobs and i normally have url as [external links are visible to admins only] for running the cron . is there any format to use with this service assuming the script is installed in  [external links are visible to admins only]

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10625
Re: 15 million post
« Reply #6 on: October 24, 2014, 04:28:08 AM »
Hello,

you can use http://www.example.com/generator/index.php?op=crawlproc&resume=1

However, command line cron task is recommended since it's running in less restricted environment usually.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

 

SMF 2.0.12 | SMF © 2014, Simple Machines
XHTML RSS WAP2