XML Sitemaps Generator

Author Topic: Massive E-commerce site  (Read 5778 times)

nicepeople

  • Registered Customer
  • Approved member
  • *
  • Posts: 1
Massive E-commerce site
« on: October 08, 2013, 08:18:38 AM »
Hi.

I have a site with almost 4 million pages. I seem to be running into some difficulties.

It had taken almost 2 hours to run before stopping and had only found 700 pages approx. I think it is having problems with certain page types and I would appreciate some direction.

Thanks
Conor

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10622
Re: Massive E-commerce site
« Reply #1 on: October 08, 2013, 08:30:30 PM »
Hello,

with website of this size the best option is to create a limited sitemap - with "Maximum depth" or "Maximume URLs" option limited so that it would gather about 100-200,000 URLs, which would be main pages representing "roadmap" sitemap for search engines.

The crawling time itself depends on the website page generation time mainly, since it crawls the site similar to search engine bots.
For instance, if it it takes 1 second to retrieve every page, then 1000 pages will be crawled in about 16 minutes.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

 

SMF 2.0.12 | SMF © 2014, Simple Machines
XHTML RSS WAP2