XML Sitemaps Generator

Author Topic: Generator cannot manage large number of links  (Read 7347 times)

etienne_stephane

  • Registered Customer
  • Approved member
  • *
  • Posts: 2
Generator cannot manage large number of links
« on: December 20, 2014, 04:42:59 PM »
My generator stop running after about crawling 40-50k pages (site has about half million pages) and when I try to start it again, I don't receive any message, just thrown back to ssh command line.

I've increase memory to 1GB and running crawler from ssh.

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10623
Re: Generator cannot manage large number of links
« Reply #1 on: December 20, 2014, 07:47:43 PM »
Hello,

with website of this size the best option is to create a limited sitemap - with "Maximum depth" or "Maximume URLs" option limited so that it would gather about 200-300,000 URLs, which would be main pages representing "roadmap" sitemap for search engines.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

etienne_stephane

  • Registered Customer
  • Approved member
  • *
  • Posts: 2
Re: Generator cannot manage large number of links
« Reply #2 on: December 21, 2014, 02:41:12 PM »
Ok. But how to get 200-300k links when generator stops working on 40-50k and whatever I try, it just got back to command line without any report.

P.S. This 40-50k links are in depth of 3.

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10623
Re: Generator cannot manage large number of links
« Reply #3 on: December 22, 2014, 10:16:29 AM »
Hello,

please let me know your generator URL/login in private message to check this.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

 

SMF 2.0.12 | SMF © 2014, Simple Machines
XHTML RSS WAP2