XML Sitemaps Generator

Author Topic: How long will it take to crawl a site with over 100K pages?  (Read 13855 times)

am1

  • Registered Customer
  • Jr. Member
  • *
  • Posts: 17
How long will it take to crawl a site with over 100K pages?
« on: February 17, 2012, 01:20:26 PM »
Purchsed the standalone version on the 15th and installed it right away.

Started the crawl and it is far from complete...

Here's the info

Already in progress. Current process state is displayed:
Links depth: 4
Current page: Venues/Southeast-Texas-Entertainment-Complex-Tickets.html
Pages added to sitemap: 38220
Pages scanned: 39180 (552,008.5 KB)
Pages left: 5388 (+ 6846 queued for the next depth level)
Time passed: 8:14:08
Time left: 1:07:57
Memory usage: 85,702.9 Kb
Auto-restart monitoring: Fri Feb 17 08:18:42 EST 2012 (38 second(s) since last update)

below it states

Fri Feb 17 08:07:32 EST 2012: resuming generator (121 seconds with no response)

Please advise

TY

am1

  • Registered Customer
  • Jr. Member
  • *
  • Posts: 17
PHP Memory Limit
« Reply #1 on: February 17, 2012, 01:47:44 PM »
What would be the memory limit recommended for a site with over 100K pages?

Currently set at 60

[external links are visible to admins only]

Any other settings that should be changed so the script can work at optimal performance?

Please advise

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10624
Re: How long will it take to crawl a site with over 100K pages?
« Reply #2 on: February 18, 2012, 09:33:05 AM »
Hello,

please let me know your generator URL/login in private message to check this.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10624
Re: PHP Memory Limit
« Reply #3 on: February 18, 2012, 09:33:24 AM »
I would recommend setting it to 256M.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

am1

  • Registered Customer
  • Jr. Member
  • *
  • Posts: 17
Re: How long will it take to crawl a site with over 100K pages?
« Reply #4 on: February 19, 2012, 01:36:18 PM »
Private message sent with url/login

[external links are visible to admins only]

am1

  • Registered Customer
  • Jr. Member
  • *
  • Posts: 17
Re: How long will it take to crawl a site with over 100K pages?
« Reply #5 on: February 20, 2012, 07:59:39 PM »
php.ini updated to 256M

Still have the same issue

Mon Feb 20 14:55:48 EST 2012: resuming generator (120 seconds with no response)
Mon Feb 20 14:53:48 EST 2012: resuming generator (120 seconds with no response)
Mon Feb 20 14:51:48 EST 2012: resuming generator (120 seconds with no response)
Mon Feb 20 14:49:48 EST 2012: resuming generator (120 seconds with no response)
Mon Feb 20 14:47:48 EST 2012: resuming generator (120 seconds with no response)
Mon Feb 20 14:45:48 EST 2012: resuming generator (120 seconds with no response)
Mon Feb 20 14:43:47 EST 2012: resuming generator (120 seconds with no response)
Mon Feb 20 14:41:47 EST 2012: resuming generator (120 seconds with no response)

5 Days working and the site still not even close to being completely crawled

Please advise

am1

  • Registered Customer
  • Jr. Member
  • *
  • Posts: 17
Re: PHP Memory Limit
« Reply #6 on: February 20, 2012, 08:00:41 PM »
Set at 256M and still not able to crawl the site completely

Please advise

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10624
Re: How long will it take to crawl a site with over 100K pages?
« Reply #7 on: February 21, 2012, 02:48:24 PM »
Looks like sitemap has been created:
Request date:
21 February 2012, 12:24
Processing time:
8:08:28s
Pages indexed:
43511
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

am1

  • Registered Customer
  • Jr. Member
  • *
  • Posts: 17
Re: How long will it take to crawl a site with over 100K pages?
« Reply #8 on: February 24, 2012, 01:30:32 AM »
the site has over 200K pages... only 43511 indexed


Links depth: 3
Current page: 1791169/cleveland-cavaliers-vs-milwaukee-bucks-tickets.html
Pages added to sitemap: 28600
Pages scanned: 29020 (768,064.7 KB)
Pages left: 61963 (+ 11937 queued for the next depth level)
Time passed: 3:47:15
Time left: 8:05:13
Memory usage: 123,780.1 Kb
Auto-restart monitoring: Thu Feb 23 20:33:17 EST 2012 (23 second(s) since last update)Resuming the last session (last updated: 2012-02-23 02:00:08)

Just here with Links deph 3 it shows it has 61963 left to go with 29020 already scanned
but it nevers completes

I will be glad to pay you to install it if you think it will help

Otherwise I will have to ask for a refund

Please advise
« Last Edit: February 24, 2012, 01:36:47 AM by am1 »

brades

  • Registered Customer
  • Approved member
  • *
  • Posts: 1
Re: How long will it take to crawl a site with over 100K pages?
« Reply #9 on: February 25, 2012, 05:03:38 PM »
I am having the same problem.  :-\

am1

  • Registered Customer
  • Jr. Member
  • *
  • Posts: 17
Re: How long will it take to crawl a site with over 100K pages?
« Reply #10 on: February 26, 2012, 02:51:47 AM »
Please advise

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10624
Re: How long will it take to crawl a site with over 100K pages?
« Reply #11 on: February 27, 2012, 06:16:27 AM »
Hello,

could you please PM me your generator URL and an example URL that is not included in sitemap and how it can be reached starting from homepage?
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

am1

  • Registered Customer
  • Jr. Member
  • *
  • Posts: 17
Re: How long will it take to crawl a site with over 100K pages?
« Reply #12 on: February 27, 2012, 12:54:57 PM »
I already sent a PM with the details to investigate the issue over a week ago.

[external links are visible to admins only]

Access info already sent as a PM

[external links are visible to admins only]

and all the pages in this folder can't be found in the sitemap as well as many other pages

These pages are linked from the home page

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10624
Re: How long will it take to crawl a site with over 100K pages?
« Reply #13 on: February 28, 2012, 04:59:57 PM »
You have this folder excluded with your robots.txt file:
Disallow: /tickets/
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

am1

  • Registered Customer
  • Jr. Member
  • *
  • Posts: 17
Re: How long will it take to crawl a site with over 100K pages?
« Reply #14 on: February 28, 2012, 07:58:01 PM »
Now it is.

Since yesterday due to an issue with google...

 

 

SMF 2.0.12 | SMF © 2014, Simple Machines
XHTML RSS WAP2