XML Sitemaps Generator

Author Topic: Problem with Crawling Website  (Read 6433 times)

Tiryaki

  • Registered Customer
  • Jr. Member
  • *
  • Posts: 22
Problem with Crawling Website
« on: November 18, 2014, 10:03:01 PM »
Hello,

the crawling of my website stops after a few minutes.
I have tried and changed some settings, but have still the problem.

Thanks in advance,
Tiryaki

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10624
Re: Problem with Crawling Website
« Reply #1 on: November 19, 2014, 10:21:43 AM »
Hello,

it looks like your server configuration doesn't allow to run the script long enough to create full sitemap. Please try to increase memory_limit and max_execution_time settings in php configuration at your host (php.ini file) or contact hosting support regarding this.
 
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

Tiryaki

  • Registered Customer
  • Jr. Member
  • *
  • Posts: 22
Re: Problem with Crawling Website
« Reply #2 on: November 19, 2014, 11:27:23 AM »
Hello,

max_execution_time is 60
memory_limit is 90

isn't that enough?

BR,
Tiryaki

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10624
Re: Problem with Crawling Website
« Reply #3 on: November 20, 2014, 05:20:38 AM »
Hello,

please try to increase it.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

Tiryaki

  • Registered Customer
  • Jr. Member
  • *
  • Posts: 22
Re: Problem with Crawling Website
« Reply #4 on: November 20, 2014, 11:19:38 AM »
Hello,

can't increase it, that is the maximum what my hosting can provide.

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10624
Re: Problem with Crawling Website
« Reply #5 on: November 20, 2014, 11:46:27 AM »
in this case I would recommend to run generator in command line if you have ssh access to your server.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

Tiryaki

  • Registered Customer
  • Jr. Member
  • *
  • Posts: 22
Re: Problem with Crawling Website
« Reply #6 on: November 20, 2014, 03:48:57 PM »
No, I have not.
Isn't there any alternative method?

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10624
Re: Problem with Crawling Website
« Reply #7 on: November 21, 2014, 02:42:51 PM »
Otherwise you would need to run generator multiple times, resuming the process until it finishes.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

Tiryaki

  • Registered Customer
  • Jr. Member
  • *
  • Posts: 22
Re: Problem with Crawling Website
« Reply #8 on: November 21, 2014, 02:57:32 PM »
Is it not possible that the script makes some brakes between the crawling or do it slowlier?

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10624
Re: Problem with Crawling Website
« Reply #9 on: November 22, 2014, 02:02:14 PM »
Hello,

there is a setting "Make delay for X seconds after each Y request" in generator configuration to slow down crawling process if it's needed.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

Tiryaki

  • Registered Customer
  • Jr. Member
  • *
  • Posts: 22
Re: Problem with Crawling Website
« Reply #10 on: November 23, 2014, 08:37:15 PM »
I have tried several settings without success, can you please have a look at it.
I will send you the link per pm.

Thanks in Advance.

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10624
Re: Problem with Crawling Website
« Reply #11 on: November 25, 2014, 07:59:50 AM »
Hello,

replied to your PM.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

 

SMF 2.0.12 | SMF © 2014, Simple Machines
XHTML RSS WAP2