XML Sitemaps Generator

Author Topic: Problem with website crawling  (Read 9616 times)

steve96

  • Registered Customer
  • Approved member
  • *
  • Posts: 4
Problem with website crawling
« on: October 10, 2013, 10:27:38 AM »
Can not crawl the website. Get this error:

HTTP Error 500.0 - Internal Server Error
d:\php\php-cgi.exe - The FastCGI process exceeded configured request timeout
Detailed Error Information
Module   FastCgiModule
Notification   ExecuteRequestHandler
Handler   php-5.2.6
Error Code   0x80070102
Requested URL   [external links are visible to admins only]
Physical Path   D:\Domains\CharityJobSC\Website\generator\index.php
Logon Method   Anonymous
Logon User   Anonymous
« Last Edit: October 10, 2013, 10:59:51 AM by steve96 »

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10624
Re: Problem with website crawling
« Reply #1 on: October 11, 2013, 07:42:53 AM »
Hello,

in case of Windows server it might be needed to increase script timeout in IIS configuration as described on:
http://forums.iis.net/p/1076662/1585343.aspx#1585343
http://www.iisadmin.co.uk/?p=7
 
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

steve96

  • Registered Customer
  • Approved member
  • *
  • Posts: 4
Re: Problem with website crawling
« Reply #2 on: October 11, 2013, 11:23:31 AM »
Thanks Oleg
We increased script timeout using FastCGI Settings, also increased Activity Timeout.
Now when I run crawling it starts working (no progress bar! but Page Title says Crawling:XML,ROR, Text...) and it never stops. How can I be sure its working properly? Should it list the directions which have already been crawled?

Thanks
Olga

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10624
Re: Problem with website crawling
« Reply #3 on: October 12, 2013, 01:12:14 PM »
Hello,

depending on server configuration the progress details could not be visible.
You can open generator crawling page in another browser and see progress details on that page.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

steve96

  • Registered Customer
  • Approved member
  • *
  • Posts: 4
Re: Problem with website crawling
« Reply #4 on: October 14, 2013, 09:41:07 AM »
Hi Oleg

We increased the timeout but get the error again. Also when I try to change settings in Configuration tab I get the timeout error as well and no settings were saved.
Does it mean that script is not working? I understand that configuration settings should be saved straight away?

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10624
Re: Problem with website crawling
« Reply #5 on: October 16, 2013, 01:27:03 PM »
Hello,

> Does it mean that script is not working?

That usually means that your browser keeps the session open with the website if you have started generator. It would work if you open another browser or completely close your browser (not just a tab) and open it again.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

steve96

  • Registered Customer
  • Approved member
  • *
  • Posts: 4
Re: Problem with website crawling
« Reply #6 on: October 16, 2013, 02:00:55 PM »
Hi Oleg

I do not think its a session problem. I closed the browser and now trying to save configuration settings. Its trying to connect but nothing happening and the settings are not saved. In few minutes it returns a timeout error.

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10624
Re: Problem with website crawling
« Reply #7 on: October 19, 2013, 07:22:10 AM »
Hello,

please let me know your generator URL/login in private message to check this.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

 

SMF 2.0.12 | SMF © 2014, Simple Machines
XHTML RSS WAP2