XML Sitemaps Generator

Author Topic: bloked during crawl  (Read 21916 times)

franck.box

  • Registered Customer
  • Approved member
  • *
  • Posts: 3
bloked during crawl
« on: August 08, 2007, 04:39:17 AM »
hello,

I was using sitemap generator without problems until few days ago. Now the sitemap generator can crawl only a limited number of pages. I saw the error when I analysed the html sitemap generated. The sitemap display "You were blocked" for a lot of urls and the sitemap generated have only 130 pages instead thousands.

Can you help please?

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10625
Re: bloked during crawl
« Reply #1 on: August 08, 2007, 07:02:35 PM »
Hello,

perhaps your site blocks requests since sitemap generator crawls your site too intensively. Please try to use "Make delay for X seconds" option to insert delays between requests and make crawling slower.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

franck.box

  • Registered Customer
  • Approved member
  • *
  • Posts: 3
Re: bloked during crawl
« Reply #2 on: August 09, 2007, 04:08:23 AM »
Hello,

I resolved the problem removing a script I installed to prevent SQL injection, php injection and flood.  ;)

Thanks!

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10625
Re: bloked during crawl
« Reply #3 on: August 09, 2007, 09:09:51 PM »
I see, thank you for update :)
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

 

SMF 2.0.12 | SMF © 2014, Simple Machines
XHTML RSS WAP2