Toggle navigation
PRO Sitemaps
Standalone Generator
About Sitemaps
News
SEO Tools
Sitemap Generator Forum
Guest
Login
Register
Menu
Home
Help
Search
Login
Register
XML Sitemaps Discussions Category
/
Unlimited PHP Sitemap Generator
/
Crawling one page every 1 or 2 seconds
Tools for Any SEO Challenge:
Get more traffic to your website with versatile solutions for every step of your SEO journey.
« previous
next »
Pages:
1
william1
3
Crawling one page every 1 or 2 seconds
«
on:
March 19, 2010, 12:15:20 AM »
Hi All
My crawls are taking forever and after looking at teh access logs i see that a page is being requested every 1 or 2 seconds. how can i speed this up?
Ww
Logged
XML-Sitemaps Support
11775
Re: Crawling one page every 1 or 2 seconds
«
Reply #1 on:
March 19, 2010, 12:30:39 PM »
Hello,
sitemap generator has a built-in crawler and it requests the page after the previous one has been processed, so in case if page load time is about 1 second, then that's a time period between requests.
Logged
Oleg Ignatiuk
https://www.xml-sitemaps.com
Send me a Private Message
SEM and SEO Reports, more than 45M domains
: The world's leading Competitive Intelligence Tool for digital marketing.
william1
3
Re: Crawling one page every 1 or 2 seconds
«
Reply #2 on:
March 19, 2010, 03:26:32 PM »
Thanks for the pointers, i will see what i can do to improve me site speed.
Is there a way of disabling this as my server has very little load on it at the moment.
Ww
Logged
XML-Sitemaps Support
11775
Re: Crawling one page every 1 or 2 seconds
«
Reply #3 on:
March 20, 2010, 08:17:03 AM »
That is the way generator uses to find all pages on the site, it cannot be disabled.
Logged
Oleg Ignatiuk
https://www.xml-sitemaps.com
Send me a Private Message
SEM and SEO Reports, more than 45M domains
: The world's leading Competitive Intelligence Tool for digital marketing.
Pages:
1
« previous
next »