XML Sitemaps Generator

Author Topic: Crawled links  (Read 48098 times)


  • Registered Customer
  • Approved member
  • *
  • Posts: 8
Crawled links
« on: July 07, 2005, 12:47:12 PM »
I crwaled two  of my sites, once using the scipt installed at my server and once by your online script at www.xml-sitemaps.com.

first site
Your free script : crawled 435 url
My script : crawled 98 url only and gave completed message

second site :
your free script: crawled 431
My script : crawled 484 url

while in a third site it stops after crawling 2000 url, so I had to split crawling to crawl all my site.

Do yo have an explanation for this matter.
Thank you

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10601
Re: Crawled links
« Reply #1 on: July 07, 2005, 02:11:29 PM »

1. Although both online generator and standalone script use the same crawler core, we're updating them separately from time to time, so there may be some difference :) The general to find out what wrong with listing is to find the URLs that are listed in one case and are not in other.
2. It is possible that crawling stops because the script can't extend the execution time. Although default php configuration allows to do it, some hosting environments disable this function and limit the exectution to certain time limit.
Oleg Ignatiuk
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.


SMF 2.0.12 | SMF © 2014, Simple Machines