XML Sitemaps Generator

Author Topic: I can not get the sitemap to crawl my website.  (Read 21908 times)

Tech-Mentor

  • Registered Customer
  • Approved member
  • *
  • Posts: 2
I can not get the sitemap to crawl my website.
« on: May 25, 2007, 10:23:08 PM »
I have database driven dynamic website. I can not get the site-map to work. The most I got was about 14000 pages. I got up to 14k by restarting from the last saved point several times. But the site-map never saved that file. All my other attempts have been successful for about 2 - 3k. Thats it.  The crawler gets stuck. I have left it running overnight. I have experimented with setting different limits and other available options. I always get the same results. It just stops working. I even reinstalled the site-map on a different server.

Right now I am very frustrated with this product. I think I am doing everything I suppose to.

Can someone please explain the entire process to crawl a large website?

Tech-Mentor

  • Registered Customer
  • Approved member
  • *
  • Posts: 2
Re: I can not get the sitemap to crawl my website.
« Reply #1 on: May 27, 2007, 06:22:57 PM »
I wanted to update my latest attempt to create a site map. The crawler seems to be stuck at 3560 pages and it is in a suspended state where I do see that the browser is trying to get the information and the status bar shows transferring data from the website, but nothing else is changing. No error messages to report. Its just stuck. Its been like this for the last hour or so. Stuck in one place much like how I feel using this program, stuck in one place.
« Last Edit: May 27, 2007, 06:24:32 PM by Tech-Mentor »

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10624
Re: I can not get the sitemap to crawl my website.
« Reply #2 on: May 28, 2007, 08:50:25 PM »
Hello,

it looks like your server configuration doesn't allow to run the script long enough to create full sitemap. Please try to increase memory_limit and max_execution_time settings in php configuration at your host (php.ini file) or contact hosting support regarding this.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

 

SMF 2.0.12 | SMF © 2014, Simple Machines
XHTML RSS WAP2