XML Sitemaps Generator

Author Topic: Crawler is not working at all! - serious problem  (Read 11496 times)

mhz

  • Registered Customer
  • Approved member
  • *
  • Posts: 3
Crawler is not working at all! - serious problem
« on: March 01, 2011, 02:36:31 PM »
I installed it successfully and tried crawling my big web site. after hours crawling faced problem with memory.

Now I can access crawling interface at all. Half of the page loads and the crawling button don't show at all.

Please help me urgently

Screen shot was attached .

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10623
Re: Crawler is not working at all! - serious problem
« Reply #1 on: March 01, 2011, 02:48:36 PM »
Hello,

that happens when memory limit is exceeded when generator is trying to read the crawl dump file, you will need to increase memory_limit setting in PHP configuration to resolve that.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

mhz

  • Registered Customer
  • Approved member
  • *
  • Posts: 3
Re: Crawler is not working at all! - serious problem
« Reply #2 on: March 01, 2011, 03:15:45 PM »
1-How can I do that?
2-Is it possible to limit total memory used by crawler?

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10623
Re: Crawler is not working at all! - serious problem
« Reply #3 on: March 01, 2011, 08:09:46 PM »
You should modify php.ini file on your server or contact your hosting support for that.
You can set "Maximum URLs" setting in generator cofniguration to keep smaller sitemap and correspondingly less memory usage.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

mhz

  • Registered Customer
  • Approved member
  • *
  • Posts: 3
Re: Crawler is not working at all! - serious problem
« Reply #4 on: March 04, 2011, 08:36:51 AM »
I raised the memory to 512 MB but now the program stop responding at all.

There are many free tools which capable of producing 50,000 pages site map, I paid you to have a better tool. but it seems it can't work even as good as free software.

I can't generate a site map with more than 42,000 pages. it just crashes. I use a dedicated server with 4GB Ram, so there shouldn't be any problem with CPU or Ram.

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10623
Re: Crawler is not working at all! - serious problem
« Reply #5 on: March 05, 2011, 07:44:46 AM »
Hello,

please let me know your generator URL/login in private message to check this.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

phil.rose

  • Registered Customer
  • Approved member
  • *
  • Posts: 2
Re: Crawler is not working at all! - serious problem
« Reply #6 on: May 02, 2011, 06:18:59 AM »
I have exactly the same problem.

I have set the memory limit on the server to unlimited and I still have the problem.

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10623
Re: Crawler is not working at all! - serious problem
« Reply #7 on: May 03, 2011, 06:59:59 PM »
Hello,

please let me know your generator URL/login in private message to check this.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

phil.rose

  • Registered Customer
  • Approved member
  • *
  • Posts: 2
Re: Crawler is not working at all! - serious problem
« Reply #8 on: May 03, 2011, 07:30:40 PM »
Hi Oleg,  Message sent

 

SMF 2.0.12 | SMF © 2014, Simple Machines
XHTML RSS WAP2