XML Sitemaps Generator

Author Topic: Sitemap crawling never stops/ends  (Read 163414 times)

dp

  • Registered Customer
  • Approved member
  • *
  • Posts: 2
Re: Sitemap crawling never stops/ends
« Reply #45 on: September 01, 2010, 03:07:03 PM »
I just installed the unlimited site map yesterday
Jommla is installed on site .I have installed generator in the root folder

I am on shared hosting

The script runs and stops.In the last 12 hrs has scanned 5000 pages.The site has about 24000 pages

What are the suggestions with regard to php.ini file and config

I can install it only in generator folder.Can u send me a php.ini file if needed to be installed.What else do i need to do as regards config

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10621
Re: Sitemap crawling never stops/ends
« Reply #46 on: September 01, 2010, 08:32:11 PM »
Hello,

it looks like your server configuration doesn't allow to run the script long enough to create full sitemap. Please try to increase memory_limit and max_execution_time settings in php configuration at your host (php.ini file) or contact hosting support regarding this.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

dp

  • Registered Customer
  • Approved member
  • *
  • Posts: 2
Re: Sitemap crawling never stops/ends
« Reply #47 on: September 02, 2010, 04:24:02 AM »
It is a shared hosting and i dont think i will have much luck with installing php.ini in the server.Can it be installed somewhere in the root folder?I saw a post by you suggesting that.

If so can you provide me a sample of the php.ini file format

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10621
Re: Sitemap crawling never stops/ends
« Reply #48 on: September 02, 2010, 09:01:13 PM »
Hello,

an example of php.ini would be:
Code: [Select]
memory_limit = 256M
max_execution_time = 9000
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

johny.shery

  • Registered Customer
  • Approved member
  • *
  • Posts: 4
Re: Sitemap crawling never stops/ends
« Reply #49 on: September 29, 2010, 06:51:18 PM »
pmed you.same problem.plz resolve asap.

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10621
Re: Sitemap crawling never stops/ends
« Reply #50 on: September 29, 2010, 09:03:14 PM »
Replied to your PM.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

info1746

  • Registered Customer
  • Approved member
  • *
  • Posts: 7
Re: Sitemap crawling never stops/ends
« Reply #51 on: August 08, 2011, 12:29:02 PM »
Hi, dont know if this is the right topic, but it looks like it.
I running a cron job and it seems to work fine, but when I go into the generator manualy.
It always shows an unfinished job at the crawling tab.

When I select it and click run it finishes the job in just a few seconds ( while for the complete job 1400 pages are indexed) So I guess its just at the end that something with the job goes wrong.
Please help me out here.

Kund regards
John

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10621
Re: Sitemap crawling never stops/ends
« Reply #52 on: August 09, 2011, 12:21:31 PM »
Hello,

please let me know your generator URL/login in private message to check this.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

sam16

  • Registered Customer
  • Jr. Member
  • *
  • Posts: 10
Re: Sitemap crawling never stops/ends
« Reply #53 on: October 30, 2011, 11:42:21 AM »
I'm having similar problem here, but with extreme case.
The log files for the generator reached the size of 680 GB, yes GB.
And my server CPU reached of 100%, and stopped working many times.

Its very confusing, I'm sure there is something wrong, my site have no more than 50,000 URLs, but that for sure will not need of this resources.

I will appreciate if you take a look at generator configuration.
I will send you the URL of my site in private message, hopefully you can fix it.
Thanks

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10621
Re: Sitemap crawling never stops/ends
« Reply #54 on: October 30, 2011, 05:05:17 PM »

perhaps it's a long list of historical files, there is a setting in generator config to automatically cleanup old log files to avoid this issue.
Can you check if you have a suspiciously large file in data/ folder (much larger than others) or what is an averag log file size in your case?
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

angela7

  • Registered Customer
  • Approved member
  • *
  • Posts: 3
Re: Sitemap crawling never stops/ends
« Reply #55 on: January 28, 2013, 10:27:46 AM »
Looks like I have a similar issue.

Sitemap crawl just never completes.

I do NOT have a php.ini file to edit, but did add ini_set('memory_limit','192M'); to the index.php file to solve another plugin memory issue.

Not sure how to add the memory execute line of code within this file to see if that helps?

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10621
Re: Sitemap crawling never stops/ends
« Reply #56 on: January 28, 2013, 11:12:41 AM »
Hello,

you'd need to update it in php.ini file (PHP configuration) or contact your hosting support to do that.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

angela7

  • Registered Customer
  • Approved member
  • *
  • Posts: 3
Re: Sitemap crawling never stops/ends
« Reply #57 on: January 28, 2013, 12:08:15 PM »
As I said, I do not have that file.

I CAN make edits to another file to do pretty much the same thing, but what I need to know is WHAT to add regarding the memory execute code?

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10621
Re: Sitemap crawling never stops/ends
« Reply #58 on: January 30, 2013, 12:49:29 PM »
It depends on the size of website, you can try:
memory_limit 256M
max_execution_time 9000
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

 

SMF 2.0.12 | SMF © 2014, Simple Machines
XHTML RSS WAP2