Full site not crawled - urgent help required!
« on: May 27, 2008, 04:21:55 PM »
Hi,

I have a domain at [ External links are visible to forum administrators only ] and a forum at [ External links are visible to forum administrators only ] and a news section at [ External links are visible to forum administrators only ]

The sitemap is at the root domain and crawling the root and the forum perfectly. But unable to crawl the news section.

The news (word-press) section actual url is [ External links are visible to forum administrators only ] but is redirected to [ External links are visible to forum administrators only ] - so all the posts appear as [ External links are visible to forum administrators only ] usinf .htaccess

Please advice me how to include the news section in the sitemap.

Regards.

Jack
Re: Full site not crawled - urgent help required!
« Reply #1 on: May 27, 2008, 08:03:27 PM »
Hello,

please PM me your generator URL so that I can have a detailed look.
Re: Full site not crawled - urgent help required!
« Reply #2 on: May 27, 2008, 08:33:29 PM »
Sent you pm with generator url and details.

Thanks,

Jack.
Re: Full site not crawled - urgent help required!
« Reply #3 on: May 27, 2008, 08:58:46 PM »
Hello,

your news pages return 404 http code instead of code 200 (although page content is displayed), you can check that with http headers viewer.
« Last Edit: May 27, 2008, 09:01:15 PM by admin »
Re: Full site not crawled - urgent help required!
« Reply #4 on: May 28, 2008, 09:12:33 AM »
I am using the permalink to change the news/index.php/topic.html  -> news/topic.html

Any help how this can be resolved...

Thnaks.
Re: Full site not crawled - urgent help required!
« Reply #5 on: May 29, 2008, 01:39:03 AM »
You should correct your news script to resolve the issue - currently it returns 404 http code. Even if those pages will be manually included in sitemap, search engines won't index them until you address the http code problem.
Re: Full site not crawled - urgent help required!
« Reply #6 on: May 29, 2008, 08:41:11 AM »
Solved the http header issue which was coming from a mix of wordpress and .htaccess. Now the sitemap is crawling the pages.

Cheers.
Re: Full site not crawled - urgent help required!
« Reply #8 on: May 30, 2008, 09:39:12 AM »
I believe I can use this script for my other sites also in the same server ... but when I uploaded it in another site in the same server - could not save the config details or even the login pass - if I do a direct edit of the config file it saves but when I try to start crawling from the sitemap page button - it redirects me back to the index page of config - permissions are all OK.

Please help.

Cheers.
Re: Full site not crawled - urgent help required!
« Reply #9 on: May 30, 2008, 07:23:35 PM »
Hello,

make sure that you have 0666 permissions set for config.inc.php file.
Re: Full site not crawled - urgent help required!
« Reply #10 on: May 30, 2008, 08:15:21 PM »
Thats done - but some hoe for some reason its not writing the config file and also not crawling .

Regards.
Re: Full site not crawled - urgent help required!
« Reply #11 on: May 30, 2008, 11:42:25 PM »
Please try to reinstall sitemap generator on your another site (remove generator folder and reupload files)
Re: Full site not crawled - urgent help required!
« Reply #12 on: May 31, 2008, 11:31:51 AM »
Do I need to remove the sitemap html/xml/txt/gz files generated earlier with your earlier version ? They are all writeable - if the new sitemaps are not generated then I will be left with nothing...

Regards.
Re: Full site not crawled - urgent help required!
« Reply #13 on: May 31, 2008, 02:24:12 PM »
You can rename them (or move to subfolder) and create empty sitemap files and set 0666 permissions for them.
Re: Full site not crawled - urgent help required!
« Reply #14 on: May 31, 2008, 04:40:30 PM »
Done that no use - maybe ownership problem or something else... Sending you the genrator url by pm please check - the earlier sitemaps were 78 in number !

PHP execution time is also increased at 100 from 30 earlier.

Thanks.