I'm only getting one page indexed in my sitemap

[ian13]

Hi

I seem to have the same problem as a lot of people - only my home page is indexed.

I have tried this script on another site - very similar to tis site and it runs no problem - well I say no problem it runs till it runs out of memory.

What could me problem be please?


Ian

[scott10]

Same here 

[admin]

Hello,

in MOST cases the problem is with specifying Starting URL in sitemap generator options.
Please open your site and check what is the exact domain name in your links, if it is:
http://www.domain.com/page_url.html
then Starting URL must be with www prefix:

Code:

http://www.domain.com/

If your links look like http://domain.com/page_url.html
then Starting URL must be without ww prefix:

Code:

http://domain.com/

Please double check that and let me know if that works.

[ian13]

Hi Oleg,

I've tried all permutations of it now:

with www and without
with http:// and without
also a fully qualified URL direct to my index page

I can PM you the URL to the generator folder if you like so you can take a look

BTW I changed the name of the generator folder, would that make a difference?

Thanks for the reply

Ian Blackford

[ian13]

Some more info,

When I hit the crawl button, I get a flash of text then I get presented with the page where I can view the sitemap, after a few attempts I got the page to stop and I as able to copy out the text.

It's an error report and looks like this:

With .gz option ON:

Links depth: -
Current page: -
Pages added to sitemap: -
Pages scanned: - (- Kb)
Pages left: - (+ - queued for the next depth level)
Time passed: -
Time left: -
Memory usage: -
Completed
Total pages indexed: 0
Calculating changelog...
Warning: gzwrite(): supplied argument is not a valid stream resource in /www/site/generator/pages/class.xml-creator.inc.php(2) : eval()'d code on line 185

Warning: gzclose(): supplied argument is not a valid stream resource in /www/site/generator/pages/class.xml-creator.inc.php(2) : eval()'d code on line 186

Creating HTML sitemap...
Warning: ksort() expects parameter 1 to be array, null given in /www/site/generator/pages/class.html-creator.inc.php(2) : eval()'d code on line 49

Warning: Invalid argument supplied for foreach() in /www/site/generator/pages/class.html-creator.inc.php(2) : eval()'d code on line 51

With the .gz option OFF:

Warning: fwrite(): supplied argument is not a valid stream resource in /www/site/generator/pages/class.xml-creator.inc.php(2) : eval()'d code on line 185

Warning: fclose(): supplied argument is not a valid stream resource in /www/site/generator/pages/class.xml-creator.inc.php(2) : eval()'d code on line 186

Creating HTML sitemap...
Warning: ksort() expects parameter 1 to be array, null given in /www/site/generator/pages/class.html-creator.inc.php(2) : eval()'d code on line 49

Warning: Invalid argument supplied for foreach() in /www/site/generator/pages/class.html-creator.inc.php(2) : eval()'d code on line 51



I don't know if that is the full error report sorry - but I hope it helps to throw some light on the problem.

Thanks again

Ian

[nermi]

I think you should rename that folder back to /generator/ because the php script is trying to use it.


Nermi

[ian13]

Hi

Thanks for the reply, but I don't think that's the case. I altered the paths you see there so as not to expose my domain name or the new name of the generator folder.

Sorry for the confusion.

Ian

[nermi]

Will see what Oleg says, but I still have a feeling that might be a problem.
And you can always protect your files with password instead of changing the folder name.
But who knows, maybe I am wrong. Been wrong on many things lately. 

Nermi

[admin]

It should work fine with other folder names as well. Please PM me your generator URL and I will check it.

Thank you.

[admin]

Update: the URLs were blocked in robots.txt file thus none of the pages were included in sitemap. robots.txt shoould be updated in this case.

[ian13]

Oleg - thank you for your reply.

Sorry I wasn't aware that your crawler would take any notice of the robots.txt file.

If I wanted to enable in the robost.txt file, what should I add?

Thanks once again

Ian

[admin]

Sitemap Generator reads robots.txt specifically to avoid further issues - if you will create a sitemap, but will have disallow directive in robots.txt then search engines will not crawl your site. You should remove the "Disallow: /" line from robots file.