XML Sitemaps Generator

Author Topic: your SE Bot Simulator says my robots.txt file disallows the sitemap.xml.gz  (Read 22192 times)

paypal353

  • Registered Customer
  • Jr. Member
  • *
  • Posts: 23
...as header. My (dynamic) robots.txt file does NOT bar the new
sitemap file. I have also appended the Sitemap: fullpathURL. Is there
something I must now check or do you have a problem over there?
----best wishes, Robert

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10624
Please PM me your sitemap URL and robots.txt link so that I can reproduce the issue.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

paypal353

  • Registered Customer
  • Jr. Member
  • *
  • Posts: 23
Oleg----
Done.
----best wishes, Robert

paypal353

  • Registered Customer
  • Jr. Member
  • *
  • Posts: 23
Oleg----
Tried using a non .gz flavour and also put the sitemap
into the root... as your 'test' [sic] tool seems to be
a facility more akin to the 'tail wagging the dog' in
terms of robustness. However even THIS trick did
not fix your tool which STILL says that my now
root-based sitemap.xml file is disallowed by the
robots.txt file (no it isn't). It's late g'night.
---best wishes, Robert

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10624
Hello,

I get no errors for your sitemap: [removed]
« Last Edit: April 15, 2008, 03:21:33 PM by admin »
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

paypal353

  • Registered Customer
  • Jr. Member
  • *
  • Posts: 23
Oleg----
That is because I decided to help matters by placing the file
in the root despite the new area being not quite ready for
full production. [removed/with thanks]
----best wishes, Robert

[postedit by Robert: unnecessary text removed]
« Last Edit: April 15, 2008, 03:24:27 PM by paypal353 »

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10624
Yes, that is one of the requirements of xml sitemap protocol: http://sitemaps.org
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

paypal353

  • Registered Customer
  • Jr. Member
  • *
  • Posts: 23
Oleg----
You've already seen that I've moved my sitemap.xml file to the
root of the site (which has new and old model content areas as
described in another thread). However the initial problem is
STILL resident. Your tool inexplicably says that my robots.txt
file appearently disallows even the root-resident sitemap.xml
file. You know the URL of my site so please advise why your
tool seems to have got it so wrong. (My site URL should not
appear in live forum threads, I much prefer it that way).
I run a dynamic robots.txt file functionality. It's actually a
PHP file;~) Maybe it is this that is messing up your tool? It
shouldn't affect it but I mention this just in case.
----best wishes, Robert

paypal353

  • Registered Customer
  • Jr. Member
  • *
  • Posts: 23
Oleg----
Fixed it;~) My enforced requirement to move the sitemaps.xml
file to the root had repercussions. Already existing within the
disallowed section of the dynamic robots.txt file is a standing
no-no for /site as referring to a directory /site/foo.bah that
was previously in use. However the logic extends that into
disallowing the sitemap.xml file too... hence your tool's report.
Success - my thanks;~)
----best wishes, Robert

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10624
I'm glad it is sorted now :)
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

paypal353

  • Registered Customer
  • Jr. Member
  • *
  • Posts: 23
Oleg----
Agreed and thanks again for the help.
Now I have a mailbomber at the ramparts...
Still, looking at the bright side of it, I get a massive
list of broken or vulnerable sites to add to the filters;~/
May I ask how long it will be before the 'next update'
ie the one with the user-option to remove branding
remnant in the generated output? Days/weeks/months?
----best wishes, Robert

 

SMF 2.0.12 | SMF © 2014, Simple Machines
XHTML RSS WAP2