Sitemap Generator Forum

Just as the title suggests, I am having trouble indexing my website. Accessing the page [ External links are visible to forum administrators only ] isn't an issue, simply getting the crawler to see my pages appears to be an issue.

I am sure this is a simple issue but need direction.

I should mention that I have checked with the link https://www.xml-sitemaps.com/se-bot-simulator.html and the website, both http and https can be crawled.

Hello,

please let me know your generator URL/login in private message to check this.

Update: all pages are blocked with robots.txt :
Disallow: /

Thank you very much, we had recently migrated servers and it had not occurred to me to check our robots.txt file to make sure we updated it once the dust had settled.

My next issue has to do with an empty sitemap.xml file.

I have started the crawling for a small set of links so that testing is easier.

Currently it is retrieving 3 URLS: [ External links are visible to forum administrators only ]

However, none of those are saved in the sitemap.xml file in the public_html/ directory: [ External links are visible to forum administrators only ]


One thing I have noticed is that I change the permissions for sitemap.xml to 666, only to have it change to 644 once the crawler has been used. Is there any reason for this, and could that be where some of the issues are stemming from?

I just reinstalled everything and now it works.

-shrug-

Thank you for the help on the previous issue.