XML Sitemaps Generator

Author Topic: Crawling refuses to complete! Only 300 pages in 2 days  (Read 325 times)

support344

  • Registered Customer
  • Approved member
  • *
  • Posts: 1
Crawling refuses to complete! Only 300 pages in 2 days
« on: July 27, 2017, 07:03:15 PM »
I apologize the frustrated tone of my message in advance. I have a Wordpress website that I simply want to generate an XML sitemap for. The site has about 300 main pages that need to be crawled, but there are three unnecessary directories that have over 700 pages that DO NOT need to be crawled. If I use the Free Online Sitemap Generator, it crawls EVERYTHING, maxing out at 500 while skipping the other remaining pages that needed to be crawled. There’s no option to exclude directories with the free version. 

So I reluctantly (I say reluctantly because I had a funny feeling it wasn’t going to work very well) purchased the Unlimited Sitemap Generator. I installed it with no problem, followed the instructions, and added the directories that DO NOT need to be crawled under the “Excluded URLs” section. I used the following format for the excluded directories:

excludedirectory1/
excludedirectory2/

I believe this is correct, right?

Anyway, I started the Crawling process TWO DAYS ago, and it’s STILL running! Why is this taking so darn long for only 300 pages?!?! The Free Generator processed 500 pages in less than 5 minutes, while the paid version has only made it to 193 in two whole days. After several hours, I would switch tabs and go back to resume the process, but it only inches up a few more pages each time. It seems to stall after every few pages. 

Why on God’s green earth is the process taking so long? I wish I could use the free version because it’s light years faster, however I can’t exclude directories however  >:(
« Last Edit: July 27, 2017, 07:05:13 PM by support344 »

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10849
Re: Crawling refuses to complete! Only 300 pages in 2 days
« Reply #1 on: July 28, 2017, 05:00:27 AM »
Hello,

please let me know your generator URL/login in private message to check this.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

 

SMF 2.0.14 | SMF © 2017, Simple Machines
XHTML RSS WAP2