SPR

*
  • *
  • 5
Sitemap takes long time to generate
« on: May 18, 2017, 02:12:37 PM »
Hello there,

Sitemap generator takes long time to generate and very slow and stops crawling frequently and no refresh or continuing uninterrupted.

Earlier we use to generate thousands of pages, but with new version, we face problem.

Request assistance.

Thanks

SPR

*
  • *
  • 5
Re: Sitemap takes long time to generate
« Reply #1 on: May 18, 2017, 02:16:59 PM »
It has been weeks since I am trying to generate the sitemap. Whenever it stops I am continuing without any end.
Re: Sitemap takes long time to generate
« Reply #2 on: May 19, 2017, 02:58:23 AM »
I am getting 130000 pages in 23 minutes. Used to having 130000 pages in days.
Hint was in Narrow Indexed Pages Set. Try Exclusion preset. I am now using this software without a hassle.

SPR

*
  • *
  • 5
Re: Sitemap takes long time to generate
« Reply #3 on: May 20, 2017, 11:41:43 AM »
I have been using this tool for long time, only with updated version i am facing problem.

In 8 minutes it crawled 1463 pages, and there is no responsethereafer.  Even the auto-restart trying to resume but receiving following response

:resuming generator (10 seconds with no response)

Same message is being printed for more than an hour a d there is action in the progress section.

SPR

*
  • *
  • 5
Re: Sitemap takes long time to generate
« Reply #4 on: May 20, 2017, 11:56:30 AM »
Even if its crawled through SSH terminal, it take 25 minutes to crawl 3108 pages
Re: Sitemap takes long time to generate
« Reply #5 on: May 21, 2017, 05:40:06 AM »
Hello,

The crawling time itself depends on the website page generation time mainly, since it crawls the site similar to search engine bots.
For instance, if it it takes 1 second to retrieve every page, then 1000 pages will be crawled in about 16 minutes.

SPR

*
  • *
  • 5
Re: Sitemap takes long time to generate
« Reply #6 on: May 21, 2017, 01:22:58 PM »
Hello there,

My site has more than 900,000 pages. Considering your calculation for 16 minutes 1000 pages, your software is not good for big sites?  It would takes week to build site map.

I believe there is some other problem thats delaying the crawling.  Because the crawling was good with previous version.  Is the new version failed to satisfy customers. I see so many people have vomplained about slow crawling
Re: Sitemap takes long time to generate
« Reply #7 on: May 21, 2017, 06:19:17 PM »
Hello,

it depends on how fast the pages are served by a website. If it takes less time to load them, crawling is faster correspondingly.