Re: Limited pages added to sitemap
« Reply #15 on: June 01, 2020, 01:59:11 PM »
Hello,

with website of this size the best option is to create a limited sitemap - with "Maximum depth" or "Maximume URLs" option limited so that it would gather about 200-300,000 URLs, which would be main pages representing "roadmap" sitemap for search engines.

The crawling time itself depends on the website page generation time mainly, since it crawls the site similar to search engine bots.
For instance, if it it takes 1 second to retrieve every page, then 1000 pages will be crawled in about 16 minutes.

 
Re: Limited pages added to sitemap
« Reply #16 on: June 01, 2020, 03:19:35 PM »
So basically I will never get the whole site added to the sitemap?

Last run did 114497 pages in 24mins. So its pretty fast, just not finishing
Re: Limited pages added to sitemap
« Reply #17 on: June 02, 2020, 04:21:55 AM »
Hello, 

the more pages are there, the more resources and time it requires.
In this case I would recommend to run generator in command line if you have ssh access to your server.
Re: Limited pages added to sitemap
« Reply #18 on: June 02, 2020, 12:23:43 PM »
What is the command line to run under SSH on IIS

Also I noticed a lot of these errors below as well

Warning: count(): Parameter must be an array or an object that implements Countable in D:\IIS\mysite\generator\pages\page-chlog.inc.php on line 110
0)
Re: Limited pages added to sitemap
« Reply #19 on: June 02, 2020, 06:26:18 PM »
Hello,

the command line is:
php /path/to/generator-folder/runcrawl.php
Re: Limited pages added to sitemap
« Reply #20 on: June 02, 2020, 08:10:56 PM »
Right trying via cmd line

Any idea on the other error
Re: Limited pages added to sitemap
« Reply #21 on: June 03, 2020, 04:58:01 AM »
That is a warning message and can be ignored, it doesn't affect sitemaps.
Re: Limited pages added to sitemap
« Reply #22 on: June 03, 2020, 10:28:03 AM »
So ran through comand line and still same

Server is 12 core cpu , 64GB ram and ssds so no issue with server resource
Re: Limited pages added to sitemap
« Reply #23 on: June 04, 2020, 10:22:34 AM »
What is the output you get when running in command line?
Re: Limited pages added to sitemap
« Reply #24 on: June 04, 2020, 02:31:19 PM »
I shows the paes building, but stops at random places. Yesterday max I could get was 8k then 128k then 8k then 247k

SO jsut no consistancy.

I jsut think this is taking more time to get a site map built than I was expecting I might as well pay service to do it for a few $ Angling towards a refund unfortunately
Re: Limited pages added to sitemap
« Reply #25 on: June 08, 2020, 04:48:22 PM »
How do I get a refund please?