Limited pages added to sitemap

[XML-Sitemaps Support]

Hello,

with website of this size the best option is to create a limited sitemap - with "Maximum depth" or "Maximume URLs" option limited so that it would gather about 200-300,000 URLs, which would be main pages representing "roadmap" sitemap for search engines.

The crawling time itself depends on the website page generation time mainly, since it crawls the site similar to search engine bots.
For instance, if it it takes 1 second to retrieve every page, then 1000 pages will be crawled in about 16 minutes.

 

[accounts178]

So basically I will never get the whole site added to the sitemap?

Last run did 114497 pages in 24mins. So its pretty fast, just not finishing

[XML-Sitemaps Support]

Hello, 

the more pages are there, the more resources and time it requires.
In this case I would recommend to run generator in command line if you have ssh access to your server.

[accounts178]

What is the command line to run under SSH on IIS

Also I noticed a lot of these errors below as well

Warning: count(): Parameter must be an array or an object that implements Countable in D:\IIS\mysite\generator\pages\page-chlog.inc.php on line 110
0)

[XML-Sitemaps Support]

Hello,

the command line is:
php /path/to/generator-folder/runcrawl.php

[accounts178]

Right trying via cmd line

Any idea on the other error

[XML-Sitemaps Support]

That is a warning message and can be ignored, it doesn't affect sitemaps.

[accounts178]

So ran through comand line and still same

Server is 12 core cpu , 64GB ram and ssds so no issue with server resource

[XML-Sitemaps Support]

What is the output you get when running in command line?

[accounts178]

I shows the paes building, but stops at random places. Yesterday max I could get was 8k then 128k then 8k then 247k

SO jsut no consistancy.

I jsut think this is taking more time to get a site map built than I was expecting I might as well pay service to do it for a few $ Angling towards a refund unfortunately

[accounts178]

How do I get a refund please?

[XML-Sitemaps Support]

You can send us an email with details on: https://www.xml-sitemaps.com/contact.html