its stopped in the middle
« on: March 05, 2018, 08:44:50 PM »
Hello,
last time i created the site map was 7 months ago and i added a lot of pages to my site because of this i decided to run the generator again.
the problem is that the generator stopped in the middle of the scan (this is the 3rd time happening) as you see in the attachment appear zero (before its stopped it was in level 4 and was 18k pages scanned from 40k).

what can be the problem?
Re: its stopped in the middle
« Reply #1 on: March 06, 2018, 04:57:22 AM »
Hello,

it looks like your server configuration doesn't allow to run the script long enough to create full sitemap. Please try to increase memory_limit and max_execution_time settings in php configuration at your host (php.ini file) or contact hosting support regarding this.
 
Re: its stopped in the middle
« Reply #2 on: March 07, 2018, 11:42:05 AM »
I have the same problem. What are the recommended settings?
Re: its stopped in the middle
« Reply #3 on: March 07, 2018, 12:48:51 PM »
Hello,

it looks like your server configuration doesn't allow to run the script long enough to create full sitemap. Please try to increase memory_limit and max_execution_time settings in php configuration at your host (php.ini file) or contact hosting support regarding this.
 

i spoke with the hosting and we applied with no limit for execution time and  512M for the memory.

i will run again the script now and will update you.

Thanks
Re: its stopped in the middle
« Reply #4 on: March 07, 2018, 01:47:03 PM »
Its not a good idea to make it unlimited. I think for example if there is bad code that gets itself into an infinite loop it my crash the site altogether. So my guy tells me to make it really high but dont make it unlimited.
Re: its stopped in the middle
« Reply #5 on: March 11, 2018, 10:43:18 PM »
Its not a good idea to make it unlimited. I think for example if there is bad code that gets itself into an infinite loop it my crash the site altogether. So my guy tells me to make it really high but dont make it unlimited.
i will return it to normal settings later.

Thx
Re: its stopped in the middle
« Reply #6 on: March 11, 2018, 10:47:21 PM »
Hello,

it looks like your server configuration doesn't allow to run the script long enough to create full sitemap. Please try to increase memory_limit and max_execution_time settings in php configuration at your host (php.ini file) or contact hosting support regarding this.
 

Hi
as i said above that i changed the settings, and after almost 4 days running the generator its reached around 90k scanned and 60k added to sitemap (level 4). unfortunately now i see that scan 300 pages level 1 
its look  like its stopped and start from the beginning and this is the 3rd time is happening.

i can't generate the sitemap.... pls help

Thx
Re: its stopped in the middle
« Reply #7 on: March 12, 2018, 05:05:06 AM »
Hello,
in this case I would recommend to run generator in command line if you have ssh access to your server.
Re: its stopped in the middle
« Reply #8 on: March 12, 2018, 09:58:59 AM »
Hello,
in this case I would recommend to run generator in command line if you have ssh access to your server.

I did this also before and i had the same problem...
i have VPS... maybe to run the script from this VPS to scan my site? this will work?
Re: its stopped in the middle
« Reply #9 on: March 12, 2018, 05:08:15 PM »
Hello,

what are the last lines of output you get when running in command line?
Re: its stopped in the middle
« Reply #10 on: March 16, 2018, 09:57:19 AM »
Hello,

i still waiting to finish the scan, and i hope this time will not stop :(
pls check the attachment its normal that scanned pages different from the pages added to sitemap?

Thanks
Re: its stopped in the middle
« Reply #11 on: March 16, 2018, 04:52:38 PM »
Hello,

yes, it means that some pages were not added (in case if robots meta tag blocks it or it's redirected or canonical meta tag points to a different URL).
Re: its stopped in the middle
« Reply #12 on: March 16, 2018, 08:13:46 PM »
Hello,

what are the last lines of output you get when running in command line?

below :

[root@host ~]# /usr/bin/php /home/mydomain/public_html/generator/runcrawl.php
Content-type: text/html

Resuming the last session (last updated: 2018-03-15 20:30:44)
88268 | 0 | 11,532,179.8 | 66:54:04 | 0:00:00 | 5 | 109,643.1 Kb | 58490 | 2793 | 109643
88269 | 19233 | 11,532,307.4 | 66:54:10 | 14:34:39 | 5 | 109,944.2 Kb | 58491 | 2793 | 109944
88271 | 19231 | 11,532,563.1 | 66:54:16 | 14:34:33 | 5 | 109,955.4 Kb | 58493 | 2793 | 109955
88273 | 19229 | 11,532,818.4 | 66:54:22 | 14:34:28 | 5 | 109,964.6 Kb | 58495 | 2793 | 109964
88275 | 19227 | 11,533,074.3 | 66:54:29 | 14:34:23 | 5 | 109,975.6 Kb | 58497 | 2794 | 109975
[root@host ~]#
Re: its stopped in the middle
« Reply #13 on: March 16, 2018, 08:23:06 PM »
hi
its look like stack in specific page, every time i am clicking Resume last session... nothing happening

@Admin  i PM you with my details, can you pleas have a look

Thanks
 
Re: its stopped in the middle
« Reply #14 on: March 17, 2018, 05:27:27 AM »
Replied to your PM.