ViVa Computers

*
  • *
  • 13
  • Allways look on the bright site of life!
Crawling stops all the time
« on: October 27, 2020, 04:04:05 PM »
Hello, I have the unlimited crawler script running on my site. In the beginning it is working fine but after a while it gets stuck when it reaches a area on my site. This area is were the shop is.
I tried to ban it from scanning but that doesn't work but actually I would like it to be scanned.

What can I do?

The site is [ External links are visible to forum administrators only ] and if you want I can give in private access to the generator.
If you need help with your computer please check: .

Also for Laptop - Tablet - Smartphone repair!
Re: Crawling stops all the time
« Reply #1 on: October 27, 2020, 09:01:10 PM »
Hello,

it looks like your server configuration doesn't allow to run the script long enough to create full sitemap. Please try to increase memory_limit and max_execution_time settings in php configuration at your host (php.ini file) or contact hosting support regarding this.
 

ViVa Computers

*
  • *
  • 13
  • Allways look on the bright site of life!
Re: Crawling stops all the time
« Reply #2 on: October 28, 2020, 03:46:37 AM »
Ok I will do that but what is strange is also that I bought the web crawler service but it only comes to 6 pages while the unlimited script comes til far more?
If you need help with your computer please check: .

Also for Laptop - Tablet - Smartphone repair!
Re: Crawling stops all the time
« Reply #3 on: October 28, 2020, 05:37:19 AM »
Hello,

could you please PM me an example URL that is not included in sitemap and how it can be reached starting from homepage?
 

ViVa Computers

*
  • *
  • 13
  • Allways look on the bright site of life!
Re: Crawling stops all the time
« Reply #4 on: October 30, 2020, 06:53:53 AM »
Hello Oleg,

I have send you a PM. How can I prevent the script from crawling certain areas of the site? If I can do that then for sure it will not be a problem.
If you need help with your computer please check: .

Also for Laptop - Tablet - Smartphone repair!
Re: Crawling stops all the time
« Reply #5 on: October 30, 2020, 09:44:11 AM »
You can use "Exclude URLs" setting to block parts of website from crawling.
Re: Crawling stops all the time
« Reply #6 on: November 23, 2020, 04:41:20 PM »
Hi.

i already increase memory and limit time and i have still the same issue - stuck

any help?

brS
Re: Crawling stops all the time
« Reply #7 on: November 23, 2020, 06:27:20 PM »
Hello, 

in this case I would recommend to run generator in command line if you have ssh access to your server.
Re: Crawling stops all the time
« Reply #8 on: November 24, 2020, 10:20:00 AM »
yes this is work for me.
BR


Hello, 

in this case I would recommend to run generator in command line if you have ssh access to your server.