Crawling Stops after a few secs at 8K pages
« on: November 03, 2015, 10:32:49 AM »
I have just brought Unlimited PHP Sitemap Generator. When I started Crawling it worked fine for 8K pages but suddenly it got struck there. I tired it to start again but after few secs it gets stuck again.

What can be done to fix this and for fast crawling.
Re: Crawling Stops after a few secs at 8K pages
« Reply #1 on: November 04, 2015, 06:44:25 AM »
Hello,

it looks like your server configuration doesn't allow to run the script long enough to create full sitemap. Please try to increase memory_limit and max_execution_time settings in php configuration at your host (php.ini file) or contact hosting support regarding this.
 
Re: Crawling Stops after a few secs at 8K pages
« Reply #2 on: November 17, 2015, 06:10:44 AM »
We have increased memory_limit and max_execution_time settings but the issue is still there. The script stops after sometime.
Re: Crawling Stops after a few secs at 8K pages
« Reply #3 on: November 17, 2015, 06:44:41 AM »
Hello,
in this case I would recommend to run generator in command line if you have ssh access to your server.
Re: Crawling Stops after a few secs at 8K pages
« Reply #4 on: November 18, 2015, 06:57:31 AM »
i have made changes in Configuration settings and crawling is going on now but it's very slow and my site have large number of pages. Can you share what are the best Configuration settings for quick crawling of all the pages.
Re: Crawling Stops after a few secs at 8K pages
« Reply #5 on: November 18, 2015, 12:32:40 PM »
Hello,

The crawling time itself depends on the website page generation time mainly, since it crawls the site similar to search engine bots.
For instance, if it it takes 1 second to retrieve every page, then 1000 pages will be crawled in about 16 minutes.
Re: Crawling Stops after a few secs at 8K pages
« Reply #6 on: November 19, 2015, 09:25:52 AM »
I'm facing one another problem with the Crawling option. Few hours ago I interrupted the crawling and when I tried to start crawling again there is no response from the page. I'm not able to access the Crawling page when I try to open that page nothing happens.

All the other options are working fine.
Re: Crawling Stops after a few secs at 8K pages
« Reply #7 on: November 19, 2015, 04:23:28 PM »
It means that connection is still open to the website. Please try to close browser completely or open generator page in another browser (not new tab though).
Re: Crawling Stops after a few secs at 8K pages
« Reply #8 on: November 19, 2015, 04:24:29 PM »
In case if the issue is only on Crawling page though, you need to increase memory_limit setting again as it's exceeded when generator tries to load current session state.
Re: Crawling Stops after a few secs at 8K pages
« Reply #9 on: November 20, 2015, 05:28:19 AM »
Is there any option to create partial sitemap. There are lot of pages added to sitemap and lot of pages are queued for the next depth level.

In the meantime can the sitemap be created for pages added to sitemap till now.
Re: Crawling Stops after a few secs at 8K pages
« Reply #10 on: November 20, 2015, 05:46:19 AM »
Hello,

yes, you can use  "Maximum pages" setting to limit the number of URLs to index.
Re: Crawling Stops after a few secs at 8K pages
« Reply #11 on: November 20, 2015, 06:52:14 AM »
Can Crawling be restarted from that point by again resetting the maximum number of pages to be crawled
Re: Crawling Stops after a few secs at 8K pages
« Reply #12 on: November 21, 2015, 07:42:59 AM »
You can save a copy of "generator/data/crawl_dump.log" file which contains current crawling session, create sitemap, then restore that file from a copy, change settings if needed and run generator again from that point.
Re: Crawling Stops after a few secs at 8K pages
« Reply #13 on: November 24, 2015, 06:54:13 AM »
I'm not able to access Crawling option since last 2-3 days. I interrupted crawling few day back but when I try to resume crawling nothing happens and that page doesn't open. It has been so long i'm trying to generate Sitemap but nothing is happening.
Re: Crawling Stops after a few secs at 8K pages
« Reply #14 on: November 25, 2015, 06:07:20 AM »
Hello,

you need to increase memory_limit setting in PHP configuration since currently it's not sufficient to read crawl_dump file.