Crawling Stops after a few secs at 8K pages

[sunil1]

I have just brought Unlimited PHP Sitemap Generator. When I started Crawling it worked fine for 8K pages but suddenly it got struck there. I tired it to start again but after few secs it gets stuck again.

What can be done to fix this and for fast crawling.

[XML-Sitemaps Support]

Hello,

it looks like your server configuration doesn't allow to run the script long enough to create full sitemap. Please try to increase memory_limit and max_execution_time settings in php configuration at your host (php.ini file) or contact hosting support regarding this.
 

[sunil1]

We have increased memory_limit and max_execution_time settings but the issue is still there. The script stops after sometime.

[XML-Sitemaps Support]

Hello,
in this case I would recommend to run generator in command line if you have ssh access to your server.

[sunil1]

i have made changes in Configuration settings and crawling is going on now but it's very slow and my site have large number of pages. Can you share what are the best Configuration settings for quick crawling of all the pages.

[XML-Sitemaps Support]

Hello,

The crawling time itself depends on the website page generation time mainly, since it crawls the site similar to search engine bots.
For instance, if it it takes 1 second to retrieve every page, then 1000 pages will be crawled in about 16 minutes.

[sunil1]

I'm facing one another problem with the Crawling option. Few hours ago I interrupted the crawling and when I tried to start crawling again there is no response from the page. I'm not able to access the Crawling page when I try to open that page nothing happens.

All the other options are working fine.

[XML-Sitemaps Support]

It means that connection is still open to the website. Please try to close browser completely or open generator page in another browser (not new tab though).

[XML-Sitemaps Support]

In case if the issue is only on Crawling page though, you need to increase memory_limit setting again as it's exceeded when generator tries to load current session state.

[sunil1]

Is there any option to create partial sitemap. There are lot of pages added to sitemap and lot of pages are queued for the next depth level.

In the meantime can the sitemap be created for pages added to sitemap till now.

[XML-Sitemaps Support]

Hello,

yes, you can use  "Maximum pages" setting to limit the number of URLs to index.

[sunil1]

Can Crawling be restarted from that point by again resetting the maximum number of pages to be crawled

[XML-Sitemaps Support]

You can save a copy of "generator/data/crawl_dump.log" file which contains current crawling session, create sitemap, then restore that file from a copy, change settings if needed and run generator again from that point.

[sunil1]

I'm not able to access Crawling option since last 2-3 days. I interrupted crawling few day back but when I try to resume crawling nothing happens and that page doesn't open. It has been so long i'm trying to generate Sitemap but nothing is happening.

[XML-Sitemaps Support]

Hello,

you need to increase memory_limit setting in PHP configuration since currently it's not sufficient to read crawl_dump file.