<?php $initurl = 'https://www.yourdomain.com'; $ch = curl_init(); curl_setopt($ch, CURLOPT_URL, $initurl); curl_setopt($ch, CURLOPT_HEADER, 1); curl_setopt($ch, CURLOPT_VERBOSE, 1); curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, FALSE); curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1); if($errno = curl_errno($ch)) { $error_message = curl_error ($ch); echo "cURL error ({$errno}):\n {$error_message}"; } $info = curl_getinfo($ch); print_r($info); $fdata = curl_exec($ch); print_r($fdata); curl_close($ch); ?>
Logged Oleg Ignatiukhttps://www.xml-sitemaps.comSend me a Private MessageSEM and SEO Reports, more than 45M domains: The world's leading Competitive Intelligence Tool for digital marketing.
sharingsunshine 20 Re: Moved Hosts Now Crawls 1 Page Only « Reply #9 on: July 30, 2018, 07:06:42 PM » this is the page I get when I run that script - [ External links are visible to forum administrators only ]I ran a netstat and got this. From what I know this seems to show the ports are open:[root@ip-172-31-8-214 conf.d]# netstat -a | grep -i LISTENtcp 0 0 0.0.0.0:ssh 0.0.0.0:* LISTENtcp 0 0 localhost:smtp 0.0.0.0:* LISTENtcp 0 0 0.0.0.0:sunrpc 0.0.0.0:* LISTENtcp6 0 0 [::]:ssh [::]:* LISTENtcp6 0 0 [::]:https [::]:* LISTENtcp6 0 0 [::]:mysql [::]:* LISTENtcp6 0 0 [::]:sunrpc [::]:* LISTENtcp6 0 0 [::]:http [::]:* LISTEN Logged XML-Sitemaps Support 11792 Re: Moved Hosts Now Crawls 1 Page Only « Reply #10 on: July 31, 2018, 04:28:13 AM » As it's seen on a screenshot, the test script receives 403 forbidden response. It means that port is open, but your website blocks access from our server IP address. Logged Oleg Ignatiukhttps://www.xml-sitemaps.comSend me a Private MessageSEM and SEO Reports, more than 45M domains: The world's leading Competitive Intelligence Tool for digital marketing. XML-Sitemaps Support 11792 Re: Moved Hosts Now Crawls 1 Page Only « Reply #11 on: July 31, 2018, 04:35:23 AM » It is also possible that your website blocks access from generator bot user-agent, please try to add this setting in generator/data/generator.conf file:Code: [Select]<option name="xs_crawl_ident">Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:61.0) Gecko/20100101 Firefox/61.0</option> Logged Oleg Ignatiukhttps://www.xml-sitemaps.comSend me a Private MessageSEM and SEO Reports, more than 45M domains: The world's leading Competitive Intelligence Tool for digital marketing. sharingsunshine 20 Re: Moved Hosts Now Crawls 1 Page Only « Reply #12 on: August 01, 2018, 12:25:59 PM » You were correct, I have a plugin to stop bots. I never dreamed your bot's name would be in the bot table but it was. Once I whitelisted it, everything worked perfectly.Thanks for sticking with me on this. Logged
<option name="xs_crawl_ident">Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:61.0) Gecko/20100101 Firefox/61.0</option>
Pages: 1 « previous next »
Guest