Form > Crawler doesn't follow select dropdown links
« on: September 10, 2017, 06:35:27 AM »
Hi,

A question here:
I've got a SELECT dropdown & form on my homepage visitors need to use in order to select a town. The problem is the XML-sitemap doesn't crawl the different select options (all unique url's) but it should do that. I'm only getting back some general pages (links in footer / header menu).

Site: https://www.tabr.nl/
OR in English: https://translate.google.com/translate?sl=auto&tl=en&js=y&prev=_t&hl=en&ie=UTF-8&u=https%3A%2F%2Fwww.tabr.nl%2F&edit-text=&act=url

Sitemap: https://www.tabr.nl/sitemap.xml

How can I make sure the sitemap is crawling through all the subpages...
The selection a user makes is stored in a cookie so he/she only needs to select a city once. Maybe that's a problem (but I don't see ANY cities in my xml sitemap)...

Anyone to the rescue?
Re: Form > Crawler doesn't follow select dropdown links
« Reply #1 on: September 11, 2017, 05:44:24 AM »
Hello,

all pages must be accessible via <a href=""> links and should be working with javascript disabled in the browser to allow crawler bots to find them.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.
Re: Form > Crawler doesn't follow select dropdown links
« Reply #2 on: September 11, 2017, 07:59:29 AM »
That did the trick indeed. Thanks for the quick reply.
Should all these links be visible or can I "display:none" them so they are only visible for bots?
Or will SEO punish me for this kind of trics.
Re: Form > Crawler doesn't follow select dropdown links
« Reply #3 on: September 11, 2017, 08:36:35 PM »
Those can be invisible for the sitemap generator bot, although it's not recommended for search engines bots.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.