Moz Q&A is closed.
After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.
How to stop Search Bot from crawling through a submit button
-
On our website http://www.thefutureminders.com/, we have three form fields that have three pull downs for Month, Day, and year. This is creating duplicate pages while indexing. How do we tell the search Bot to index the page but not crawl through the submit button?
Thanks
Naren
-
Hi Dan
What is happening is this - since we have all the months [12], all the dates [31] and years[1921 through 2011] in the form fields, the robot seems to be taking these incrementally and then using the submit button. After the submit button, user is presented with a registration page. While we do want the search to index the rest of the page and the crawl through the rest of the page links we do not want it to crawl through that submit button. I hope I am making sense.
Naren
-
The advantage of blocking a page from being indexed via a meta tag is it is less likely to have unexpected consequences. I've often seen in the past cases where an incorrectly modified robots.txt file leads to a site being blocked by accident.
-
Hi
To my knowledge, you don't stop it from crawling through the button (like a nofollowed link), rather you block the robot at the page it ends up on after clicking submit.
Say the user hits submit and it takes them to mydomain.com/confirm.html On that page you'll want to add;
....if you want it to NOT index the page but follow the links on it.
or
...if you want it to NOT index and NOT follow the links on that page.
Its advised that its better to do this with the meta tag than in robots.txt.
Hopefully I've understood the question correctly!
-Dan
-
Block the pages/folders you do not wish to be indexed with robots.txt file:
User-agent: * Disallow: /folder1/ Disallow: /folder2/
OR you can add canonical tags to the other pages which are creating duplicate content.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How to get rid of bot verification errors
I have a client who sells highly technical products and has lots and lots (a couple of hundred) pdf datasheets that can be downloaded from their website. But in order to download a datasheet, a user has to register on the site. Once they are registered, they can download whatever they want (I know this isn't a good idea but this wasn't set up by us and is historical). On doing a Moz crawl of the site, it came up with a couple of hundred 401 errors. When I investigated, they are all pages where there is a button to click through to get one of these downloads. The Moz error report calls the error "Bot verification". My questions are:
Technical SEO | | mfrgolfgti
Are these really errors?
If so, what can I do to fix them?
If not, can I just tell Moz to ignore them or will this cause bigger problems?0 -
Google image search filter tabs and how to rank on them
I have noticed Google image search has included suggestion tabs (e.g,. design, nature... when searching background) on the top of the image search.
Technical SEO | | Mike555
Are there specific meta tags I can add into my images so that my images will show up on each tab?
Do those filters just show content based on image keywords or something else? IRme7gQ0 -
Indexed, but not shown in search result
Hi all We face this problem for www.residentiebosrand.be, which is well programmed, added to Google Search Console and indexed. Web pages are shown in Google for site:www.residentiebosrand.be. Website has been online for 7 weeks, but still no search results. Could you guys look at the update below? Thanks!
Technical SEO | | conversal0 -
Why does Bing bot crawl so aggressively?
We observer that the Bing bot is crawling our site very aggressively. We set Bing's crawl control so that it should not crawl us during heavy traffic hours, but that did not change a thing. Does anyone have the problem and even better a solution?
Technical SEO | | Roverandom1 -
Stop Words and keyword optimization
Ok, so I understand Google doesn't use stopwords (like "a" or "the"). Therefore if I am optimize for a keyword phrase, and say find an opportunity for say: "how to create stuff something" But it actually reads better as (although doesn't sound completely out of place as above) "how to create stuff in something" Which is better for SEO? (ignore usability \ readability in your replies please and assume it reads reasonably either way as that was just an example)
Technical SEO | | TheWebMastercom0 -
Should i Disavow links from Secret Search Engine Labs
Hi, I'm doing a link audit. My sites' keyword rankings and organic traffic have been sent to the Phantom Zone since the last Penguin Update. I've got 70+ and counting follow backlinks to my main domain and one of the my subdomains from http://www.secretsearchenginelabs.com/; should i disavow them? There's a load of links to many recognisable sites in there and my instinct's all out of whack with this decision. I think that these links were all manually added by a link building company on our behalf. It does look manipulated to me but i'd like a second opinion before I dump all of those links. Thanks Thanks
Technical SEO | | McCaldin0 -
Difference between search volume in KWT and Impressions in GWT
Hi there, Sorry I've been a bit quiet of late, we're going through a huge rebranding exercise as well as staying on top of client work. Anyway. I've got an issue with keyword research phase of a client remarketing. Trying to decide which keywords to target (aren't we all?) The client has 3 months of back data in Google Webmaster Tools, which helps us to see Impressions, CTR and actual click-throughs etc. Now, they rank #1 on Google.com for a certain keyword (logged out, of course). According to Google Keyword tool (Logged in) there are 2.7million searches per month for this keyword. With the average CTR being 18% for a #1 keyword that should be bringing in 400k visits. However, take the same keyword in Google Webmaster Tools and the impressions are actually around 1,600 per month with a CTR of 9%. Different CTR's for different sectors I can accept. What I don't get is the vast difference between the impressions in GWT compared to the alleged search volume coming from the Keyword tool. Really need to understand this so we can better select keywords and judge approximate traffic expected if ranking #1 for a keyword. Any help would be really useful. Thank you!
Technical SEO | | Nobody15609869897230 -
Does a CDN affect search rankings?
I feel kind of stupid asking this, but if i use one it would speed things up quite a bit. It is for a ecommerce website, any guidance on this would be awesome!
Technical SEO | | Hyrule0