Does Google crawl the pages which are generated via the site's search box queries?
-
For example, if I search for an 'x' item in a site's search box and if the site displays a list of results based on the query, would that page be crawled? I am asking this question because this would be a URL that is non existent on the site and hence am confused as to whether Google bots would be able to find it.
-
Google crawls the pages. Google sometimes even tries the search box by typing in a random word to see what happens.
Have a look at this url: https://www.google.com/search?q=site%3Agoogle.com%20inurl%3A%22search%3Fq%22 You'll see that search queries have been indexed. It happens if people link to it. It doesn't matter that the url is "non existent". It actually is, because it doesn't display a 404 error or any other error. It gives a 200 OK code, so search engines see it as a normal page. Google will probably not index a page it "makes" by filling in a random search term, but will index such a page when it is linked to.
-
Google could crawl the dynamic URLs created by your searchbox - but it usually doesn't unless there is a link to such a dynamic url somewhere. Internal searches don't create much problems anymore, but if you want to be sure, you could always block your dynamic search results pages via robots.txt or Google Webmaster Tools (>Site configuration >URL parameters).
So if the URL generated by internal searches is http://www.site.com/search/?searchword=search+query+here, you could add this to robots.txt:
User-agent: *
Disallow: /search/
-
No I am not talking about the google search box incorporated in sites but the site's own search box. Answering your 2nd question, I meant that the URL won't be found via site navigation as it is a dynamic URL that is generated. Awaiting your response on the same.
-
"if I search for an 'x' item in a site's search box and if the site displays a list of results based on the query, would that page be crawled?"
Is it google search-box for sites your talking about?" I am asking this question because this would be a URL that is non existent on the site"
**If it does'nt exist you wouldent find it? or do you mean that the page is not to find in the menu-navigation? **
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Is there a difference between 'Mø' and 'Mo'?
The brand name is Mø but users are searching online for Mo. Should I changed all instances of Mø to be Mo on my clients website?
Intermediate & Advanced SEO | | ben_mozbot010 -
Magento: Should we disable old URL's or delete the page altogether
Our developer tells us that we have a lot of 404 pages that are being included in our sitemap and the reason for this is because we have put 301 redirects on the old pages to new pages. We're using Magento and our current process is to simply disable, which then makes it a a 404. We then redirect this page using a 301 redirect to a new relevant page. The reason for redirecting these pages is because the old pages are still being indexed in Google. I understand 404 pages will eventually drop out of Google's index, but was wondering if we were somehow preventing them dropping out of the index by redirecting the URL's, causing the 404 pages to be added to the sitemap. My questions are: 1. Could we simply delete the entire unwanted page, so that it returns a 404 and drops out of Google's index altogether? 2. Because the 404 pages are in the sitemap, does this mean they will continue to be indexed by Google?
Intermediate & Advanced SEO | | andyheath0 -
Google Sitelinks Search Box
For some reason, a search for our company name (“hometalk”) does not produce the search box in the results (even though we do have sitelinks). We are adding schema markup as outlined here, but we're not sure about: Will adding the code make the search bar appear (or at least increase the chances), or is it only going to change the functionality of the search box (to on-site search) for results that are already showing a search bar?
Intermediate & Advanced SEO | | YairSpolter0 -
Shoemaker with ugly shoes : Agency site performing badly, what's our best bet?
Hi everyone,
Intermediate & Advanced SEO | | AxialDev
We're a web agency and our site www.axialdev.com is not performing well. We have very little traffic from relevant keywords. Local competitors with worse On-page Grader scores and very few backlinks outrank us. For example, we're 17th for the keyword "agence web sherbrooke" in Google.ca in French. Background info: In the past, we included 3 keywords-rich in the footer of every site we made (hundreds of sites by now). We're working to remove those links on poor sites and to use a single nofollow link on our best sites. Since this is on-going and we know we won't be able to remove everything, our link profile sucks (OSE). We have a lot of sites on our C-Block, some of poor quality. We've never received a manual penalty. Still, we've disavowed links as a precaution after running Link D-Tox. We receive a lot of trafic via our blog where we used to post technical articles about Drupal, Node js, plugins, etc. These visits don't drive business. Only a third of our organic visits come from Canada. What are our options? Change domain and delete the current one? Disallow the blog except for a few good articles, hoping it helps Google understand what we really do. Keep donating to Adwords? Any help greatly appreciated!
Thanks!2 -
We sold our site's domain and have a new one. Where do we go from here?
We recently sold our established domain -- for a compelling price -- and now have the task of transitioning to our new domain. What steps would you recommend to lesson the anticipated decline from search engines in this scenario?
Intermediate & Advanced SEO | | accessintel0 -
What's the best way to redirect categories & paginated pages on a blog?
I'm currently re-doing my blog and have a few categories that I'm getting rid of for housecleaning purposes and crawl efficiency. Each of these categories has many pages (some have hundreds). The new blog will also not have new relevant categories to redirect them to (1 or 2 may work). So what is the best place to properly redirect these pages to? And how do I handle the paginated URLs? The only logical place I can think of would be to redirect them to the homepage of the blog, but since there are so many pages, I don't know if that's the best idea. Does anybody have any thoughts?
Intermediate & Advanced SEO | | kking41200 -
There's NO reason these sites should be beating mine...Or is there?
Hi Over the past 10 months, my internal page rankings (previously excellent) have plummeted. I'm now trying to recover them. I haven't received an unnatural links warning in Google Webmaster Tools. Also, I used to have hundreds of internal links to each of these 21 pages using the same exact-match anchor text eg, Tuscany real estate, Umbria real estate, etc. I changed this about 6 months ago. So why am I still ranking poorly for these (only moderately competitive keywords) behind sites with poorer metrics? 1) Keyword: lake como real estate My page here – **http://tinyurl.com/d34k8m ** -- used to rank No1 or No2 neck-and-neck with this page www.immobiliarevacanzelago.com/. He's still No1 but I’m down to about No13. Yet when I look in Open Site Explorer virtually all my metrics beat his.
Intermediate & Advanced SEO | | Jeepster0 -
Hidden text that's not show in default view, does it hurt my on page optimization?
Hello, I am developing our new site that will create tabs such as "design", "tech specs", "customer ratings" etc. just like http://www.dell.com/us/p/inspiron-15z-5523/pd. My question is, if most of my content is on the 2nd and 3rd tabs, would my content effect my overall SEO if it's not on it's default view? Because, if users don't physically click on the 2nd and 3rd tab, does it mean that since there is no impression of that content, that content will be allocated to half of its SEO juice? Let me know, I would love to know! Shawn
Intermediate & Advanced SEO | | Shawn1241