Robots.txt and canonical tag
-
In the SEOmoz post - http://www.seomoz.org/blog/robot-access-indexation-restriction-techniques-avoiding-conflicts, it's being said -
If you have a robots.txt disallow in place for a page, the canonical tag will never be seen.
Does it so happen that if a page is disallowed by robots.txt, spiders DO NOT read the html code ?
-
Thanks Ryan for explaining things very clearly.
-
What we know is there have been many cases where a page that is blocked in robots.txt has appeared in search results. The explanation provided is that robots.txt blocks crawlers during normal site visits, but not necessarily on visits where they are following links from other sites.
-
If spiders follow links to an article on my site, will they read the contents then ? If the canonical tag is on article page itself, will canonical tag will be seen ?
-
Daylan offered a great answer but I would like to add one exception. When crawlers from the major SEs visit your site they will honor your robots.txt file but sometimes they will follow links from other sites to an article on your site, and during that particular visit they will not see the robots.txt file and index your page.
This is one of the reasons why your robots.txt file should be used as minimally as possible, and when it is used you should have a backup process in place such as the canonical or noindex tag on a page.
-
Thanks Daylan for your quick response. I just wanted a second opinion that canonical tag will never be seen if a page is disallowed.
-
Thats correct in most cases:
It works likes this: a robot wants to vists a Web site URL, say http://www.example.com/welcome.html. Before it does so, it firsts checks for http://www.example.com/robots.txt, and finds:
User-agent: *
Disallow: /The "User-agent: *" means this section applies to all robots. The "Disallow: /" tells the robot that it should not visit any pages on the site.
Robots can ignore your /robots.txt. Especially malware robots that scan the web for security vulnerabilities, and email address harvesters used by spammers will pay no attention.
More information available here about:
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
If Google's index contains multiple URLs for my homepage, does that mean the canonical tag is not working?
I have a site which is using canonical tags on all pages, however not all duplicate versions of the homepage are 301'd due to a limitation in the hosting platform. So some site visitors get www.example.com/default.aspx while others just get www.example.com. I can see the correct canonical tag on the source code of both versions of this homepage, but when I search Google for the specific URL "www.example.com/default.aspx" I see that they've indexed that specific URL as well as the "clean" one. Is this a concern... shouldn't Google only show me the clean URL?
Technical SEO | | JMagary0 -
Robots.txt Download vs Cache
We made an update to the Robots.txt file this morning after the initial download of the robots.txt file. I then submitted the page through Fetch as Google bot to get the changes in asap. The cache time stamp on the page now shows Sep 27, 2013 15:35:28 GMT. I believe that would put the cache time stamp at about 6 hours ago. However the Blocked URLs tab in Google WMT shows the robots.txt last downloaded at 14 hours ago - and therefore it's showing the old file. This leads me to believe for the Robots.txt the cache date and the download time are independent. Is there anyway to get Google to recognize the new file other than waiting this out??
Technical SEO | | Rich_A0 -
Is there any value in having a blank robots.txt file?
I've read an audit where the writer recommended creating and uploading a blank robots.txt file, there was no current file in place. Is there any merit in having a blank robots.txt file? What is the minimum you would include in a basic robots.txt file?
Technical SEO | | NicDale0 -
Isnt it better to have headlines in H1 and H2 tags instead of p tags?
I am working with a simple site http://http://lightsigns.com/Uniko_Manufacturing_Limited.html They seek more SEO traffic. However, the two big headlines that read "Wholesale Supply to the Sign and Display Industries" which is on line 241 and 242 of the source code, its in a p tag, i.e. <p <span class="webkit-html-tag">style</p <span>="padding-top: 0pt; " class="paragraph_style_1">Wholesale Supply to the and <p <span class="webkit-html-tag">style</p <span>="padding-bottom: 0pt; " class="paragraph_style_1">Sign and Display Industries Likewise, the product titles are in p tags, also. For example, on the Slide-in Light Box product page, http://lightsigns.com/Slide_In_light_box.html , I have done keyword research and no one is using the words slide in light box.Plus, it is also a p tag, ie. line 43 reads style="padding-bottom: 0pt; padding-top: 0pt; " class="paragraph_style">Slide-in Light Box If I suggest that they make an H2 tag with SEO-optimized keywords such as Display Light Box - Slide-In LIght Box, would this indeed help SEO? In summary, is it correct to say that H1 and H2 tags are stronger signals to the search bots of what the page is about?
Technical SEO | | BridgetGibbons1 -
Google insists robots.txt is blocking... but it isn't.
I recently launched a new website. During development, I'd enabled the option in WordPress to prevent search engines from indexing the site. When the site went public (over 24 hours ago), I cleared that option. At that point, I added a specific robots.txt file that only disallowed a couple directories of files. You can view the robots.txt at http://photogeardeals.com/robots.txt Google (via Webmaster tools) is insisting that my robots.txt file contains a "Disallow: /" on line 2 and that it's preventing Google from indexing the site and preventing me from submitting a sitemap. These errors are showing both in the sitemap section of Webmaster tools as well as the Blocked URLs section. Bing's webmaster tools are able to read the site and sitemap just fine. Any idea why Google insists I'm disallowing everything even after telling it to re-fetch?
Technical SEO | | ahockley0 -
Tag archives in wordpress
I have duplicate content issue on my site, because i allow to index tags in my wordpress. And the content overlaps on them. What could be a solution to this? How do i fight it, if still want my tag pages to be indexed in Google, but i don't want to to influence my traffic negatively? Currently i have 596 tags! 🙂 Site:
Technical SEO | | pycckuu
richclubgirl.com My idea was to put canonical tag for the post i want to rank from the most popular tag pages (with biggest page authority). Would love to hear from You!1 -
How similar do pages need to be in order to utilize the canonical tag
Here is my specific situation. My company released new versions of a few documents in the fall. I was hoping that over time the old version would decline and the new version would rise but after 6 months the old version continues to rank #1 and the new version #3. The old version needs to stay on our site but users should really be getting to the most recent version. I think utilizing the canonical tag would solve the issue but i am concerned because the content on the actual pages is not duplicate but it is updated. Below are the two URLs to see the differences in the content. http://www.sei.cmu.edu/library/abstracts/reports/06tr008.cfm http://www.sei.cmu.edu/library/abstracts/reports/10tr033.cfm Is this an appropriate situation to use the canonical tag? If not, is there a better solution.
Technical SEO | | SEI0 -
Should I set up a disallow in the robots.txt for catalog search results?
When the crawl diagnostics came back for my site its showing around 3,000 pages of duplicate content. Almost all of them are of the catalog search results page. I also did a site search on Google and they have most of the results pages in their index too. I think I should just disallow the bots in the /catalogsearch/ sub folder, but I'm not sure if this will have any negative effect?
Technical SEO | | JordanJudson0