Does Google respect User-agent rules in robots.txt?
-
We want to use an inline linking tool (LinkSmart) to cross link between a few key content types on our online news site.
LinkSmart uses a bot to establish the linking.
The issue: There are millions of pages on our site that we don't want LinkSmart to spider and process for cross linking.
LinkSmart suggested setting a noindex tag on the pages we don't want them to process, and that we target the rule to their specific user agent.
I have concerns. We don't want to inadvertently block search engine access to those millions of pages. I've seen googlebot ignore nofollow rules set at the page level. Does it ever arbitrarily obey rules that it's been directed to ignore?
Can you quantify the level of risk in setting user-agent-specific nofollow tags on pages we want search engines to crawl, but that we want LinkSmart to ignore?
-
Does Google respect User-agent rules in robots.txt?
Yes
I've seen googlebot ignore nofollow rules set at the page level.
Google honors the nofollow rules set at the page level. The issue is there may be other links on your site or elsewhere on the web that Google will find and follow those links.
Robots.txt is the absolute last means to use for blocking pages. You should not block a page with robots.txt unless you have exhausted all other options. A more appropriate method of keeping a page out of the index is the noindex tag. If you use the tag appropriately, Google will honor the tag.
-
Hi,
I would advise to block the directories which the files sit in in robots.txt, over adding no index tags to specific pages.
Yet then this would also leave these pages to not be indexed by Google, other search engines and also this Link Smart software you are referring to.
The thing is if you add a no index tag or if you add a robots .txt block to pages it will also block all search engines too.
So yes their is some risk involved, you have to do things carefully around this area.
Kind Regards,
James.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Is the meta title written only for google (try to stuff in as many keywords as possible) or is there a user experience aspect as well?
Is the meta title written only for google (try to stuff in as many keywords as possible) or is there a user experience aspect as well?
On-Page Optimization | | whiteonlySEO0 -
Website Titles in Google
I currently have a Wordpress platform website and previously I noticed that when I optimized my pages, if I indicated what I wanted my page names to be (through an application like SEO Yoast) that most times, the keyword would show up exactly how I had it typed in. Recently I have noticed that the title of my website is showing in my page titles too. So for example: Before: Shoe Stores Windsor - XYZ Company Now: XYZ Company | Shoe Stores Windsor - XYZ Company In SEO practices, I know it's most often best to have the keyword you would like as close to the front of your title tag, but now this recent search adds my website title first. Plus this also seems to be making my titles longer. I know Google ultimately has the 'final say' in a page title and I have ensured that I have the "rewrite titles/descriptions option" check in Wordpress to allow me to overwrite titles, but I am hoping someone can possibly provide me with a tip or trick to avoid this in search rankings. I think it's important to have the name of my site entered through Wordpress so that any pages that I have no optimized default to the page name and site name, but the ones I have optimized seem to be showing differently all of a sudden. Any help is greatly appreciated! Thanks!
On-Page Optimization | | MainstreamMktg0 -
Google Increases Titles and Meta Descriptions Length
So, we now know that Google are changing the Title and Meta Desc length limits to 70 and 200 respectively. Does this mean we all need to jump to immediate attention and modify all our current pages OR, do we wait?
On-Page Optimization | | dynamyt1000 -
Inches or " Feet or ' Does Google translate the symbols?
I have a client who sells things that the size is important. In their industry some people say "15 Inch Blue Widget" and others say "15" Blue Widget" using the symbol " for inches. On the page I know we could say both to cover all the bases but I want to get the title right. In their industry there is not one more preferred than the other. Does anybody know if Google translates ' to feet and " to inches. Should I work both into the title for a product or only one?
On-Page Optimization | | JoshuaLindley0 -
Is there a way to prevent Google Alerts from picking up old press releases?
I have a client that wants a lot of old press releases (pdfs) added to their news page, but they don't want these to show up in Google Alerts. Is there a way for me to prevent this?
On-Page Optimization | | IdeaGarden0 -
Google not showing the proper title
I noticed that google is not showing the proper title in the search results. If you search for PhraseExpander, the title that google reports is: PhraseExpander: Text Expander for Windows but in the title of the page I've set Text Expander for Windows - PhraseExpander Why is that? How can I make google report the proper title? Thanks. Andrea
On-Page Optimization | | nagar0 -
Google Panda This Past Weekend Impact
I understood that Google was implementing a major Panda refresh this past weekend. Did it happen? Anyone notice any impacts? What changed?
On-Page Optimization | | lbohen0 -
Google's Page Layout Algorithm Change
Hello Everyone, Google says they've implemented this change because they are answering the complaints of users who have to search for actual content after they've clicked on a result. They go on to say users want to see content right away. Now while most of this talk is about ads, I wonder if this will also apply to websites that are image and flash heavy above the fold with very little content. I am working on a few auto dealer sites where 99% of the content above the fold are flash banners and images. Below all of this noise you can find about 200 words of text talking about their dealerships. I'd love to know everyone's thoughts on this...Does the new page layout algorithm change apply to only ads or to images and flash as well? Thanks
On-Page Optimization | | wparlaman0