How to disallow google and roger?
-
Hey Guys and girls,
i have a question, i want to disallow all robots from accessing a certain root link:
Get rid of bots
User-agent: *
Disallow: /index.php?_a=login&redir=/index.php?_a=tellafriend%26productId=*
Will this make the bots not to access any web link that has the prefix you see before the asterisk? And at least google and roger will get away by reading "user-agent: *"? I know this isn't the standard proceedure but if it works for google and seomoz bot we are good.
-
In short, yes that will work.
To be clear, you can have multiple links which all lead to the same target page. The target of that link could still be indexed if there are other unblocked links which point to it.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Google Analytics and Adwords Linkup
I'm in the middle of a log jam with linking google analytics to google adwords. I have a client whose account lists properties for five distinct websites. Company A www.website1.com/
Technical SEO | | OOMDODigital
www.website2.com/
www.website3.com/ There are individual adwords accounts for each property that I'm trying to link up. Here's the issue I"m having. I can link up Website1.com Analytics to Website1.com Adwords. Then I'll link up Website2.com Analyitics to Website2.com Adwords. This will cause Website1.com to be linked up to Website2.com's Adwords. How do I fix this?0 -
Website is not indexed in Google
Hi Guys, I have a problem with a website from a customer. His website is not indexed in Google (except for the homepage). I could not find anything that can possibly be the cause. I already checked the robots.txt, sitemap, and plugins on the website. In the HTML code i also couldn't find anything which makes indexing harder than usual. This is the website i am talking about: http://www.xxxx.nl/ (Dutch) The only thing that i am guessing now is the Google sandbox, but even that is quite unlikely. I hope you guys discover something i could not find! Thanks in advance 🙂
Technical SEO | | B.Great0 -
Site being indexed by Google before it has launched
We are currently coming towards the end of a site migration, and are at the final stage of testing redirects etc. However, to our horror we've just discovered Google has started indexing the new site. Any ideas on how this could have happened? I have most recently asked for robots.txt to exclude anything with a certain parameter in URL. Is there a chance this, wrongly implemented, could have caused this?
Technical SEO | | Sayers0 -
Remove Site from Google
How can I get my website out of google? I want all pages completely gone. Thanks!
Technical SEO | | tylerfraser0 -
Schema.org support by google
Hi, Some time ago I have implemented schema.org product schema on all of my product pages. The rich snippet tool provided by Google shows that the relevant info is extracted (they say that preview is not supported so no preview there) My issue is that I can't see any rich snippet displayed for any on my searches, which leads me to think that there is something wrong with my implementation. sample page: www.funstuff.co.il/tabid/62/ProdID/933/products.aspx [editor's note: adult oriented content, NSFW] Any one has an Idea? Thanks, Asaf
Technical SEO | | AsafY0 -
Google Off/On Tags
I came across this article about telling google not to crawl a portion of a webpage, but I never hear anyone in the SEO community talk about them. http://perishablepress.com/press/2009/08/23/tell-google-to-not-index-certain-parts-of-your-page/ Does anyone use these and find them to be effective? If not, how do you suggest noindexing/canonicalizing a portion of a page to avoid duplicate content that shows up on multiple pages?
Technical SEO | | Hakkasan1 -
Does Google care how you write internal links?
I am changing ecommerce platforms. For my internal linking on the old site there was a lot of old links written like this: http://www.domain.com/page-name But now i am writing links mostly like this: /page-name Will that make a difference to search engines? Is one easier than the other for them to interpret?
Technical SEO | | Hyrule0 -
Some site pages are removed from Google Index
Hello, Some pages of my clients website are removed from Google Index. We were in top 10 position for some keywords but now I cannot find those pages neither in top 1000. Any idea what to do in order to get these pages back? thank you
Technical SEO | | besartbajrami0