Welcome to the Q&A Forum

Browse the forum for helpful insights and fresh discussions about all things SEO.

Your browser does not seem to support JavaScript. As a result, your viewing experience will be diminished, and you have been placed in read-only mode.

Please download a browser that supports JavaScript, or enable it if it's disabled (i.e. NoScript).

How to disallow google and roger?

Technical SEO

1425

iFix last edited by

Hey Guys and girls,

i have a question, i want to disallow all robots from accessing a certain root link:

Get rid of bots

User-agent: *

Disallow: /index.php?_a=login&redir=/index.php?_a=tellafriend%26productId=*

Will this make the bots not to access any web link that has the prefix you see before the asterisk? And at least google and roger will get away by reading "user-agent: *"? I know this isn't the standard proceedure but if it works for google and seomoz bot we are good.
1 Reply Last reply
Reply Quote 0
RyanKent last edited by

In short, yes that will work.

To be clear, you can have multiple links which all lead to the same target page. The target of that link could still be indexed if there are other unblocked links which point to it.
1 Reply Last reply
Reply Quote 1

Got a burning SEO question?

Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.

Start my free trial

Browse Questions

View

From

Sorted by

With category

Explore more categories

Related Questions

Google Not Indexing Pages (Wordpress)

google search console indexed urls indexation

Hello, recently I started noticing that google is not indexing our new pages or our new blog posts. We are simply getting a "Discovered - Currently Not Indexed" message on all new pages. When I click "Request Indexing" is takes a few days, but eventually it does get indexed and is on Google. This is very strange, as our website has been around since the late 90's and the quality of the new content is neither duplicate nor "low quality". We started noticing this happening around February. We also do not have many pages - maybe 500 maximum? I have looked at all the obvious answers (allowing for indexing, etc.), but just can't seem to pinpoint a reason why. Has anyone had this happen recently? It is getting very annoying having to manually go in and request indexing for every page and makes me think there may be some underlying issues with the website that should be fixed.
Technical SEO | | Hasanovic

1
Roger returned single page

A search query in Google shows 156 indexed results. Yet Roger has completed a crawl and returned a single page. The website is table based, but there appear to be no redirects or Javascript blocking bots so I'm unsure why Roger has under delivered. The problem is not unique to Roger. I ran the site through Screaming Frog's desktop crawler and it also returned a single page. I'm wondering if there is something in the site code I don't know to look for that is stopping Roger & Screaming Frog from crawling the site. Appreciate any insights you can offer. PS. I've read https://seomoz.zendesk.com/entries/409821-Why-Isn-t-My-Site-Being-Crawled-You-re-Not-Crawling-All-My-Pages- and don't think these suggested causes apply.
Technical SEO | | NicDale

0
Google Places Page Changes

We had a client(dentist) hire another marketing firm(without our knowledge) and due to some Google page changes they made, their website lost a #1 ranking, was disassociated with the places page and was placed at result #10 below all the local results. We quickly made some changes and were able to bring them up to #2 within a few days and restore their Google page after about a week, but the tracking/forwarding phone number the marketing company was using shows up on the page despite attempts to contact Google through updating the business in places management as well as submit the phone number as incorrect while providing the correct phone number. And because the client fired that marketing company, the phone number will no longer be active in a few days. Of course this is very important for a dental office. Has anyone else had problems with the speed and updating Google Places/Plus pages for businesses? What's the most efficient way to make changes like this?
Technical SEO | | tvinson

0
Website is not indexed in Google

Hi Guys, I have a problem with a website from a customer. His website is not indexed in Google (except for the homepage). I could not find anything that can possibly be the cause. I already checked the robots.txt, sitemap, and plugins on the website. In the HTML code i also couldn't find anything which makes indexing harder than usual. This is the website i am talking about: http://www.xxxx.nl/ (Dutch) The only thing that i am guessing now is the Google sandbox, but even that is quite unlikely. I hope you guys discover something i could not find! Thanks in advance 🙂
Technical SEO | | B.Great

0
Having trouble removing homepage from google

For various reasons my client wants their homepage removed from google, no just the content of the page off but the page not to be indexed (yep strange request but we are mere service providers) today I requested in webmaster tool that default.asp was removed. Wht says done but the sites homepage is still listed. The page also has a no index tag on but 24 hours and 18k Google bot hits later it still remains. Anyone got any other suggestions to deindex just the homepage asap please
Technical SEO | | Grumpy_Carl

0
Odd Google Indexing Issue

I have encountered something odd with Google indexing. According to the Google cache my site was last updated on April 6. I had been making a series of changes on April 7th and none of them show up in the cached version of the site (naturally). Then, on the 8th, my rankings seem to have dropped about 6 places and the main SERP is showing a text that isn't even on the Web site. The cached version has the correct page title from the page that was indexed on the 6th. How do I learn where Google is picking this up from? There is a clean page title tag on my Web site. I've checked the server, etc to see what's going on. The text isn't completely unrelated, but it definitely impacted my ranking. Does Google ever have these hiccups when indexing?
Technical SEO | | VERBInteractive

0
Google support eTag?

Hello~ People! I have a questions regarding eTag. I know Google support If-Modified-HTTP-Header aka last modified header. I used eTag instead of last modified header. It seems like Google does support, yet here is my questions. code.google suggest as following. GData-Version: 2.0
ETag: "C0QBRXcycSp7ImA9WxRVFUk." but I used etag as following . ETag: "10cd712-eaae-b279a480" I didnt include "GData-Version: 2.0". is this mean Google may not support my etag?
Technical SEO | | Artience

0
Google Website Optimizer

So if you are AB testing two pages: index.html and indexB.html Shouldn't I nofollow indexB.html? It has all the same content, just a different design.
Technical SEO | | tylerfraser

0