Welcome to the Q&A Forum

Browse the forum for helpful insights and fresh discussions about all things SEO.

Your browser does not seem to support JavaScript. As a result, your viewing experience will be diminished, and you have been placed in read-only mode.

Please download a browser that supports JavaScript, or enable it if it's disabled (i.e. NoScript).

Robots.txt question

Intermediate & Advanced SEO

635

rbai last edited by

I notice something weird in Google robots. txt tester

I have this line

Disallow: display=

in my robots.text but whatever URL I give to test it says blocked and shows this line in robots.text

for example this line is to block pages like

http://www.abc.com/lamps/floorlamps?display=table

but if I test

http://www.abc.com/lamps/floorlamps or any page

it shows as blocked due to Disallow: display=

am I doing something wrong or Google is just acting strange? I don't think pages with no display= are blocked in real.
1 Reply Last reply
Reply Quote 0
Mobilio last edited by

Yes - there is bug in your robots.txt. You should wrote some as:
Disallow: /?display=table
or:
Disallow: /?display=*
1 Reply Last reply
Reply Quote 5

Got a burning SEO question?

Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.

Start my free trial

Browse Questions

View

From

Sorted by

With category

Explore more categories

Related Questions

XML Sitemap Question!

Hi All, I know that the sitemaps.xml URL must be findable but what about the sitemaps/pageinstructions.xml URL? Can we safely noindex the sitemaps/pageinstructions.xml URL? Thanks! Yael
Intermediate & Advanced SEO | | yaelslater

0
Google Search Console Site Property Questions

I have a few questions regarding Google Search Console. Google Search Console tells you to add all versions of your website https, http, www, and non-www. 1.) Do I than add ALL the information for ALL versions? Sitemaps, preferred site, etc.? 2.) If yes, when I add sitemaps to each version, do I add the sitemap url of the site version I'm on or my preferred version? - For instance when adding a sitemap to a non-www version of the site, do I use the non-www version of the sitemap? Or since I prefer a https://www.domain.com/sitemap.xml do I use it there? 3.) When adding my preferred site (www or non-www) do I use my preferred site on all site versions? (https, http, www, and non-www) Thanks in advance. Answers vary throughout Google!
Intermediate & Advanced SEO | | Mike.Bean

0
If Robots.txt have blocked an Image (Image URL) but the other page which can be indexed has this image, how is the image treated?

Hi MOZers, This probably is a dumb question but I have a case where the robots.tags has an image url blocked but this image is used on a page (lets call it Page A) which can be indexed. If the image on Page A has an Alt tags, then how is this information digested by crawlers? A) would Google totally ignore the image and the ALT tags information? OR B) Google would consider the ALT tags information? I am asking this because all the images on the website are blocked by robots.txt at the moment but I would really like website crawlers to crawl the alt tags information. Chances are that I will ask the webmaster to allow indexing of images too but I would like to understand what's happening currently. Looking forward to all your responses 🙂 Malika
Intermediate & Advanced SEO | | Malika1

1
Robots.txt help

Hi Moz Community, Google is indexing some developer pages from a previous website where I currently work: ddcblog.dev.examplewebsite.com/categories/sub-categories Was wondering how I include these in a robots.txt file so they no longer appear on Google. Can I do it under our homepage GWT account or do I have to have a separate account set up for these URL types? As always, your expertise is greatly appreciated, -Reed
Intermediate & Advanced SEO | | IceIcebaby

0
Backlinks question: High Domain Authority, Lower Page Authority

We have a possibility of contributing guest blogs (with followed backlinks) to a site with very high domain authority (and highly trafficked), but when we've looked at the blog entires they already have, most of them have a much lower page authority. How do relevant links from a page with a lower PA but on a domain with a really high DA end up impacting our overall backlink profile? Can an expert or two give me some advice on what this may mean for us if we choose to go for it? In your opinion, does having lots of relevant links from a site with a much higher domain authority than ourselves (to give you an idea, our domain authority is in the low 60's, this site has a domain authority of almost 90) worth the time/effort/resources unto itself? Thanks!
Intermediate & Advanced SEO | | GrowOrganic

0
Diversifying anchor text question

Hi, I've seen a new article by Dr. Pete on diversifying links for 2013 (http://www.seomoz.org/blog/top-1-seo-tips-for-2013), now my question is this: Dr. Pete talks about mixing up the anchor text for links, is so we don't get caught out by Google or actually mixing it has a better impact? For example: 1. 20 anchor text links targeting just the target term. 2. 20 anchor text links targeting 4 variations of the target term. Is number 2 recommended so things look natural or does it actually have a better impact on SEO. Thanks
Intermediate & Advanced SEO | | activitysuper

0
Panda Updates - robots.txt or noindex?

Hi, I have a site that I believe has been impacted by the recent Panda updates. Assuming that Google has crawled and indexed several thousand pages that are essentially the same and the site has now passed the threshold to be picked out by the Panda update, what is the best way to proceed? Is it enough to block the pages from being crawled in the future using robots.txt, or would I need to remove the pages from the index using the meta noindex tag? Of course if I block the URLs with robots.txt then Googlebot won't be able to access the page in order to see the noindex tag. Anyone have and previous experiences of doing something similar? Thanks very much.
Intermediate & Advanced SEO | | ianmcintosh

0
301 Redirect question

Which is the best way to set up the 301 redirect on my main home page? http://horsebuggy.com to http://www.horsebuggy.com Or does it make a difference? Boodreaux
Intermediate & Advanced SEO | | Boodreaux

0