Robot.txt File Not Appearing, but seems to be working?
-
Hi Mozzers,
I am conducting a site audit for a client, and I am confused with what they are doing with their robot.txt file. It shows in GWT that there is a file and it is blocking about 12K URLs (image attached). It also shows in GWT that the file was downloaded 10 hours ago successfully. However, when I go to the robot.txt file link, the page is blank.
Would they be doing something advanced to be blocking URLs to hide it it from users? It appears to correctly be blocking log-ins, but I would like to know for sure that it is working correctly. Any advice on this would be most appreciated. Thanks!
Jared
-
There is an old webmaster world thread that explains how to hide the robots.txt file from browsers.... not sure why one would do this however....
http://www.webmasterworld.com/forum93/74.htm
Perhaps they are doing something like this?
-
I verified that I was checking /robots.txt. I had trouble verifying if it was under the non-www because everything redirects to the www. I also checked to see if it was being blocked, and it is not.
I went to Archive.org (Wayback Machine), and I can see the robot.txt file in previous versions of the site. I cannot, however, view it online, even though Google says they are downloading it successfully, and the robots.txt file is successfully blocking URLs from the search index.
-
Be sure you are visiting /robots.txt In all of your copy above, you are referencing robot.txt
Also, check to see if it possibly is only showing up on the www. version or the site or the non-www version of the site.
To be sure if it's working, you can test URLs of your website within Google Webmaster Tools. Go to Crawl->Blocked URLs and scroll down to the bottom.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
What happens to crawled URLs subsequently blocked by robots.txt?
We have a very large store with 278,146 individual product pages. Since these are all various sizes and packaging quantities of less than 200 product categories my feeling is that Google would be better off making sure our category pages are indexed. I would like to block all product pages via robots.txt until we are sure all category pages are indexed, then unblock them. Our product pages rarely change, no ratings or product reviews so there is little reason for a search engine to revisit a product page. The sales team is afraid blocking a previously indexed product page will result in in it being removed from the Google index and would prefer to submit the categories by hand, 10 per day via requested crawling. Which is the better practice?
Intermediate & Advanced SEO | | AspenFasteners1 -
Working on new link structure
Hello Mozzzzzzzzzzz I'm currently working on the new link structure for our website. We currently organize our content in sub folder =Main category
Intermediate & Advanced SEO | | floaumet
= = Sub category
= = =Specialty
= = = Product main name
= = = ==Product specific name
= = = == =Manufacturer Each of them has some potential strong KW and I will be happy to use it on the URL. URL are more than 50 kw long when I use all This are very niche item for which people may refer to them through different names (potential folders) My current concerns will be Should we make long url respecting the structure (Main category)/(Sub category)/(Specialty)/(Product main name)/(Product specific name)/(maufacturer) Should I combine some like (Main category)/(Sub category)/(Specialty)/(Product main name)/(Product specific name)-(maufacturer) Should I keep them simple /(product_main name) Should I keep the main folders just to display the articles belonging to this category (Main category)/(Sub category)/(Specialty)/(Product main name) and then keep the product under a sub folder only? Any other idea?0 -
How to work with schema.org together with Wordpress
Hi schema.org seems to be quite simple when working with a plain HTML website. You just look up the code you need on schame.org and implement it in the HTML file as required. But when using Wordpress things become more complicated. I have to use plugins for schemata and then I can only use the schemata that exist in those plugins which are very limited. How do you deal with this issue? Cheers Marc
Intermediate & Advanced SEO | | RWW1 -
Images not appearing in Google Images SERPS
Hi there We pushed a new version of our website live more than 6 months ago. So far, none of the images that are in the product gallery on this page http://www.ingleandrhode.co.uk/bespoke-rings/inspiration/ are appearing in the Google Images SERPS (I tested this by searching Google Images for "site:www.ingleandrhode.co.uk"). I understand that the gallery uses Javascript, so Googlebot doesn't see the image files in the HTML, but in Webmaster Tools, if I "fetch as Google" with rendering, this suggests that Googlebot does see the gallery images. My website developer tried adding an image sitemap about two weeks ago, which is being indexed, but so far this hasn't made any difference. Any suggestions on what needs to be done for these gallery images to start appearing in Google Images SERPS? Many thanks!
Intermediate & Advanced SEO | | TimIngle0 -
Why isn't the Google change of address tool working for me?
Last night I switched my site from http to https. Both sites are verified in Webmaster Tools but when I try to use the change of address it says- Your account doesn't contain any sites we can use for a change of address. Add and verify the new site, then try again. How do I fix this?
Intermediate & Advanced SEO | | EcommerceSite0 -
Link exchanges of specific blogs work if relevant?
Hello, I've always wondered if I have a tech blog and wrote about "why Droid phones are better than Iphones", i would need more links pointed to my specific blog. Doing so, i find another blog that's reputable with high domain authority that talks about the SAME blog/subject. Is it wise and good for SEO if i contact the blogger and have each other reference each other's blog with the anchor text link as the brand name in our respective blogs? It's a typical link exchange, but this is more niche. Would this help my efforts? And would Google accept our good faith linking to a great article vice versa. Thanks, Shawn
Intermediate & Advanced SEO | | Shawn1240 -
Does this work as a tactic for including keyword in URL structure
Howdy, I'm planning out a website and need to plan out the URL structure for best SEO value. Generally I would do something like this:
Intermediate & Advanced SEO | | IrvCo_Interactive
site.com/widgetssite.com/widgets/large
site.com/widgets/large/blue
etc. I think this is a pretty straight forward SEO tactic. The issue I have with it is in terms of natural language the "thing" you are searching for in this case is a widget, so typically you would type/search [adjective] [noun], or in this case "large blue widgets." So one proposal I have is to instead append the "widget" to the end of the URL:
site.com/large-widgets
site.com/large/blue-widgets
site.com/large/blue/square-widgets
etc. Obviously this breaks the whole silo concept since the square-widgets page is inside the /blue directory but the blue widgets page isn't at /blue it is /blue-widgets. My solution is to setup 301 redirects from /blue to /blue-widgets (even thought there are no site links pointing to that page). Does this seem like a good idea? Or does this break the whole folder silo concept? What I like about it is that it feels more user friendly in terms of natural language and for certain high value keywords we can get certain pairings of words into the URL more like how a person would type them in.0 -
XML Sitemap instruction in robots.txt = Worth doing?
Hi fellow SEO's, Just a quick one, I was reading a few guides on Bing Webmaster tools and found that you can use the robots.txt file to point crawlers/bots to your XML sitemap (they don't look for it by default). I was just wondering if it would be worth creating a robots.txt file purely for the purpose of pointing bots to the XML sitemap? I've submitted it manually to Google and Bing webmaster tools but I was thinking more for the other bots (I.e. Mozbot, the SEOmoz bot?). Any thoughts would be appreciated! 🙂 Regards, Ash
Intermediate & Advanced SEO | | AshSEO20110