Not sure how we're blocking homepage in robots.txt; meta description not shown
-
Hi folks!
We had a question come in from a client who needs assistance with their robots.txt file.
Metadata for their homepage and select other pages isn't appearing in SERPs. Instead they get the usual message "A description for this result is not available because of this site's robots.txt – learn more".
At first glance, we're not seeing the homepage or these other pages as being blocked by their robots.txt file: http://www.t2tea.com/robots.txt.
Does anyone see what we can't? Any thoughts are massively appreciated!
P.S. They used wildcards to ensure the rules were applied for all locale subdirectories, e.g. /en/au/, /en/us/, etc.
-
I can see the meta descriptions in SERPs. do you have any sample pages where it does not show up?
-
According to screamingfrog the current line:
Line:40 http://www.t2tea.com/on/demandware.store/
Is the line on robots.txt is causing you an issue.
-
Hi,
It looks like they are 302 redirecting the homepage to internal language/region specific storefronts but are doing that based on an internal url structure that includes /on/demandware.store/ which is indeed being blocked in the robots.txt. It looks like those urls are then being 301 redirected to the user friendly url you see in the browser so there is a potentially odd redirect chain going on there. The original blocked urls are probably the immediate issue (although the 302 redirects and region/language redirect logic might be putting more complication on top of that).
-
The best way to test this is to head into Search Console and use the Robots.txt tester. If a URL is being blocked, or suspect it is, just add that URL to be tested and it will show you.
https://support.google.com/webmasters/answer/6062598?hl=en
-Andy
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Is it good practice to use "SAVE $1000's" in SEO titles and Meta Descriptions?
Our company sells a product system that will permanently waterproof almost anything. We market it as a DIY system. I am working on SEO titles and descriptions. This topic came up for discussion, if using "SAVE $1000's.." would help or hurt. We are trying to create an effective call to action, but we are wondering if search engines see it as click bait. Can you
Intermediate & Advanced SEO | | tyler.louth0 -
I'm invisible!
Dear all, Earl 2015 i launched www.taobao.nl. Its a platform that shows the products of chinese marketplaces in our local language (auto translated) and provides the option to pay in our local currency & methods. (i blocked all the auto translated product pages via http://www.taobao.nl/robots.txt) All content (category pages, general pages etc) are unique and written by us. In the past I've set up several websites and shops and i know it always takes time and effort to get a site to good positions, but i never experienced that my efforts did not pay of. Until now 😞 The url is not blocked by Google and i see a (very) limited amount of traffic in WMT & Analytics, but the site is >page 30 on its most important keyword 'taobao'. This, while the competition on this keyword is low in my country (google.nl). My main competitor (taobao.com.nl) seems to do worse on most SEO elements i was able to check, but they rank very well on 'taobao' and all related keywords. What could be the reason of this very poor ranking and what should i really adjust / improve?
Intermediate & Advanced SEO | | benhond
(besides backlinks: i'm working hard on that) Your input is deeply appreciated!! Sander1 -
Google not displaying meta description
Hi, one of my clients is receiving the following error in SERP - "A description of the page is not available because of this site's robots.txt". The site is built on WordPress and I realized that by default, the settings were checked to blocks bots from crawling the site. So, I turned it off, fixed robots.txt and submitted the sitemap again. Since, then it's been almost 10 days, the problem still exists. Can anyone tell me what should be done to fix it or if there's a way to get Google to recrawl the pages again.
Intermediate & Advanced SEO | | mayanksaxena0 -
Robot.txt error
I currently have this under my robot txt file: User-agent: *
Intermediate & Advanced SEO | | Rubix
Disallow: /authenticated/
Disallow: /css/
Disallow: /images/
Disallow: /js/
Disallow: /PayPal/
Disallow: /Reporting/
Disallow: /RegistrationComplete.aspx WebMatrix 2.0 On webmaster > Health Check > Blocked URL I copy and paste above code then click on Test, everything looks ok but then logout and log back in then I see below code under Blocked URL: User-agent: * Disallow: / WebMatrix 2.0 Currently, Google doesn't index my domain and i don't understand why this happening. Any ideas? Thanks Seda0 -
Google changing the meta title for the homepage causing branding issues
A client of mine." Ross X Bute" current meta title is "Luxury designer clothing | Womens designer clothing" for the homepage. If i search for luxury designer clothing it will show the full meta title for the homepage. however if i search for the brand name.... "Ross & Bute" will show instead of the meta title. Whats the problem? Well my client a few month ago has decided to re brand the business to have a "X" to show instead of the "And". The rest of the site is branded with an "X" rather than "And" The URL www.rossandbute.com, so you can understand where google is getting this assumption from. Is there anyway to change this so it reads the the meta title in the SERPs? Thanks
Intermediate & Advanced SEO | | Martin_Harris0 -
Faceted navigation, Affiliate links, Meta descriptions - Oh My!
Hi, I have recently come across an issue with the faceted navigation / dynamic URLs for one of my client sites: From a top level category you can filter by product material, size, type and colour. The URLs which are generated go a little something like this: www.domainname.co.uk/category.aspx?finish=leather&colour=--+no+filter+-- When selected, a 'facet' 302 redirects from the main category URL (no canonical tags in place yet - working on it). The 'facets' are indexed (although when clicked on from SERPs actually go to a slightly different URL than by navigating there from the site) but they don't display the Meta description in SERPs (instead displaying a list of items from a drop down menu held within a table - probably as it's the first bit of copy the search engines see on the page). How can I get the Meta description to display in SERPs? Also, I tried to add a link to a 'facet' from my blog (just for testing purposes) and I got redirected to the page via their affiliate program. Ideally I want to 'link build' to these pages both internally and via the clients blog but it seems as though there'll be no value in it. Has anyone come across this before and if so, what can I do about it? FYI they are using IIS 6 with asp.net Thanks in advance!
Intermediate & Advanced SEO | | WalkerM0 -
Avoiding 301 on purpose; Landing homepage linking to another domain with "Click here to go" and 5 sec meta refresh
Hello, Some users when they search for our site by using "ourbrand" keyword that ignore the first result (we will call it here ourbrand.de -not real name-) and they look for ourbrand.com . Even though we have that domain name also registered (indeed it also has a high ranking power) we are doing a 301 from the dot com to the dot.de . What we want to do is to index the homepage of the dot com, that is http://www.ourband.com as a secondary result while doing a 301 to any other internal URL of the dot com to the dot .de. Yes, we will loose link juice for the main domain but at least we will not loose visits from the brand traffic (which is our main traffic). So the question is, would Google index ourbrand.com if we show just a landing page that just show our logo, a "Click here to go to ourbrand.de" with a link to http://www.ourbrand.de and a meta refresh of 6 seconds to that URL? Additionally a cookie would be sent to the first time visitors, so in the next time they would be automatically redirected. PS: The 6 seconds is to avoid search engine consider it a "301" like it do with short meta refresh (not sure what time is the minimum to avoid be considered a 301). Any other suggestions on how to deal with this problem are welcomed
Intermediate & Advanced SEO | | Zillo0 -
Meta HTTP-EQUIV
Hi people! When you try to access http://www.ufam.edu.br you are redirected to http://portal.ufam.edu.br. Here's the code: <title>UFAM - Universidade Federal do Amazonas</title> What are the implications for SEO of this ? Won't the juice be passed? Isn't it better for passing juice using redirect 301?
Intermediate & Advanced SEO | | marcelocustodio0