Have a Robots.txt Issue
-
I have a robots.txt file error that is causing me loads of headaches and is making my website fall off the SE grid. on MOZ and other sites its saying that I blocked all websites from finding it. Could it be as simple as I created a new website and forgot to re-create a robots.txt file for the new site or it was trying to find the old one? I just created a new one.
Google's website still shows in the search console that there are severe health issues found in the property and that it is the robots.txt is blocking important pages. Does this take time to refresh? Is there something I'm missing that someone here in the MOZ community could help me with?
-
Hi primemediaconsultants!
Did this get cleared up?
-
You not always have to do this, if you would go to domain.com/robots.txt then it should be removed maybe already. If that's the case you should be starting to see an increase in the number of pages crawled in Google Search Console.
-
This seems very helpful as I did remove it, and fetch as google, but i'm a complete novice. How do you clear server cache?
-
What does your robots.txt file contain? (or share the link)
Try removing it, clearing server cache and fetching as google again.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Any issues with having a seperate shop section?
Ive had a bit of a dilemma whether to go for a full ecommerce site or having a seperate shop section. My main goal is to push our installation services so Ive decided to go with the latter option. The main categories will be focused solely on Installation services and then Ill have a seperate category which will take the customer to mydomain.com/shop where well have our products for them to buy and fit themselves. The only issue I see is that Im going to have two pages competing against each other. Theyll both have different content but one will be focusing on us installing a particular product and the other will focus on the customer buying it to fit themselves. Will it make things more difficult to rank or wont it make a difference?
Intermediate & Advanced SEO | | paulfoz16090 -
Localized Domain Issue - Can I use Search Console to solve this?
Struggling through trying to resolve a complicated search issue - would appreciate any community input or suggestions. The Background Info We have several brand sites and each one has both a .ca and .com domain. For some reason, our website platform was created in a way that hundreds of pages on the .com domain have an equivalent page on the .ca domain, which are all 301'ed to the appropriate .com pages. Example below for clarity: www.domain.ca/gadget/brand - 301 Redirected to: www.domain.com/gadget/brand www.domain.ca/gadget/en/brandcanada = Proper .ca Canadian URL (where en is the language - fr exists as well) The Problem Because these .com pages exist under the .ca domain as well, they have started to outrank the correct .ca pages on Google. This has led to Canadian customers finding incorrect information, pricing, and reviews for these products - causing all sorts of customer service issues and therefore affecting our sales. I am being told that to properly fix the issue, and remove the incorrect URLs under the .ca domain would be prohibitively expensive in terms of resources, so I'm left trying to fix this via means available to me (i.e. anything but a change to how the platform is currently setup). The Attempted Fix I've submitted proper sitemaps for the .ca brand sites, and we have also created a robots.txt file to be accessed only when the site is crawled through the .ca domain. In that robots.txt, we have Disallowed crawling of any /gadget/brand/ URLs for the .ca domain. This was done a week ago and I am still seeing the .com URL show up in search results. The Question Should I be submitting any www.brand.ca/gadget/brand/ URLs to be temporarily removed from Google? Because of the 301 redirect in place from www.brand.ca/gadget/brand to www.brand.com/gadget/brand, I am hesitant to do so, as I do not want the .com URL removed. Will Google simply remove the .ca URL and not follow the 301 redirect to remove that URL as well? Any additional insight or feedback would be awesome as well.
Intermediate & Advanced SEO | | Trevor-O0 -
Not sure how we're blocking homepage in robots.txt; meta description not shown
Hi folks! We had a question come in from a client who needs assistance with their robots.txt file. Metadata for their homepage and select other pages isn't appearing in SERPs. Instead they get the usual message "A description for this result is not available because of this site's robots.txt – learn more". At first glance, we're not seeing the homepage or these other pages as being blocked by their robots.txt file: http://www.t2tea.com/robots.txt. Does anyone see what we can't? Any thoughts are massively appreciated! P.S. They used wildcards to ensure the rules were applied for all locale subdirectories, e.g. /en/au/, /en/us/, etc.
Intermediate & Advanced SEO | | SearchDeploy0 -
Rankings disappeared on main 2 keywords - are links the issue?
Hi, I asked a question around 6 months ago about our rankings steadily declining since April of 2013. I did originally reply to that topic a few days ago, but as it's so old I don't think it's been noticed. I'm posting again here, if that's an issue I'm happy to delete. Here it is for reference: http://moz.com/community/q/site-rankings-steadily-decreasing-do-i-need-to-remove-links Since the original post, I have done nothing linkbuilding-wise except posting blog posts and sharing them on Facebook, G+ and Twitter. There are some links in there which don't look great (ie spammy seo directories, which I'm sending removal requests to) although quite a lot of others are relevant. Here's my link profile: <a rel="nofollow" target="_blank">http://www.opensiteexplorer.org/links?site=www.thomassmithfasteners.com</a> I've tried to make the site more accessible - we now have a simple, responsive design and I've tried to make the content clear and concise. In short, written for humans rather than search engines. As of the end of November, 'nuts and bolts' has now disappeared completely, and 'bolts and nuts' is page 8. There are many pages much higher which are not as relevant and have no links. We still rank highly for more specialised terms - ie 'bsw bolts' and 'imperial bolts' are still page 1, but not as high as before. We get an 'A' grade on the on-page grader for 'nuts and bolts, and most above us get F. I was cautious about removing links as our profile doesn't seem too bad but it does seem as if it's that. There are a fair few questionable directories in there, no doubt about that, but our overall practice in recent years has been natural building and link earning. So - I've created a spreadsheet and identified the bad links - ie directories with any SEO connotations. I am about to submit removal requests, I thought two polite requests a couple of weeks apart prior to disavowing with Google. But am I safe to disavow straight away? I say this as I don't think I'll get too many responses from those directories. I am also gradually beefing up the content on the shop pages in case of any 'thin content' issues after advice on the previous post. I noticed 100s of broken links in webmaster tools last week due to 2 broken links on our blog that repeated on every page and have fixed those. I have also been fixing errors W3C compliance-wise. Am I right to do all this? Can anyone offer any suggestions? I'm still not 100% sure if this is Panda, Penguin or something else. My guess is Penguin, but the decline started in March 2013, which correlates with Panda. Best Regards and thanks for any help, Stephen
Intermediate & Advanced SEO | | stephenshone0 -
This is a clear-cut canonical issue, right?
Hello, A client is having one of their daily blogs published on a industry news site along with on their own site. This is a clear-cut case of having a canonical tag implemented on the client's site on each blog page, right? Thanks
Intermediate & Advanced SEO | | Martin_S0 -
Entire site code copied - potential SEO issues?
Hi folks, We have noticed that our site has been directly duplicated by another site. They have copied the entire code, including the JS, CSS and most of the HTML and have simply switched their own text and images onto the template. (We discovered it because they even copied over our analytics tracking and were appearing in our reports - duh!) Does anyone know if there are potential SEO issues in copying the code like that, or do duplicate content issues only apply to indexable HTML content? Thanks! Matthew (I didn't want to out them by sharing their URL because it could have been an external contractor that built the site and they probably had no idea.)
Intermediate & Advanced SEO | | MattBarker0 -
Ecombuffet.com are offering a Rescue Review focused on Panda - Penguin and identifying issues. Has anyone used this service or aware of the organisation in general?
http://www.ecombuffet.com/rescue-review.htm . I have 2 sites that have definitely been hit by penguin and getting worse so am thinking of paying for this service as nothing I do seems to stop the slide (more like a plummet). Any comments welcome.
Intermediate & Advanced SEO | | Shaann1 -
Canonicalization issue? - URLs with and without trailing slashes showing up as unique
For some reason, our urls are set to change from “www.apprenda.com/ANYTHING" to “apprenda.com/ANYTHING” These register as different pages though? We have rankings in SEOMoz Pro for terms where our homepage shows up 6th on google, but SEOMoz says it's not on the first page because it's checking against apprenda.com and not www.apprenda.com Also, it seems like for some reason pages with trailing slashes also register differently than those without. Should we be doing something for that? Something to make sure all pages get rewritten to having the trailing slash or not? For instance, this url: http://apprenda.com/saasgrid/features/multi-tenancy/ and this url” http://apprenda.com/saasgrid/features/multi-tenancy are really the same page. Yet in our analytics, they register as different pages with their own stats, etc. What should we do in our particular case, and how can we get this fixed? I really appreciate the help, and thanks in advance! Jesse
Intermediate & Advanced SEO | | ApprendaPlatform0