Do robot.txts permanently affect websites even after they have been removed?
-
A client has a Wordpress blog to sit alongside their company website. They kept it hidden whilst they were developing what it looked like, keeping it un-searchable by Search Engines. It was still live, but Wordpress put a robots.txt in place. When they were ready they removed the robots.txt by clicking the "allow Search Engines to crawl this site" button.
It took a month and a half for their blog to show in Search Engines once the robot.txt was removed.
Google is now recognising the site (as a "site:" test has shown) however, it doesn't rank well for anything. This is despite the fact they are targeting keywords with very little organic competition.
My question is - could the fact that they developed the site behind a robot.txt (rather than offline) mean the site is permanently affected by the robot.txt in the eyes of the Search Engines, even after that robot.txt has been removed?
Thanks in advance for any light you can shed on the situation.
-
No problem! Good Luck!
-
That is a very fair point. It is a completely new site and I hadn't even thought about things like the domain age. It does show up under a "site:http://www.____.com" search, I was just wondering if this is one of those things Google keeps a memory of, if that makes sense.
Thanks for your response Mike.
-
That is a very good suggestion. I'll try it (a useful URL also so thanks for sharing).
Thanks for the response Matthew.
-
I think the much more likely culprit is that it is a new site. What do you get when you enter "site:http://www._____.com" in google? If the pages are indexed, one can't blame for the robots file for lack of rank.
Good luck!
Mike
-
Have you submitted the updated robots.txt to google? This is separate from updating the sitemap. Here is a google page to help you do this.
https://support.google.com/webmasters/answer/6078399?hl=en
Best!
Matthew
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Inspection of the website is required.
Hello, Recently We have a taken a project " http://customerconnect-services.com/ " and are responsible for the Digital Marketing for the website. FYI, we have been following the best On-Page SEO practices ever since we took the project: Performing Keyword Research, Finalizing the Keywords,Using those in the Page-Title, Meta-Descriptions, Heading Tags and of course in the Content as per MOZ suggestions & SEO standards. But, we are unable to rank in first page; which is a serious matter of concern for us. We have also checked whether the Domain / URL has been blacklisted, but it's not (Not even by Google). We are therefore unable to figure out what is going wrong even after following so many best practices to get the keywords a good ranking (1st & 2nd page of SERPs). Therefore I would like to request you to provide your expert opinions in this regard by checking what is it that we are not getting right. The website url is http://customerconnect-services.com/. As this is a high-priority issue for us & the client is a prestigious one of course, please help. Looking forward to hearing from you at the earliest. Thanks & Regards,
Technical SEO | | Harini.M
Harini0 -
Is Removing Breadcrumbs Detrimental for SEO?
We have full navigational breadcrumbs on our site for the menu and the brand menu. i.e. Home > Clothing > Jackets Brand > Brand Name > Brand Jackets There's been talk of removing this and having it like Chico's does, where on item pages they just have a link at the top to previous category (i.e. you're on a shirt product page and at the top it says "Back to Tops" instead of listing Home > Clothing > Tops) Is doing something like this detrimental to SEO? From what I've read Breadcrumbs are for user experience but I just want to be sure.
Technical SEO | | AliMac261 -
SEO impact classifieds website
Hi, I'm part of an organization running a classifieds platform in Spain. (Mercadonline.es) We are hit by Google penalties since a few weeks, possibly caused by numerous errors we are experiencing. Most frequent errors are 404's and duplicate content (titles tags etc) since the nature of our website is dynamic. Many ads change daily, are added or removed, causing Googlebots (and others) to flag us and not being able to see our more unique content. Until what part of our platform should we be indexed? Since we have +34,000 pages indexed (mostly due to internal filter pages) I would need a systematic solution for us to display relevant and unique content, with enough usage of keywords that can bring us back up - we are actually ranked <50 on google for most of our main keywords. It is costing us precious time and money since we can only aquire our visitors (adwords etc) and not being to attract any organically. I can go in more detail with someone who can give me a bit more direction. Your answer is much appreciated! Ivor
Technical SEO | | ivordg0 -
Sitemap international websites
Hey Mozzers,Here is the case that I would appreciate your reply for: I will build a sitemap for .com domain which has multiple domains for other countries (like Italy, Germany etc.). The question is can I put the hreflang annotations in sitemap1 only and have a sitemap 2 with all URLs for EN/default version of the website .COM. Then put 2 sitemaps in a sitemap index. The issue is that there are pages that go away quickly (like in 1-2 days), they are localised, but I prefer not to give annotations for them, I want to keep clear lang annotations in sitemap 1. In this way, I will replace only sitemap 2 and keep sitemap 1 intact. Would it work? Or I better put everything in one sitemap?The second question is whether you recommend to do the same exercise for all subdomains and other domains? I have read much on the topic, but not sure whether it worth the effort.The third question is if I have www.example.it and it.example.com, should I include both in my sitemap with hreflang annotations (the sitemap on www.example.com) and put there it for subdomain and it-it for the .it domain (to specify lang and lang + country).Thanks a lot for your time and have a great day,Ani
Technical SEO | | SBTech0 -
Redirects for new website
Hi Moz community,
Technical SEO | | JSimmons17
I'm a fairly new SEO Specialist with a brand new website. We initially had a very basic holding website until the fully functional website was completed. I have to do some redirects as we have both .html and .php files & we don't want to lose SEO value for specific pages (like the index, news, etc). I also want to redirect from a www url to a non-www url. I am trying to accomplish redirects with the following code: RewriteEngine on
RewriteCond %{HTTP_HOST} ^www.mywebsite.com [NC]
RewriteRule (.*) http://mywebsite.com/$1 [R=301,L] RedirectMatch 301 /index.html (.*).(php|html) http://mywebsite.com/index.php RedirectMatch 301 /cupcakes-slideshow/glutenfree-slideshow.html (.*).(php|html) http://mywebsite.com/gluten-and-glutenfree.php RedirectMatch 301 /press.html (.*).(php|html) http://mywebsite.com/news-and-reviews.php Please let me know if I am on the right track. Thanks so much in advance!0 -
Website Grader Report - Permanent Redirect Not Found
Have you ever checked HubSpot's website grader at www.websitegrader.com? I usually notice that the tool gives an error namely "Permanent Redirect Not Found" with below explanation: "Search engines may think www.example.com and example.com are two different sites.You should set up a permanent redirect (technically called a "301 redirect") between these sites. Once you do that, you will get full search engine credit for your work on these sites. :(Website Grader) Can we trust this tool?
Technical SEO | | merkal20050 -
Robots.txt question
I want to block spiders from specific specific part of website (say abc folder). In robots.txt, i have to write - User-agent: * Disallow: /abc/ Shall i have to insert the last slash. or will this do User-agent: * Disallow: /abc
Technical SEO | | seoug_20050