Do robot.txts permanently affect websites even after they have been removed?
-
A client has a Wordpress blog to sit alongside their company website. They kept it hidden whilst they were developing what it looked like, keeping it un-searchable by Search Engines. It was still live, but Wordpress put a robots.txt in place. When they were ready they removed the robots.txt by clicking the "allow Search Engines to crawl this site" button.
It took a month and a half for their blog to show in Search Engines once the robot.txt was removed.
Google is now recognising the site (as a "site:" test has shown) however, it doesn't rank well for anything. This is despite the fact they are targeting keywords with very little organic competition.
My question is - could the fact that they developed the site behind a robot.txt (rather than offline) mean the site is permanently affected by the robot.txt in the eyes of the Search Engines, even after that robot.txt has been removed?
Thanks in advance for any light you can shed on the situation.
-
No problem! Good Luck!
-
That is a very fair point. It is a completely new site and I hadn't even thought about things like the domain age. It does show up under a "site:http://www.____.com" search, I was just wondering if this is one of those things Google keeps a memory of, if that makes sense.
Thanks for your response Mike.
-
That is a very good suggestion. I'll try it (a useful URL also so thanks for sharing).
Thanks for the response Matthew.
-
I think the much more likely culprit is that it is a new site. What do you get when you enter "site:http://www._____.com" in google? If the pages are indexed, one can't blame for the robots file for lack of rank.
Good luck!
Mike
-
Have you submitted the updated robots.txt to google? This is separate from updating the sitemap. Here is a google page to help you do this.
https://support.google.com/webmasters/answer/6078399?hl=en
Best!
Matthew
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Images on Website for SEO
Good Morning, We have a magento website with hundreds of different products that have slight size variations. The image for each of these products looks the same (the only difference between the products is some of the dimensions) .... Would you recommend using the same image for each of these products and just use a generic file name that describes the overall product or would you give each product its own image with it's specific product name as the file name? Should I use 1 image for 500 different sku's or should i rename the file the name of each individual sku and load an individual image? The end user will not know the difference since all of the images will appear identical, simply asking from an SEO perspective. Thanks
Technical SEO | | Prime850 -
Redirecting pages from a website to another
Hello Moz community, I’ve got a question and hope you can help! I’ve been working to improve my website’s ranking for the keywords “singing lessons London”. My current website url is http://www.sonic-crew-london.com and the page dedicated to the singing lessons is http://www.sonic-crew-london.com/booking/singinglessons.php I’ve recently bought the url http://www.singing-lessons-london.com which I hope will help to climb Google’s ranks a bit more easily for my chosen keywords. I thought I could redirect the old singing page to the new url. Is that something you would recommend me to do? Is there any specific procedure I should follow to make sure the transition runs smoothly? Any help really appreciated! Many thanks
Technical SEO | | SonicCrewLondon0 -
Website Down
Hello guys, My website hasn't been reachable for couple of hours today and I can't really understand why as no links have been built, all the best practices have been followed regarding on page optimization. I also checked google webmaster tools and there are no warning messages, crawl problems or anything so I don't understand why this has happened. Now for some reason the website is up and running again.
Technical SEO | | PremioOscar1 -
301 permanent re-direct
My site can be accessed either with or without the www. Does this affect my search positions and do error 301-redirect cause any ill effects?
Technical SEO | | fireman0 -
When will seomoz rank my website?
Hi all, I am really excited to be here. I joined seo moz three weeks ago. Seo moz crawled my website twice and still under competitive domain analysis my domain authority is 1 and everything else underneath is zero. I fixed some errors and warning after first crawl but still nothing in root domain metrics. Can anyone please explain whats happening? Thank you!
Technical SEO | | proaspirant0 -
Two websites with similar products
I have a website with similar products with different tld . I want to redirect a page for a keyword to another domain by using 301 redirect.Is it okay or google will consider as spam?Please help me out.
Technical SEO | | Alick3000 -
Does using parentheses affect the crawlers?
Quick question: if you using a parantheses around a keyword, do search bots still recognize the keyword? Fox ex: Welcome to a website about the National Basketball Association (NBA). Will the bots recognize that I'm trying to optimize to NBA and not (NBA)? Is this different for tags vs. actual body copy?
Technical SEO | | BPIAnalytics2