Robots.txt Help
-
I need help to create robots.txt file.
Please let me know what to add in the file. any real example or working example.?
-
Michael, from what i can tell, your website is built using WordPress. We typically recommend installing the Yoast SEO plugin and using that--which will help with your robots.txt file. If you need more information, take a look here: https://yoast.com/wordpress-robots-txt-example/
Generally, most of your site won't need to be disallowed in the robots.txt file, unless you're using tags and categories on your site. Yoast typically helps disallow the proper directories that you need to disallow.
One thing that you need to be aware of is the fact that you don't want to disallow your .CSS or .JS files on your site, many of the themes nowadays will put those files in your wp-admin folder--which by default typically gets disallowed.
-
This is the site I used to really get a good understanding of how to create a robots.txt file: http://www.robotstxt.org/
-
A very basic robots.txt file would look something like the below
User-agent: *
Sitemap: http://www.yourwebsite.com/sitemap.xml
Disallow: http://www.yourwebsite.com/url-you-dont-want-indexed
Disallow: http://www.yourwebsite.com/another-url-you-dont-want-indexedHope that helps
-
Include sitemaps. Disallow: Pages that you don't want indexed: search pages, login pages, core admin files.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Help me to understand why this page doesn't rank
Hello everyone. I am trying to understand why most of my website category pages don't show up in the in the first 50 organic results on Google, despite my high website DA and high PA of those pages. We used to rank high a few years ago, not clear why most of those pages have almost completely disappeared. So, just to take one as an example, please, help me to understand why this page doesn't shows up in the first 50 organic search results for the keyword "cello sheet music": http://www.virtualsheetmusic.com/downloads/Indici/Cello.html I really can't explain why, unless we are under some sort of "penalization" or similar (a curse?!)... I have analyzed any possible metric, and can't find a logical explanation. Looking forward for your thoughts guys! All the best, Fab.
Intermediate & Advanced SEO | | fablau0 -
Unexplained Drop In Ranking and Traffic-HELP!
I operate a real estate web site in New York City (www.nyc-officespace-leader.com). It was hit by Penguin in April 2012, with search volume falling from 6,800 per month in March 2012 to 3,300 by June 2012. After refreshing content and changing the theme, volume recovered to 4,300 per month in October 2013. There was a big improvement in early October 2013, perhaps tied to a Panda update. In November 2013 I hired an SEO company. They are reputable; on MOZ's recommended list. After following all their suggestions (searching and removing duplicate content, disavowing toxic links, improving the site structure to make it easier for Google to index listings, re-writing ten key landing pages, improving the design of the user interface) ranking and traffic started to decline in April of 2014 and crashed in June 2014 after an upgraded design with improved user interface was launched. Search volume is went from 4700 in March to around 3800 in June. However ranking on the keywords that generate conversions has really declined, and clicks from those terms are down at least 65%. My online business is severely compromised after I have spent almost double the anticipated budget to improve ranking and conversion. A few questions: 1. Could a drop in the number of domains lining to our site have led to this decline? About 30 domains that had toxic links to us agreed to remove them. We had another 70 domains disavowed in late April. We only have 78 domains pointing to our domain now, far less than before (see attached AHREFs image). It seems there is a correlation in the timeline between the number of domains pointing to us and ranking performance. The number of domains pointing to us has never been this low. Could this be causing the drop? My SEO firm believes that the quality of these links are very low and the fact that many are gone is in fact a plus. 2. The number of indexed pages has jumped to 851 from 675 in early June (see attached image from Google Webmaster tools), right after a site upgrade. The number of pages in the site map is around 650. Could the indexation of the extra 175 page somehow have diluted the quality of the site in Google's eyes? We have filed removal request for these pages in Mid June and again last week with Google but they still appear. In 2013 we also launched an upgrade and Google indexed an extra 500 pages (canonical tags were not set up correctly) and search volume and ranking collapsed. Oddly enough when the number of pages indexed by Google fell, ranking improved. I wonder if something similar has occurred. 3. May 2014 Panda update. Many of our URLs are product URLs of listings. They have less than 100 words. Could Google suddenly be penalizing us for that? It is very difficult to write descriptions of hundreds of words for products that change quickly. I would think the Google takes this into account. If someone could present some insight into this issue I would be very, very grateful. I have spent over $25,000 on SEO reports, wireframe design and coding and now find myself in a worse position than when I started. My SEO provider is now requesting that I purchase even more reports for several thousand dollars and I can't afford it, nor can I justify it after such poor results. I wish they would take it upon themselves to identify what went wrong. In any case, if anyone has any suggestions I would really appreciate it. I am very suspicious that this drop started in earnest at the time of link removal and the disavow and accelerated at the time of the launch of the upgrade. Thanks, Alan XjSCiIdAwWgU2ps e5DerSo tYqemUO
Intermediate & Advanced SEO | | Kingalan10 -
Urgent Help - Ecommerce URL best practice for SEO
Guys i need some urgent help here as we need to get this sorted out soon. We have a page similar to wayfair shop the look: www.wayfair.com/Shop-The-Look/ What are the best practices for URL structure if we applies 2-3 filters? Is wayfair style good for SEO? FYI: We create our crawlable, link friendly AJAX website using pushstate() but unsure of the structure for this case. We followed http://moz.com/blog/create-crawlable-link-friendly-ajax-websites-using-pushstate advice.
Intermediate & Advanced SEO | | WayneRooney0 -
Duplicate Page Content / Titles Help
Hi guys, My SEOmoz crawl diagnostics throw up thousands of Dup Page Content / Title errors which are mostly from the forum attached to my website. In-particular it's the forum user's profiles that are causing the issue, below is a sample of the URLs that are being penalised: http://www.mywebsite.com/subfolder/myforum/pop_profile.asp?mode=display&id=1308 I thought that by adding - http://www.mywebsite.com/subfolder/myforum/pop_profile.asp to my robots.txt file under 'Ignore' would cause the bots to overlook the thousands of profile pages but the latest SEOmoz crawl still picks them up. My question is, how can I get the bots to ignore these profile pages (they don't contain any useful content) and how much will this be affecting my rankings (bearing in mind I have thousands of errors for dup content and dup page titles). Thanks guys Gareth
Intermediate & Advanced SEO | | gaz33420 -
How to Disallow Tag Pages With Robot.txt
Hi i have a site which i'm dealing with that has tag pages for instant - http://www.domain.com/news/?tag=choice How can i exclude these tag pages (about 20+ being crawled and indexed by the search engines with robot.txt Also sometimes they're created dynamically so i want something which automatically excludes tage pages from being crawled and indexed. Any suggestions? Cheers, Mark
Intermediate & Advanced SEO | | monster990 -
Will links to a subdomain help it rank?
I have an affiliate subdomain on a larger company's domain. (For example I have: www.victor.company.com on www.company.com). Would working to attain backlinks to the subdomain help it rank or will I just be putting forth my effort and helping the domain rank?
Intermediate & Advanced SEO | | VictorVC0 -
I am not a SEO expert , need a gentleman to help me
I have been hit by Latest google update . dont have enough money to pay seo consultants , just wondering if anyone expert can off a free analysis of my site and can point problems. Its not a adsence site . My living depends on this site so cant just keep on doing experiments , by the looks I have been hit hard by negative seo by competitor . I give url in private . Is there any one kind here who can help in private . Would really appreciate .
Intermediate & Advanced SEO | | HateDoingSEO0 -
Block all but one URL in a directory using robots.txt?
Is it possible to block all but one URL with robots.txt? for example domain.com/subfolder/example.html, if we block the /subfolder/ directory we want all URLs except for the exact match url domain.com/subfolder to be blocked.
Intermediate & Advanced SEO | | nicole.healthline0