Robots.txt Help
-
I need help to create robots.txt file.
Please let me know what to add in the file. any real example or working example.?
-
Michael, from what i can tell, your website is built using WordPress. We typically recommend installing the Yoast SEO plugin and using that--which will help with your robots.txt file. If you need more information, take a look here: https://yoast.com/wordpress-robots-txt-example/
Generally, most of your site won't need to be disallowed in the robots.txt file, unless you're using tags and categories on your site. Yoast typically helps disallow the proper directories that you need to disallow.
One thing that you need to be aware of is the fact that you don't want to disallow your .CSS or .JS files on your site, many of the themes nowadays will put those files in your wp-admin folder--which by default typically gets disallowed.
-
This is the site I used to really get a good understanding of how to create a robots.txt file: http://www.robotstxt.org/
-
A very basic robots.txt file would look something like the below
User-agent: *
Sitemap: http://www.yourwebsite.com/sitemap.xml
Disallow: http://www.yourwebsite.com/url-you-dont-want-indexed
Disallow: http://www.yourwebsite.com/another-url-you-dont-want-indexedHope that helps
-
Include sitemaps. Disallow: Pages that you don't want indexed: search pages, login pages, core admin files.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Content Help for Dealers
What type of content would anyone recommend writing for power sports dealers (ATV dealers, Motorcycle Dealers, Jet Ski Dealers, etc) or outdoor power equipment dealers (tractors, lawn mowers, etc) when they're website consists mainly of inventory pages? These dealers are trying to improve brand awareness, but creating content that answers searchers' questions/intent is tough and I want to make sure I am on the right track. I'm trying to create unique content. I am optimizing existing pages and then so far I've been writing brand pages, describing the brands, advertising that they carry this brand, creating links and call-to-actions to the inventory pages,etc. I want to first create authority and crawlable content for this brand. From there, I have been trying to create product category pages, describing the top products under that brand and working to creating product comparison content instead of simply describing it. Why Buy type of stuff, but that gets tricky to make unique. Any suggestions on unique content or better strategies versus just brand descriptions, product descriptions/comparisons, etc? I also want to make sure that creating multiple pages focused on one brand and an overall category isn't cannibalization of a topic. Obviously each page is slightly different and gradually going into more detail, but I want to make sure. Any recommendations on types of content or different strategies would be helpful! Also - I should mention that I am limited by the platform. I cannot create/utilize a blog page or anything like that. Thanks!
Intermediate & Advanced SEO | | Crichardson19900 -
Did not get Good reply on my previous query. Can anyone help me?
I did not get satisfactory answer on my previous question here - https://moz.com/community/q/please-provide-solution-for-my-website-duplicate-content-problem Please help me.
Intermediate & Advanced SEO | | Alexa.Hill0 -
Help with facet URLs in Magento
Hi Guys, Wondering if I can get some technical help here... We have our site britishbraces.co.uk , built in Magento. As per eCommerce sites, we have paginated pages throughout. These have rel=next/prev implemented but not correctly ( as it is not in is it in ) - this fix is in process. Our canonicals are currently incorrect as far as I believe, as even when content is filtered, the canonical takes you back to the first page URL. For example, http://www.britishbraces.co.uk/braces/x-style.html?ajaxcatalog=true&brand=380&max=51.19&min=31.19 Canonical to... http://www.britishbraces.co.uk/braces/x-style.html Which I understand to be incorrect. As I want the coloured filtered pages to be indexed ( due to search volume for colour related queries ), but I don't want the price filtered pages to be indexed - I am unsure how to implement the solution? As I understand, because rel=next/prev implemented ( with no View All page ), the rel=canonical is not necessary as Google understands page 1 is the first page in the series. Therefore, once a user has filtered by colour, there should then be a canonical pointing to the coloured filter URL? ( e.g. /product/black ) But when a user filters by price, there should be noindex on those URLs ? Or can this be blocked in robots.txt prior? My head is a little confused here and I know we have an issue because our amount of indexed pages is increasing day by day but to no solution of the facet urls. Can anybody help - apologies in advance if I have confused the matter. Thanks
Intermediate & Advanced SEO | | HappyJackJr0 -
Content question please help
Would content behind a drop down on this site Https://www.homeleisuredirect.com/pool_tables/english_pool_tables/ you have to click the - more about English pool tables text under the video Work just as well for SEO as content on the page like this site http://www.pooltablesonline.co.uk/uk-slate-bed-pool-tables.asp
Intermediate & Advanced SEO | | BobAnderson0 -
HELP! How do I stop scraper sites - is there any recourse?
Our site has lots of unique content and photos and it is constantly being scraped and posted on other websites. Most of these are no-name sites that pop up and exist for adwords revenue. Aside from the fact that we don't want our content being copied, this is an SEO nightmare because they often link back to us from pages that are stuffed with keywords and have very low domain authority (it's a form of negative SEO). My question is: Does anyone have experience with fighting this phenonmenon? What have you done that is effective? Does anyone have experience with a service such as http://www.dmca.com/ProtectionPro.aspx ? Does it work/is it worth it? Any input is appreciated!
Intermediate & Advanced SEO | | YairSpolter0 -
I want to Disavow some more links - but I'm only allowed one .txt file?
Hey guys, Wondering if you good people could help me out on this one? A few months back (June 19) I disavowed some links for a client having uploaded a .txt file with the offending domains attached. However, recently I've noticed some more dodgy-looking domains being indexed to my client's site so went about creating a new "Disavow List". When I went to upload this new list I was informed that I would be replacing the existing file. So, my question is, what do I do here? Make a new list with both old and new domains that I plan on disavowing and replace the existing one? Or; Just replace the existing .txt file with the new file because Google has recognised I've already disavowed those older links?
Intermediate & Advanced SEO | | Webrevolve0 -
About to launch a new e-commerce site need help with anchor text's
Hey guys, I need some advice regarding the keywords I want to target for a new website. The website is e-commerce and is about 60% done but i wanted some advice. home page - http://www.diamondengagement.com/ For link building purposes I only want to target the domain name for the anchor text "diamond engagement" and "diamondengagement.com" for internal pages like .... http://www.diamondengagement.com/engagement-rings/ I want this page to rank for "engagement rings" but that's a very competitive keyword. I was thinking for the first 3 or 4 months I only build links to this page with the domain name again "diamond engagement" and phrase anchor text's using "diamond engagement" than just out right building "engagement rings" or "engagement ring" to start off. What are your thoughts??
Intermediate & Advanced SEO | | harrykabadaian0 -
Help! Is rel cononical impacting me?
Hi there. My personal site www.adamlewis.info has higher Domain Authority and Moz rank and more linking domains than the top ranking site for my name "Adam Lewis" My landing page /adam-lewis has an A Grade. Yet I am still on page 2 behind what appear to be weaker domains. The on-site report says I am not making appropriate use of Rel Cononical. This is a bit techy for me. Can anyone explain how this might or might not be affecting my ranking for "adam lewis"? Thanks guys! Adam
Intermediate & Advanced SEO | | adamlewis100