How to Configure Robots.txt File
-
How to configure correctly the Robots.txt File of the website. Need proper process to follow.Because a lot of my website URLs are excluded by google with the issues in the Robots.txt file.
-
In addition to effectdigitals comments, it partially depends on the CMS. For example a Standatrd WordPress robots.txt is formatted:
**User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php** I prefer to add a link to the sitemap.xml:**User-agent: ***
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.phpSitemap: https://www.website.com/sitemap.xml
To allow all web crawlers access to all content, just use:
**User-agent: ***
Disallow:Refer to https://moz.com/learn/seo/robotstxt for further advise.
-
It depends on how your unique website creates URLs and how they are formatted, it also depends upon the current contents of your robots.txt file. If you can share some examples of URLs that are blocked, which you think should not be blocked, and also the contents of your robots.txt file, someone can probably tell you what you did wrong
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How to check Robots.txt File
I want to know that how we are going to check that the Robots.txt File of the website is working properly. Kindly elaborate the mechanism for it please.
International SEO | | seobac1 -
Switch to Separate URL Configurations for International SEO?
We run an ecommerce website and sell to customers in the US and Canada. We recently realized that the way we serve content to our users isn't Google's recommended way. We use locale-adaptive pages in that some content changes slightly depending on where we think the user is located based on their IP address. But the URL doesn't change. Google's stance on locale-adaptive - https://support.google.com/webmasters/answer/6144055?hl=en&ref_topic=2370587 That being said, the changes are quite minor. It is usually only pricing and currency that changes depending on if we determine the user to be based in Canada or the US. However, I understand that there can be problems related to this configuration because of GoogleBot primarily using a US-based IP. We decided that the best course of action for us is to serve US content as our default content on all of our existing URLs. And Canadian content would be served using new url paths such as: example.com**/en-ca/**product1. All of this would also be configured to use hreflang tags. The problem we have run into is that it is a pretty huge development challenge to reconfigure how the site serves content when we have been using locale-adaptive for over a year. So developer resources would be taken away from other tasks and put toward this one for a relatively long time. Based on this information and the fact that we would like to both rank better in Canada and to follow Google's recommendations, how important would you say this change would be? I realize this isn't a black and white question and it depends a lot on business goals and preferences. However, I need to be able to gauge as best as I can how necessary it is to do this in order to make the decision of whether to do it or not. Any input is greatly appreciated!
International SEO | | westcoastcraig1 -
International Sites - Sitemaps, Robots & Geolocating in WMT
Hi Guys, I have a site that has now been launched in the US having originally just been UK. In order to accommodate this, the website has been set-up using directories for each country. Example: domain.com/en-gb domain.com/en-us As the site was originally set-up for UK, the sitemap, robots file & Webmaster Tools account were added to the main domain. Example: domain.com/sitemap.xml domain.com/robots.txt The question is does this now need changing to make it specific for each country. Example: The sitemap and robots.txt for the UK would move to: domain.com/en-gb/sitemap.xml domain.com/en-gb/robots.txt and the US would have its own separate sitemap and robots.txt. Example : domain.com/en-us/sitemap.xml domain.com/en-us/robots.txt Also in order to Geolocate this in WMT would this need to be done for each directory version instead of the main domain? Currently the WMT account for the UK site is verified at www.domain.com, would this need reverifying at domain.com/en-gb? Any help would be appreciated! Thanks!
International SEO | | CarlWint0 -
Help with hreflang configuration
We currently have different country versions of our site with a hreflang configuration; we have a spanish site targeted for Spain and then different subdirectories for every latin american country we're targeting to. Other than minimal differences, at the moment they are similar until we can handle providing unique content for each country. Our goal, then, would be to show the right one for every Google version, but also to avoid penalties. Our configuration is like this: ... Do you reckon this would be the right approach? Moz is currently telling us that there's duplicate content between the different country subdirectories, and to an extent there is, does that correlate with Google penalizing us for it? Also I've seen examples of self-referential alternates (such as including the in the Mexican version), or to put the canonical referencing the primary http://www.example.com/, will any of this help to our goals? Thanks a lot!
International SEO | | doctorSIM0 -
Blocking domestic Google's in Robots.txt
Hey, I want to block Google.co.uk from crawling a site but want Google.de to crawl it. I know how to configure the Robots.txt to block Google and other engines - is there a fix to block certain domestic crawlers? any ideas? Thanks B
International SEO | | Bush_JSM0 -
When I upload my app in chrome web store, it shows this error - "no manifest found in package please make sure to put manifest at the root directory of the zip package". Please explain me the process of putting manifest file.
Chrome Web store give error "no manifest found in package please make sure to put manifest at the root directory of the zip package"
International SEO | | SameerBhatia0 -
Can I add more than one XML sitemap in my Robots.txt File?
I have 3 domains all hosted on the same Magento Enterprise Platform. Each domain targets a different county. .co.uk, .com.au, and .com. Can I add all three sitemaps in my robots.txt file? www.example.com/sitemap.xml www.example.com.au/sitemap.xml www.example.co.uk/sitemap.xml
International SEO | | hfranz0 -
Robots.txt issue with indexation
Hello i have a problem with one of the rules for robots.txt i have a multilingual mutation of entire page on www.example.com/en/ I want to make indexable /allow/ the main page under /en/ but not indexable /disallow/ everything else under /en/* Please help me how to write the rule.
International SEO | | profesia0