Welcome to the Q&A Forum

Browse the forum for helpful insights and fresh discussions about all things SEO.

Your browser does not seem to support JavaScript. As a result, your viewing experience will be diminished, and you have been placed in read-only mode.

Please download a browser that supports JavaScript, or enable it if it's disabled (i.e. NoScript).

How to check Robots.txt File

International SEO

1434

seobac last edited by

I want to know that how we are going to check that the Robots.txt File of the website is working properly. Kindly elaborate the mechanism for it please.
1 Reply Last reply
Reply Quote 1
fghgkjh1233 last edited by

My site also have this problem please help.

Social Sathee
1 Reply Last reply
Reply Quote 0
jasongmcmahon last edited by

Hi,

After you've read the article Nozzle recommended, use the https://technicalseo.com/tools/robots-txt/ tool to test whatever you put together.
1 Reply Last reply
Reply Quote 1
Nozzle last edited by

Here is a great resource to answer your question: https://moz.com/learn/seo/robotstxt.
1 Reply Last reply
Reply Quote 2

Got a burning SEO question?

Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.

Start my free trial

Browse Questions

View

From

Sorted by

With category

Explore more categories

Related Questions

How to Configure Robots.txt File

How to configure correctly the Robots.txt File of the website. Need proper process to follow.Because a lot of my website URLs are excluded by google with the issues in the Robots.txt file.
International SEO | | seobac

1
International advice.... can anyone help and check my site?

Hi ALL, I'm running 3 sites, internationally .com, com.au and co.nz Can anyone please look at my site and give me feedback about the hreflang tags, I ran a W3C and i have errors stating https://validator.nu/?doc=http%3A%2F%2Fzenory.co.nz for www.zenory.com and its relevant domains
International SEO | | edward-may

0
International Sites - Sitemaps, Robots & Geolocating in WMT

Hi Guys, I have a site that has now been launched in the US having originally just been UK. In order to accommodate this, the website has been set-up using directories for each country. Example: domain.com/en-gb domain.com/en-us As the site was originally set-up for UK, the sitemap, robots file & Webmaster Tools account were added to the main domain. Example: domain.com/sitemap.xml domain.com/robots.txt The question is does this now need changing to make it specific for each country. Example: The sitemap and robots.txt for the UK would move to: domain.com/en-gb/sitemap.xml domain.com/en-gb/robots.txt and the US would have its own separate sitemap and robots.txt. Example : domain.com/en-us/sitemap.xml domain.com/en-us/robots.txt Also in order to Geolocate this in WMT would this need to be done for each directory version instead of the main domain? Currently the WMT account for the UK site is verified at www.domain.com, would this need reverifying at domain.com/en-gb? Any help would be appreciated! Thanks!
International SEO | | CarlWint

0
Blocking domestic Google's in Robots.txt

Hey, I want to block Google.co.uk from crawling a site but want Google.de to crawl it. I know how to configure the Robots.txt to block Google and other engines - is there a fix to block certain domestic crawlers? any ideas? Thanks B
International SEO | | Bush_JSM

0
When I upload my app in chrome web store, it shows this error - "no manifest found in package please make sure to put manifest at the root directory of the zip package". Please explain me the process of putting manifest file.

Chrome Web store give error "no manifest found in package please make sure to put manifest at the root directory of the zip package"
International SEO | | SameerBhatia

0
Can I add more than one XML sitemap in my Robots.txt File?

I have 3 domains all hosted on the same Magento Enterprise Platform. Each domain targets a different county. .co.uk, .com.au, and .com. Can I add all three sitemaps in my robots.txt file? www.example.com/sitemap.xml www.example.com.au/sitemap.xml www.example.co.uk/sitemap.xml
International SEO | | hfranz

0
Robots.txt issue with indexation

Hello i have a problem with one of the rules for robots.txt i have a multilingual mutation of entire page on www.example.com/en/ I want to make indexable /allow/ the main page under /en/ but not indexable /disallow/ everything else under /en/* Please help me how to write the rule.
International SEO | | profesia

0
What's the best strategy for checking international rankings?

Hi There- I am looking to optimize sites serving the UK and Austrailia markets. I feel like I have a good handle on how to go about doing that, but what I am fuzzy on is, what's the best way to monitor the SERPs for the keywords I am targeting. I know based on experience that if I just search google.com.au from here in the states, my results will be 'americanized' and may/probably won't accurately reflect what someone would see if they were search from Austrailia. Are there any good tools or tactics for seeing what searchers in the countries I am focusing on woudl see? Thanks! Jason
International SEO | | phantom

0