Unsolved What would the exact text be for robots.txt to stop Moz crawling a subdomain?
-
I need Moz to stop crawling a subdomain of my site, and am just checking what the exact text should be in the file to do this.
I assume it would be:
User-agent: Moz
Disallow: /But just checking so I can tell the agency who will apply it, to avoid paying for their time with the incorrect text!
Many thanks.
-
To disallow Moz from crawling a specific subdomain, you would need to add a robots.txt file to the root directory of that subdomain with the following content:
User-agent: rogerbot
Disallow: /This will disallow Moz's web crawler, Rogerbot, from crawling any page or file within the subdomain. Keep in mind that this will only prevent Moz from crawling the subdomain - other search engines or bots may still be able to access it unless you add specific disallow rules for them as well.
-
@Simon-Plan No, when you put just slash / you will disallow everything.
Instead you need to put /foo/ where foo is your subdomain. Please see here for a reference to some relevant examples: https://searchfacts.com/robots-txt-allow-disallow-all/
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
is my robot file correct
hi, can anyone let me know if my robot file is correct. my pages and wordpress posts are being indexed but not showing in serps and wondering if my robot file is wrong https://www.in2town.co.uk/robots.txt
SEO Tactics | | blogwoman10 -
Unsolved Crawling only the Home of my website
Hello,
Product Support | | Azurius
I don't understand why MOZ crawl only the homepage of our webiste https://www.modelos-de-curriculum.com We add the website correctly, and we asked for crawling all the pages. But the tool find only the homepage. Why? We are testing the tool before to suscribe. But we need to be sure that the tool is working for our website. If you can please help us.0 -
Unsolved Moz crawler not crawling on my site
Hi all, im facing an issue where moz crawler is unable to crawl my site. The following error keeps showing Our crawler was banned by a page on your site, either through your robots.txt, the X-Robots-Tag HTTP header, or the meta robots tag. This is my robots.txt file : https://www.wearefutureheads.com/robots.txt I'm not sure what else am I missing.. can anyone help
Product Support | | teikh0 -
Slug redirect to existing subdomain?
Hi everyone, I'm Levi, new here and new in the world of SEO so please don't judge if my question is silly. Back on the days when my website was built we tought of having subdomains to help with SEO. We have a couple subdomains set up https://stansted.tonorwich.uk, https://heathrow.tonorwich.uk, https://gatwick.tonorwich.uk, https://luton.tonorwich.uk etc .
Link Building | | Leviiii
We do have some content on these, but from seo perspective creating a slug and redirecting it to the subdomains would be beneficial? Would it make any difference at all, good or bad? Ex. Creating
https://tonorwich.uk/taxi-minibus-tofrom-stansted and redirecting it to https://stansted.tonorwich.uk
Will this help or we better create different content on the slugs? Waiting for your opinions.
Thanks in advance.0 -
KD according to MOZ.
Hello, I want to know how moz calculates the keyword difficulty? What are the basic things which moz pro consider while calculating keyword difficulty? I wanna Calculate the KD of JB Battery Keywords,
Getting Started | | Shafqat_mehmood0 -
MOZ flagging urls as longer than 75 characters when they aren't
As an example: https://www.abthermal.com/3m-70-70hdt-self-fusing-silicone-rubber-electrical-tape.html Moz has flagged this as URL too long... but guideline is that happens at 75 characters or more? I count 55..?? or does the 75 limit have to include the domain? Thanks, Adrian
Getting Started | | abtechgroup0 -
Duplicate Content after Moz Site Audit
Hello folks, So I signed up for the trial version of the Moz tool and ran an initial site audit. One of the site audit results is confusing me.
Getting Started | | jjimen03
It reports that there are two pages with duplicate content ( Each page has a duplicate page with duplicate content in it).
When I take a look at what those pages are, here is what I see: mysite.com/Contact-Us.html
mysite.com/contact-us.html
( The difference in the above is the Contact and Us, the first letters are capitalized on one of the URLS) mysite.com/index.html
mysite.com Now I am confused because for one thing, I don't have 2 Contact Us html files uploaded on my hosting server.
Why is Moz seeing 2 Contact Us pages? How to remove one? Regarding my home page, why is it flagging the same page as two different pages? How to remove of them?0 -
MOZ Monthly Custom Report Question
How do I set the Moz to produce monthly report on 1st of every month. I dont see any option there. we have 50+ Clients in Moz now and we need to generate report on 1st day of all month to send them report. Its 3rd Oct today and I don't see my previous month report being generated. Can you guys help me
Getting Started | | emarketexperts0