Unsolved What would the exact text be for robots.txt to stop Moz crawling a subdomain?
-
I need Moz to stop crawling a subdomain of my site, and am just checking what the exact text should be in the file to do this.
I assume it would be:
User-agent: Moz
Disallow: /But just checking so I can tell the agency who will apply it, to avoid paying for their time with the incorrect text!
Many thanks.
-
To disallow Moz from crawling a specific subdomain, you would need to add a robots.txt file to the root directory of that subdomain with the following content:
User-agent: rogerbot
Disallow: /This will disallow Moz's web crawler, Rogerbot, from crawling any page or file within the subdomain. Keep in mind that this will only prevent Moz from crawling the subdomain - other search engines or bots may still be able to access it unless you add specific disallow rules for them as well.
-
@Simon-Plan No, when you put just slash / you will disallow everything.
Instead you need to put /foo/ where foo is your subdomain. Please see here for a reference to some relevant examples: https://searchfacts.com/robots-txt-allow-disallow-all/
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
is my robot file correct
hi, can anyone let me know if my robot file is correct. my pages and wordpress posts are being indexed but not showing in serps and wondering if my robot file is wrong https://www.in2town.co.uk/robots.txt
SEO Tactics | | blogwoman10 -
How Can I influence the Google Selected Canonical
Our company recently rebranded and launched a new website. The website was developed by an overseas team and they created the test site on their subdomain. The only problem is that Google crawled and indexed their site and ours. I noticed Google indexed their sub domain ahead of our domain and based on Search Console it has deemed our content as the duplicate of theirs and the Google selected theirs as the canonical.
Community | | Spaziohouston
The website in question is https://www.spaziointerni.us
What would be the best course of action to get our content ranked and selected instead of being marked as the duplicate?
Not sure if I have to modify the content to make it more unique or have them submit a removal in their search console.
Our indexed pages continue to go down due to this issue.
Any help is greatly appreciated.1 -
Good to use disallow or noindex for these?
Hello everyone, I am reaching out to seek your expert advice on a few technical SEO aspects related to my website. I highly value your expertise in this field and would greatly appreciate your insights.
Technical SEO | | williamhuynh
Below are the specific areas I would like to discuss: a. Double and Triple filter pages: I have identified certain URLs on my website that have a canonical tag pointing to the main /quick-ship page. These URLs are as follows: https://www.interiorsecrets.com.au/collections/lounge-chairs/quick-ship+black
https://www.interiorsecrets.com.au/collections/lounge-chairs/quick-ship+black+fabric Considering the need to optimize my crawl budget, I would like to seek your advice on whether it would be advisable to disallow or noindex these pages. My understanding is that by disallowing or noindexing these URLs, search engines can avoid wasting resources on crawling and indexing duplicate or filtered content. I would greatly appreciate your guidance on this matter. b. Page URLs with parameters: I have noticed that some of my page URLs include parameters such as ?variant and ?limit. Although these URLs already have canonical tags in place, I would like to understand whether it is still recommended to disallow or noindex them to further conserve crawl budget. My understanding is that by doing so, search engines can prevent the unnecessary expenditure of resources on indexing redundant variations of the same content. I would be grateful for your expert opinion on this matter. Additionally, I would be delighted if you could provide any suggestions regarding internal linking strategies tailored to my website's structure and content. Any insights or recommendations you can offer would be highly valuable to me. Thank you in advance for your time and expertise in addressing these concerns. I genuinely appreciate your assistance. If you require any further information or clarification, please let me know. I look forward to hearing from you. Cheers!0 -
Blocking in Robots.txt and the re-indexing - DA effects?
I have two good high level DA sites that target the US (.com) and UK (.co.uk). The .com ranks well but is dormant from a commercial aspect - the .co.uk is the commercial focus and gets great traffic. Issue is the .com ranks for brand in the UK - I want the .co.uk to rank for brand in the UK. I can't 301 the .com as it will be used again in the near future. I want to block the .com in Robots.txt with a view to un-block it again when I need it. I don't think the DA would be affected as the links stay and the sites live (just not indexed) so when I unblock it should be fine - HOWEVER - my query is things like organic CTR data that Google records and other factors won't contribute to its value. Has anyone ever blocked and un-blocked and whats the affects pls? All answers greatly received - cheers GB
Technical SEO | | Bush_JSM0 -
Does anyone know where I can find the Moz Video explaining how to use the Craw Diagnostic Feature? Thank!
I am starting to use the craw diagnostics(specifically duplicate content) and I know there was a very helpful tutorial video i saw earlier but I cant seem to find it now
Getting Started | | John-Francis0 -
Is there a way MOZ can help me get HQ links?
I'm new to MOZ, I'm on the niche sites building. Is there an easy way to find HQ pages to post to with MOZ? Like it's with Market samurai.
Getting Started | | bishop230 -
Trending Bugs in Moz Analytics
The FAQ’s We hope we can help! Below are the trending issues at the moment. If you don’t see your question addressed or need further help, send us a message at http://moz.com/help/contact. Our Help Team gumshoes will investigate your issue and respond shortly. Where did my Campaigns Go?
Getting Started | | Abe_Schmidt
Whenever your account is suspended, your campaigns are archived. The good news is that reviving archived accounts is really simple. Go to your Campaigns section: http://pro.moz.com/campaigns
At the top of the page there is a "Archived Campaigns" Tab, give it a click.
From here, you should easily be able to "activate" your campaigns. My Google Analytics profile won’t stay connected to my campaign and/or I am missing profiles in my GA Settings. Google goes through a process called Oauth when it comes to authorizing access of services that are linked to your Google account. Under any single Google account, there are about 20 tokens account wide. Those tokens are used to provide access for a variety of services from Gmail to apps on your phone. Once you hit the limit, which is 20 for most users, the system automatically revokes the oldest token to provide a new one. I'm not certain if this is causing the issue on your account, but it is a great place to start troubleshooting. You may be able to correct the issue by manually revoking your tokens to make room for new ones on your account and then reconnecting the account to Moz. Here are the instructions on how to do that: Follow this link https://accounts.google.com/b/0/IssuedAuthSubTokens. This page displays the current OAuth tokens you are using. Once you reach the page, simply press the revoke button (illustrated on this screenshot: http://screencast.com/t/vjh3KrjRRIe) for services that you are not using right now. Once you are done with, that simply go back to your campaign settings, disconnect your GA profile and reconnect. The below process may also fix this issue for you. Head to your campaign settings page on your overview page. Disconnect your Google Analytics connection. Go back to your settings page and click on "connect account." Please make sure you log into the correct GA account. 🙂 Hurray! This should let you grab the most current traffic data! Oops! try refreshing page! I can’t access any of my Campaign data.
This issue is normally machine specific and can be a bit complex. An individual forum has been created to address this: https://seomoz.zendesk.com/entries/28203486-Oops-Try-Refreshing-the-Page-Error-showing-on-all-pages- Unable to retrieve historical ranking CSVs. Some users did not receive their requested Historical Rankings CSVs. This has mostly been affected by a change or update to your competitors after a campaign has been setup for a while. Please send your request to retrieve this data with http://moz.com/help/contact.8 -
Cant download my crawl csv
When I click on the [download csv] in my crawl campaign section nothing happens.
Getting Started | | digitalmedialounge0