Restrict rogerbot for few days
-
Hi Team,
I have a subdomain that built in Zendesk's CRM system. Now, I want to restrict Moz crawler (rogerbot) for crawling this complete subdomain for a few days, but I am not able to edit the robots.txt file of the subdomain, because this is a shared file and Zendesk is not allowing to edit it. Could you please let me know the alternative way to restrict rogerbot to crawl this subdomain?
I am eagerly awaiting your quick response.
Thanks
-
Do you have FTP access and / or Yoast installed (if this is a WordPress website)?
Thanks,
Zack -
Not from what I could think about for how a specific user-agent is being treated just like you would want with Rogerbot.
-
Is there any way to restrict via Google Tag Manager?
-
Hey there!
Thanks for reaching out to us!
Unfortunately you would need access to the robots.txt file in order to block us from crawling a certain subdomain. I'm sorry I can't be of anymore help here. Please do reach out to help@moz.com if you have any further questions,
Best,
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Are tags in blogs worth anything in SEO value these days and can i remove duplicate title tags?
I have around 12 duplicate tags in the blog and the general consensus is to noindex./nofollow the tag pages which is not an awful way to do things. Should I just arrange my tags better or remove them all together? I believe it's the fact that the same page can be accessed by 3 different url's because of the tags it's under and that in the tag urls, only teaser descriptions of the posts are being display, not the full post, but the seomoz tools still consider it as duplicated content.
Moz Pro | | SEM_at_Lees0 -
Only few pages (308 pages of 1000 something pages) have been crawled and diagnosed in 4 days, how many days till the entire website is crawled complete?
Setup campaign about 4-5 days ago and yesterday rogerbot said 308 pages were crawled and the diagnostics were provided. This website has over 1000+ pages and would like to know how long it would take for roger to crawl the entire website and provide diagnostics. Thanks!
Moz Pro | | TejaswiNaidu0 -
Does Rogerbot respect the robots.txt file for wildcards?
Hi All, Our robots.txt file has wildcards in it, which Googlebot recognizes. Can anyone tell me whether or not Rogerbot recognizes wildcards in the robots.txt file? We've done a Rogerbot site crawl since updating the robots.txt file and the pages that are set to disallow using the wildcards are still showing. BTW, Googlebot is not crawling these pages according to Webmaster Tools. Thanks in advance, Robert
Moz Pro | | AC_Pro0 -
How can I change the day my keyword rankings are updated?
I have several campaigns which currently get their weekly keyword ranking reports updated on different days. I'd like all of them updated on the same day, so I can just check in one a week for everything.
Moz Pro | | RichardDavies0 -
Can you change crawl day of week?
Can I somehow sync the day of the week for each of my campaigns' crawls, so that all campaigns are updated on the same day?
Moz Pro | | ATShock0 -
Does anyone know of a crawler similar to SEOmoz's RogerBot?
As you probably know SEOmoz had some hosting and server issues recently, and this came at a terrible time for me... We are in the middle of battling some duplicate content and crawl errors and need to get a fresh crawl of some sites to test things out before we are hit with the big one? Before I get a million thumbs downs- I love and will continue to use SEOmoz, just need something to get me through this week ( or until Roger is back! )!
Moz Pro | | AaronSchinke1 -
What time of day are keyword rankings updated
The keyword rankings in one of my campaigns update on Tuesday. It looks like they updated today but when I downloaded to CSV, some say last Tuesday. Is this because they didn't change or are they still updating? Today is Tuesday. i.e., do they update at a specific time and if so, what time?
Moz Pro | | mikescotty0 -
SEOmoz Q&A down last few days
It seems the Q&A section had some issues since the 25th. Users could post new Q&As but they were not visible to most users. Roger was caught slacking! The issue appears to be resolved at this time. I just wanted to share to anyone who asked a question the past few days who did not receive a response you may wish to repost your question as many readers will not go back and check questions from prior days.
Moz Pro | | RyanKent1