How to block my web from SeoMOz Crawler
-
Hi,
I want to block mysite.com/forum from SeoMoz Crawler, how do i do that? i just want to block the forum and let the other part of my site still crawlable by seomoz. Thanks
Regards
-
Howdy
You'll need to block the SEOMoz crawler in your robots.txt file. Edit the file and insert this bit of code:
User-Agent: rogerbot Allow: /
Disallow: /forum/If that doesn't work, you might want to add a Wildcard too, making it:
Disallow: *****/forum/
If you're not too sure about editing the robots.txt file, then it might be an idea to check the SEOMoz guide: http://www.seomoz.org/learn-seo/robotstxt - if you're really unsure, it's best to get someone else to do it for you, as you don't want to block all of your site to all of the crawlers
Hope this helps
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Quality Issues: My blog is blocked on Google Search Engine
Hi Webmasters, I got an email from google team. The email is included below. **Google Webmaster Tools: Quality Issues on http://abcdblogger.com/**August 8, 2012 Dear site owner or webmaster of http://abcdblogger.com/, We've detected that some of your site's pages may be using techniques that are outside Google's Webmaster Guidelines. If you have any questions about how to resolve this issue, please see ourWebmaster Help Forum for support. Sincerely, Google Search Quality Team My blog is completely blocked on Google Search engine. I removed all existing posts and reinstalled a fresh version of wordpress and wrote a good article. I redirected all broken links my homepage with a 301. After making those changes I submitted a reconsideration request to Google, But they declined it. I doubt that the reason for blocking could be due to the backlinks pointing to my domain. I think Google's Disavow Tool help me to remove low quality backlinks, But how can I sort low quality backlinks using Opensite Explorer? If possible can you create a text file with all possible low quality links, So that I could submit it using Google Disavow Tool. Thanks.
Technical SEO | | hafiskani0 -
Same URL in "Duplicate Content" and "Blocked by robots.txt"?
How can the same URL show up in Seomoz Crawl Diagnostics "Most common errors and warnings" in both the "Duplicate Content"-list and the "Blocked by robots.txt"-list? Shouldnt the latter exclude it from the first list?
Technical SEO | | alsvik0 -
Warnings for blocked by blocked by meta-robots/meta robots Nofollow...how to resolve?
Hello, I see hundreds of notices for blocked by meta-robots/meta robots nofollow and it appears it is linked to the comments on my site which I assume I would not want to be crawled. Is this the case and these notices are actually a positive thing? Please advise how to clear them up if these notices can be potentially harmful for my SEO. Thanks, Talia
Technical SEO | | M80Marketing0 -
Web config redirects not working where a trailing slash is involved
I'm having real trouble with getting working redirects in place to use on a site we're re-launching with a modified url structure. Old URL: http://www.example.com/example_folder/ New URL: http://www.example.com/example-of-new-folder/ Now, where the old URL's have a trailing slash the web.config simply will not accept it. It says the URL can start with a slash, but not end with a slash. However, many of my URL's do end with a slash so I need a workaround. These are the rules I'm putting in place: <location path="example_folder/"></location> Thanks
Technical SEO | | AndrewAkesson0 -
Why mobi version of the file comes up higher on SERPs when compared to the web version?
hi Please see the URL http://news.oneindia.in/2011/10/22/tech-gmail-to-get-a-makeover-soon-google.html
Technical SEO | | greyniumseo
The corresponding mobile version is http://news.oneindia.mobi/2011/10/22/886893.html If we search for "Google video leaks; Gmail to get a make over soon" on Google the mobi version comes up instead of the web version. One reason could be because of the browser title. We do use meta title in our web version of the article. For the past few months our mobi version of the file comes up higher on SERPs when compared to the web version. What could be the reason? regards0 -
Differences between Lynx Viewer, Fetch as Googlebot and SEOMoz Googlebot Rendering
Three tools to render a site as Googlebot would see it: SEOMoz toolbar.
Technical SEO | | qlkasdjfw
Lynxviewer (http://www.yellowpipe.com/yis/tools/lynx/lynx_viewer.php )
Fetch as Googlebot. I have a website where I can see dropdown menus in regular browser rendering, Lynxviewer and Fetch as Googlebot. However, in the SEOMoz toolbar 'render as googlebot' tool, I am unable to see these dropdown menus when I have javascript disabled. Does this matter? Which of these tools is a better way to see how googlebot views your site?0 -
Targeting City via Web Server
Here's a question I can't seem to find an answer to. Does web hosting within a targeting city make a different in the engines? For example, a site targeting the Denver area, with web hosting in Denver. Will this boast the ranking or is targeting limited to countries? Thanks!
Technical SEO | | mkoster0