How to set the crawler or reports to ignore
-
I have a mobile version of a site with a URL string that disables the mobile view on smartphones (view full site), string is like this example.com/page-name.html?mobile=off
I need the seomoz pro reports or crawler to ignore it because the crawler visits both versions of the site then reports them as duplicate content. Is there a setting page I haven't visited yet that will set this?
-
Thanks for the confirmation. I hadn't wanted to put that in the robots file because if inquisitive, non-robot traffic if you get my meaning, but in it goes; can't stand seeing errors.
-
Got it, thanks for jumping in so quick.
-
Hey Str8,
Donford is correct. You would need to specifically block our crawler from the mobile pages in your robots.txt file. Unfortunately, we don't currently have a way to disregard specific pages or errors in the web app, but we are looking to add that function sometime in the future.
I hope this helps. Please let me know if you have any other questions.
-
Hi Str8
**user-agent: rogerbot disallow: /*mobile=**
I think may do the trick... somebody may need to confirm this.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
The Pros & Cons of Free SEO Reports
Are the numerous Free SEO reports that are available around the internet, where I simply put a website address in and wait for the results actually any good? I tend to stick with Moz & Semrush which takes a little time pulling all the information I need together, but many of our clients are being approached by SEO's waving these free reports at them. Should I see these reports as valid, or ignore them and only concentrate on Moz. Any views will be greatly appreciated. -Christina
Moz Pro | | ChristinaRadisic1 -
Wild Fluctuations With Yahoo/Bing Moz Report
Whenever I get my weekly custom report, I usually see steady but moderate gains in Google or slight drops 1 to 3 spots. But, with Yahoo/Bing it's just crazy. +25 here, -47 here, etc, etc. Also, we even have keywords that we rank in the top 3 on with google, but not the top 50 with Bing or Yahoo. Not sure how that's possible. Nor do I understand how if Bing powers Yahoo, how we can have reports that show a dramatic increase (or decline) in Bing, but the opposite in Yahoo. Granted, not all the rankings are like that for Bing and Yahoo, but there are so many discrepancies, I'm starting to wonder if it's a software problem or something. Does this happen to anyone else? Thanks, Ruben
Moz Pro | | KempRugeLawGroup0 -
Getting my top keywords separated out in SEOmoz reports
I am using the standard functionality to produce weekly Moz reports. There does not seem to be a setting to show rankings of my most important keywords. It would be nice to have those high-level keywords on the first page. For example, I have 200 keywords in an account. I want the report to show on a page my 10 most important keywords. Is there a way to set up a Label to my keywords in order to product a report page just for those keywords?
Moz Pro | | clicktoshop0 -
Functionality of SEOmoz crawl page reports
I am trying to find a way to ask SEOmoz staff to answer this question because I think it is a functionality question so I checked SEOmoz pro resources. I also have had no responses in the Forum too it either. So here it is again. Thanks much for your consideration! Is it possible to configure the SEOMoz Rogerbot error-finding bot (that make the crawl diagnostic reports) to obey the instructions in the individual page headers and http://client.com/robots.txt file? For example, there is a page at http://truthbook.com/quotes/index.cfm month=5&day=14&year=2007 that has – in the header -
Moz Pro | | jimmyzig
<meta name="robots" content="noindex"> </meta name="robots" content="noindex"> This page is themed Quote of the Day page and is duplicated twice intentionally at http://truthbook.com/quotes/index.cfm?month=5&day=14&year=2004 and also at http://truthbook.com/quotes/index.cfm?month=5&day=14&year=2010 but they all have <meta name="robots" content="noindex"> in them. So Google should not see them as duplicates right. Google does not in Webmaster Tools.</meta name="robots" content="noindex"> So it should not be counted 3 times? But it seems to be? How do we gen a report of the actual pages shown in the report as dups so we can check? We do not believe Google sees it as a duplicate page but Roger appears too. Similarly, one can use http://truthbook.com/contemplative_prayer/ , here also the http://truthbook.com/robots.txt tells Google to stay clear. Yet we are showing thousands of dup. page content errors when Google Webmaster tools as shown only a few hundred configured as described. Anyone? Jim0 -
SeoMoz reporting 301 redirects I can't find
So I'm trying to find a bunch of redirects that SeoMoz is reporting that were created by Wordpress but I can't seem to find. I'm finding a bunch of redirects FROM pages like this: www.mysite.com/wp-content/themes/mimbo/page-name.com redirecting to www.mysite.com/page-name.com Anybody have any idea how I can track down the offending page that has a link to that url. I took a look at my templates and content with the pages and can't seem to find anything that would create it.
Moz Pro | | brandco0 -
Site explore reporting error over week
unable to dispaly anchor text error Doh! Roger is still working out the kinks with the new index and is having issues untangling anchor text data. We're currently showing anchor text data from the previous index, but we will update as soon as we can.
Moz Pro | | 1step2heaven120 -
0 values reported in anchor text distribution report
Can anyone explain the rows of 0 values for the majoity of anchor text reported in the Anchor Text Distribution section of OSE ? e.g I ran a report for a website which has around 2,500 inbound links and when I looked at the CSV file, there are about 1,500 anchor texts showing as 0. I'm sure there's either a simple explanation for this or its just a glitch. Can anyone help?
Moz Pro | | Websensejim0