How to set the crawler or reports to ignore
-
I have a mobile version of a site with a URL string that disables the mobile view on smartphones (view full site), string is like this example.com/page-name.html?mobile=off
I need the seomoz pro reports or crawler to ignore it because the crawler visits both versions of the site then reports them as duplicate content. Is there a setting page I haven't visited yet that will set this?
-
Thanks for the confirmation. I hadn't wanted to put that in the robots file because if inquisitive, non-robot traffic if you get my meaning, but in it goes; can't stand seeing errors.
-
Got it, thanks for jumping in so quick.
-
Hey Str8,
Donford is correct. You would need to specifically block our crawler from the mobile pages in your robots.txt file. Unfortunately, we don't currently have a way to disregard specific pages or errors in the web app, but we are looking to add that function sometime in the future.
I hope this helps. Please let me know if you have any other questions.
-
Hi Str8
**user-agent: rogerbot disallow: /*mobile=**I think may do the trick... somebody may need to confirm this.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Our crawler was not able to access the robots.txt file on your site.
Good morning, Yesterday, Moz gave me an error that is wasn't able to find our robots.txt file. However, this is a new occurrence, we've used Moz and its crawling ability many times prior; not sure why the error is happening now. I validated that the redirects and our robots page are operational and nothing is disallowing Roger in our robots.txt. Any advice or guidance would be much appreciated. https://www.agrisupply.com/robots.txt Thank you for your time. -Danny
Moz Pro | | Danny_Gallagher0 -
Inbound links not showing in reports
My clients site so far only has a few inbound links. If I do a report on MOZ/Open Site Explorer it only lists 10 domains linking in and, a number of important ones are not on that list. As an example, one of the leading UK national newspapers (Daily Telegraph) has a link on this page to the client www.vidavida.co.uk and yet this link/domain does not appear in the above reports. Is this an intentional nofollow type thing that the telegraph does and is there a way for me to check that on their site/page? http://fashion.telegraph.co.uk/news-features/TMG8655068/Holiday-travel-in-style.html Any comments most appreciated. Thanks,
Moz Pro | | MrFrisbee
C0 -
Error in Moz duplicate content reports
Hi - I've run the Moz campaign on a client's site. Moz is saying that there are duplicate content errors, and when I look at the errors it is showing that they are all to do with the non-www URLs having being duplicated in the www form of the URLs. However this is not the case - all the non-www URLs are all 301 redirected to the www URLs. Is this an error in the Moz tool? Has anybody experienced something similar?
Moz Pro | | rorynatkiel0 -
Getting my top keywords separated out in SEOmoz reports
I am using the standard functionality to produce weekly Moz reports. There does not seem to be a setting to show rankings of my most important keywords. It would be nice to have those high-level keywords on the first page. For example, I have 200 keywords in an account. I want the report to show on a page my 10 most important keywords. Is there a way to set up a Label to my keywords in order to product a report page just for those keywords?
Moz Pro | | clicktoshop0 -
Confused on setting up my domain in a campaign
I have a website that is www.example.com. When I first setup my campaign in SEOMoz, I missed the instructions that stated I need to set up my domain as it appeared in the browser. Thus, I set it up as example.com. When I realized my mistake, I setup a second campaign as www.example.com and am running them side by side to see the differences. My question has to do with the best setup. I have a blog that is a subdomain as blog.example.com. It appears that the www.example.com campaign is not crawling the blog pages but example.com is. Are there any downsides in setting up this campaign as example.com? I have a number of other sites with the same issue so would like to get this resolved before I setup the additional sites. Thanks for any assistance or insights into ways I may be "shooting myself in the foot" with the wrong domain structure in my campaign settings.
Moz Pro | | rfwood0 -
Help with URL parameters in the SEOmoz crawl diagnostics Error report
The crawl diagnostics error report is showing tons of duplicate page titles for my pages that have filtering parameters. These parameters are blocked inside Google and Bing webmaster tools. I do I block them within the SEOmoz crawl diagnostics report?
Moz Pro | | SunshineNYC0 -
How would you set up campaigns?
Hi Guy's, I have a question about campaigns, We run a eCommerce website with several categories with sub cats/ i.e. natural hair care Natural Shampoo Natural Conditioner Natural Treatments natural Skin Care etc etc My question is this... do you use campaigns to separate the sites sections up? or do you run one campaign, trying to get my head around the best way to target my sections for performance/keywords etc... Thanks Daniel
Moz Pro | | LushDuck0 -
Why is blocking the SEOmoz crawler considered a red "error?"
Why is blocking the SEOmoz crawler considered a red "error?" Please see attached image... Y3Vay.png
Moz Pro | | vkernel0