How to set the crawler or reports to ignore
-
I have a mobile version of a site with a URL string that disables the mobile view on smartphones (view full site), string is like this example.com/page-name.html?mobile=off
I need the seomoz pro reports or crawler to ignore it because the crawler visits both versions of the site then reports them as duplicate content. Is there a setting page I haven't visited yet that will set this?
-
Thanks for the confirmation. I hadn't wanted to put that in the robots file because if inquisitive, non-robot traffic if you get my meaning, but in it goes; can't stand seeing errors.
-
Got it, thanks for jumping in so quick.
-
Hey Str8,
Donford is correct. You would need to specifically block our crawler from the mobile pages in your robots.txt file. Unfortunately, we don't currently have a way to disregard specific pages or errors in the web app, but we are looking to add that function sometime in the future.
I hope this helps. Please let me know if you have any other questions.
-
Hi Str8
**user-agent: rogerbot disallow: /*mobile=**
I think may do the trick... somebody may need to confirm this.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
New to Moz and wanted a bit of help with my report
Hi, I have used the MOZ report to analyse one of my friends sites and I wanted to query a few warnings it highlighted and I just wanted people's thought on how important they thought these were: The first is dupliate descriptions/titles. This is mainly down the e-commerce pages. Fist duplicate content:
Moz Pro | | dannylancs
On some pages the description is identical and all that is different is the title and picture, is this an issue? Duplicate pages:
Due to the way the website folder structure/catergories has been created some pages are identical but because the product comes under 2 cetergories there is 2 seperate pages, should we use the canonical on one of the pages? Also regarding the canonical tag, they have put link rel="canonical" on every page and got it to point at itself, so not really being used in the way it is meant to be. Could something like this cause any harm? The final thing is internal linking back to the homepage. If for example the homepage is http://www.test.com, when linking back is it best to put the full URL over "index.html" even though they are the same page? Any help really appreciated Dan0 -
On Page Reports for Long-Tail Keywords?
First Q&A here, so take it easy on me. Hopefully this is not a dumb question. 99% of the on page reports I look at are for local keywords. I'm finding that some of the page reports get and F where they should be receiving A's. I noticed if I manage a long-tail keyword like "books in houston tx" that I get a grade based on the exact phrase as opposed to a combination of the keywords. So when I have text in the body saying "books in the Houston area" or "Houston Books" for example, instead of using my exact managed keyword, it will tell me that the keyword is not mentioned in the body anywhere. When it is, it's just not in the exact order... I'm trying to write my pages for users and I don't think the users want to hear "books in houston tx" several times. If I do a report on the same page for the keyword "books" I get an A. So this is where it gets a little in the grey area for me. I need a solid on page report for local long-tail search terms. Anyone have any advice? Is there a way I could be using this tool to better suit my needs?
Moz Pro | | AmericomMarketing0 -
Forward slash on URL on Duplicate Content Report
Hi I'm new to this whole Moz thing, so needing help from some kind people! I've just looked at my Duplicate Page Content report and there are loads of URLs in there which are the same but are just differentiated by adding / at the end of the URL, e.g. http://youngepilepsy.org.uk/news-and-events/events http://youngepilepsy.org.uk/news-and-events/events/ Is this be a canonical issue? I can't understand why though as these aren't at the root. However when we add inline text links within the page HTML, there are some URLs with / and some without, could that be the reason? Thanks for your help! Jackie
Moz Pro | | YoungEpilepsy1 -
Functionality of SEOmoz crawl page reports
I am trying to find a way to ask SEOmoz staff to answer this question because I think it is a functionality question so I checked SEOmoz pro resources. I also have had no responses in the Forum too it either. So here it is again. Thanks much for your consideration! Is it possible to configure the SEOMoz Rogerbot error-finding bot (that make the crawl diagnostic reports) to obey the instructions in the individual page headers and http://client.com/robots.txt file? For example, there is a page at http://truthbook.com/quotes/index.cfm month=5&day=14&year=2007 that has – in the header -
Moz Pro | | jimmyzig
<meta name="robots" content="noindex"> </meta name="robots" content="noindex"> This page is themed Quote of the Day page and is duplicated twice intentionally at http://truthbook.com/quotes/index.cfm?month=5&day=14&year=2004 and also at http://truthbook.com/quotes/index.cfm?month=5&day=14&year=2010 but they all have <meta name="robots" content="noindex"> in them. So Google should not see them as duplicates right. Google does not in Webmaster Tools.</meta name="robots" content="noindex"> So it should not be counted 3 times? But it seems to be? How do we gen a report of the actual pages shown in the report as dups so we can check? We do not believe Google sees it as a duplicate page but Roger appears too. Similarly, one can use http://truthbook.com/contemplative_prayer/ , here also the http://truthbook.com/robots.txt tells Google to stay clear. Yet we are showing thousands of dup. page content errors when Google Webmaster tools as shown only a few hundred configured as described. Anyone? Jim0 -
Why does Seomoz report other values then Google Webmaster?
Dear Seomoz Community I check my Seomoz ranking from time to and today I saw a big discrepancy between the reported domains linking to my site in the Site Explorer and Google Webmaster tools. Seomoz reports 19 linking domains
Moz Pro | | menelik
Google shows 150 domains linking to my site. Is there an explanation for this discrepancy? Thank for helping me out 😉0 -
How seomoz sets the country for google to monitor my rankings?
In some of my campaign settings I use specific country for search engine (e.g. "Google Slovakia"). Some times I check the rankings manually too. My process for this: 1. Open a new Chrome incognito window (be sure that that's the only open incognito window) 2. go to google.sk, type in the keyword 3. at the bottom of the page select advanced search, select language: slovak, resubmit the query That's it. With my process, the monitored keywords are not there were seomoz reports them. So the question: with what settings (domain, language anything else) seomoz is tracking the rankings in country specific search engines?
Moz Pro | | Brainsum0 -
Site Ranking Report
Hi guys, My site ranking report says that I've gone from being 1-20 for a variety of keywords in Google UK to not in the top 50. When I do a search myself I see that my site remains where it previously was (between 1-20). How reliable is the site ranking reporting on a weekly basis? Is it best to look at it monthly?
Moz Pro | | columbus0 -
How often are open site explorer reports updated?
I collect the information contained in open site explorer reports and csv backlinks audits on the 15th of every month. I noticed that the numbers are unchanged from 9/15-10/15. How often are the reports typically updated?
Moz Pro | | seagreen0