Does Rogerbot recognize rel="alternate" hreflang="x"?
-
Rogerbot just completed its first crawl and is reporting all kinds of duplicate content - both page content and meta title/description.
The pages it is calling duplicate are used with rel="alternate" hreflang="x", but are still being labeled as dupes.
The title and descriptions are usually exactly the same, so I am working on getting at least those translated into different languages.
I think its getting tripped up because the product page its crawling are only in English, but the chrome of the site is in the translated languages. The URLs look like so:
Original: site.com/product
Detected duplicates: site.com/fr/product, site.com/de/product, site.com/zh-hans/product
-
Hey there,
Rogerbot doesn't look for rel alts. The bot will follow meta robots, rel canonical (more used way to controlling duplicate content) and 301 redirects. Sorry about the confusion.
Best,
Nick
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Raven Reporting Alternative
Does anyone know of a tool that has similar capacity to Raven's Reports? I'm doing some white labeling of reports for another Agency that is used to sending their clients a monthly Raven report that is basicly pulling out all the Google Analytic Data of top referrers, keywords, landing pages ect and putting them into a nice format. I've used the Moz reporting and at times just combined it with a pdf of the actual GA equivalent. That's always seemed a bit clunky but I don't want to have to take on a Raven subscription just for that. Anyone have any good alternatives or know if when the new dashboard launches if it will also up its game when it comes to the generated reports?
Moz Pro | | BCutrer0 -
Allow only Rogerbot, not googlebot nor undesired access
I'm in the middle of site development and wanted to start crawling my site with Rogerbot, but avoid googlebot or similar to crawl it. Actually mi site is protected with login (basic Joomla offline site, user and password required) so I thought that a good solution would be to remove that limitation and use .htaccess to protect with password for all users, except Rogerbot. Reading here and there, it seems that practice is not very recommended as it could lead to security holes - any other user could see allowed agents and emulate them. Ok, maybe it's necessary to be a hacker/cracker to get that info - or experienced developer - but was not able to get a clear information how to proceed in a secure way. The other solution was to continue using Joomla's access limitation for all, again, except Rogerbot. Still not sure how possible would that be. Mostly, my question is, how do you work on your site before wanting to be indexed from Google or similar, independently if you use or not some CMS? Is there some other way to perform it?
Moz Pro | | MilosMilcom
I would love to have my site ready and crawled before launching it and avoid fixing issues afterwards... Thanks in advance.0 -
Rel=canonical "redirects" to double links
Our devs have set up rel=canonical on our website. First they used relative links href="/dir1/dir2/dir3" for the page http://www.mysite.com/dir1/dir2/dir3/?detail1=1?detail2=2 meaning that it will redirect to http://www.mysite.com/dir1/dir2/dir3, but no luck, the MOZ dashboard showed the tag value to be http://www.mysite.com/dir1/dir2/dir3/dir1/dir2/dir3, then we have decided to rewrite the code, and now the canonical to http://wwwmysite.com/dir1/dir2/dir3/?detail1=1?detail2=2 looks like href="http://www.mysite.com/dir1/dir2/dir3/" but the tag on MOZ looks like http://www.mysite.com/dir1/dir2/dir3http://www.mysite.com/dir1/dir2/dir3. So what is the problem? I really got a problem or MOZ does? The code on website looks exactly like href="http://www.aaa.com/en/bbb/ccc/vvv/nnn/" rel="canonical" /> for the page http://www.aaa.com/en/bbb/ccc/vvv/nnn/
Moz Pro | | apartmentGin0 -
After I make corrections of my crawl diagnostics report, how can I tell is those corrections "took". Is there a way to immediatly refresh that report. Will it eventually refresh?'
I have made corrections to the crawl diagnostics report. Can I refresh this report? I would like to see if my corrections were correct. Thanks for your anticipated answer!
Moz Pro | | Bob550 -
In Site Explorer My Blog.URL.com Shows "No Data Available for this URL"
Why when I use http://www.opensiteexplorer.org and I'm researching our Blog.URL.com's does the tool say "No Data Available for this URL"? Example: http://www.opensiteexplorer.org/links?site=blog.centurypayments.com
Moz Pro | | cfield_splashmedia.com0 -
Notice rel canonical
Hi, Why does my sites get the crawler notice for rel canonical when using the PRO account crawlers?? The canonical is there and it works, and to me it looks just like any other canonical link, the canonical is only at some links but not everyone, why is that?
Moz Pro | | careeron0 -
RogerBot does not respect some rules??
Hello; Every week when I see my stats I notice that RogerBot has crawled 10000 form my website, even pages with a no index or not allowed in the robots.txt. Is it possible to avoid him from crawling the these pages? They are form pages in my site, with are not indexed by google, they have a noindex and they are not allowed for crawling in the robots.txt. Thanks everyone for your help!!!
Moz Pro | | jgomes0 -
Rel Canonical issues for two urls sharing same IP address
Our client built a wordpress site on url A, then opted for a better url B. Rather than moving all the wordpress files/website over to the new url B, they just contacted GoDaddy, who hosted BOTH urls under the same IP address. When I do a term target on url B, I'm flagged for rel canonical use. I can only get a B grade for each keyword. (I've also tried using url A, but I get the same flag and B grade results). I'm not sure if this set-up will thwart our seo efforts for the site, because only the homepage comes up when you type in url B anyway. Every subsequent page displays the original url A. Somewhere, wordpress is also adding a rel canonical link on the homepage source to url A, too, which we can't seem to edit. So, question is: is it ok to leave this set up as is with both urls hosted on the same IP address, or should we move the whole site over to the desired url B? Thanks much!
Moz Pro | | GravitateOnline0