Welcome to the Q&A Forum

Browse the forum for helpful insights and fresh discussions about all things SEO.

Your browser does not seem to support JavaScript. As a result, your viewing experience will be diminished, and you have been placed in read-only mode.

Please download a browser that supports JavaScript, or enable it if it's disabled (i.e. NoScript).

Will moz crawl pages blocked by robots.txt and nofollow links?

Moz Pro

650

WizardOfMoz last edited by

i have over 2,000 temporary redirects in my campaign report

redirects are mostly events like being redirected to a login page before showing the actual data

im thinking of adding nofollow on the link so moz wont crawl the redirection to reduce the notification

will this solve my problem?
1 Reply Last reply
Reply Quote 0
Ryan_Watson last edited by

Hey Oscar,

Yes, Moz will not crawl nofollowed links, so this should solve your problem.

Cheers,

Ryan Watson
Business Development Associate | Moz
1 Reply Last reply
Reply Quote 0

Got a burning SEO question?

Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.

Start my free trial

Browse Questions

View

From

Sorted by

With category

Explore more categories

Related Questions

Filter Pages

Howdy Moz Forum!! I have a headache of a job over here in the UK and I'd welcome any advice! - It's sunny today, only 1 of 5 days in a year and i'm stuck on this! I have a client that currently has 22,000 pages indexed to Google with almost 4000 showing as duplicate content. The site has a "jobs" and "candidates" list. This can cause all sorts of variations such as job title, language, location etc. The filter pages all seem to be indexed. Plus the static pages are indexed. For example if there were 100 jobs at Moz being advertised, it is displaying the jobs on the following URL structure - /moz
/moz/moz-jobs
/moz/moz-jobs/page/2
/moz/moz-jobs/page/3
/moz/moz-jobs/page/4
/moz/moz-jobs/page/5 ETC ETC Imagine this with some going up to page/250 I have checked GA data and can see that although there are tons of pages indexed this way, non of them past the "/moz/moz-jobs" URL get any sort of organic traffic. So, my first question! - Should I use rel-canonical tags on all the /page/2 & /page/3 etc results and point them all at the /moz/moz-jobs parent page?? The reason for this is these pages have the same title and content and fall very close to "duplicate" content even though it does pull in different jobs... I hope i'm making sense? There is also a lot of pages indexed in a way such as- https://www.examplesite.co.uk/moz-jobs/search/page/9/?candidate_search_type=seo-consulant&candidate_search_language=blank-language These are filter pages... and as far as I'm concerned shouldn't really be indexed? Second question! - Should I "no follow" everything after /page in this instance? To keep things tidy? I don't want all the variations indexed! Any help or general thoughts would be much appreciated! Thanks.
Moz Pro | | Slumberjac

0
Is it normal for Moz to report on nofollow pages in crawl diagnostics?

I have a dev version of my website, for example, devwww.website.com. The htaccess page has a noindex and nofollow request, but I got crawl issues reported from these pages in my Moz report. Does this mean that I don't have the development site hidden from search like I thought I did?
Moz Pro | | houstonbrooke

0
I am trying to find inbound links for one of my site urls. My question is does SEOMoz able to track all internal links as the Open Site Explorer shows 0 internal links?

It shows 0 internal links when I am pretty sure we have multiple internal links.Should we use absolute urls or relative urls for internal links?
Moz Pro | | SulekhaUSLLC

0
Links & page authority crawl

I see the links and page authority have not been updated in over a month... does anyone know how often it gets updated?
Moz Pro | | nazmiyal

0
Does the SEOMoz weekly crawl that highlights no meta description tag, take into account if there is a meta robots noindex,follow tag on the pages it indicates the missing meta descriptions?

The weekly crawl website report is telling me that there are pages that have missing meta description tags, yet I've implemented meta robots tags to 'noindex, follow' those pages which are visible in those page source files. As far as Google Is concerned, surely this then won't be a problem since it is being instructed NOT to consider these specific pages for indexing. I am assuming that the weekly SEOmoz website crawl is simply throwing the missing meta description crawl findings into its report without itself observing that the particluar URL references contain the meta robots 'noindex,follow' tag ???? Appreciate if you can clairfy if this is the case. It would help me understand that (at least in terms of my efforts towards Google) your own crawl doesn't observe the meta robots tag instruction, hence the resultant report's flagging the discrepancy.
Moz Pro | | callassist

0
How can I reduce the number of links on a page and keep the site easy to navigate?

The SEOmoz Site Crawl indicates that we have too many on page links on over 9,970 pages. This is an ecommerce site with a large number of categories. I have a couple of questions regarding this issue: How important is the "too many on page links" factor to SEO? What are some methods of reducing the number of links when there are a large number of categories? We have main categories with dropdown menus currently and have found that they are used to browse and shop the store.
Moz Pro | | afmaury

1
Link Building

Hi is there any free ways of link building where you do not use exchange link sites. Also are there any free tools and paid tools that you can use to increase your links to your site. I have a couple of new sites and i am trying to bring more traffic to them through link building and other methods and trying to find the best way of doing this. I have heard of people talk about free tools and paid tools but not sure which ones they are. any help would be great
Moz Pro | | ClaireH-184886

0
Which would you chose? Link on PA56 with 88 OBL's and 80 IBL's or a link on a PA75 with 225 OBL's & 40 IBL (Same Domain)

Which would you chose? Link on PA56 with 88 OBL's and 80 IBL's or a link on a PA75 with 225 OBL's & 40 IBL (Same Domain) Pretty self explanatory. I want to know what metrics SEOMozzers rely on most. If you are not an expert at evaluating links for large scale development please don't muddy the waters on this question with a thin and vague answer.
Moz Pro | | DavidWolf58

0