Implications of Disallowing A LOT of Pages
-
Hey everyone,
I just started working on a website and there are A LOT of pages that should not be crawled - probably in the thousands. Are there any SEO risks of disallowing them all at once, or should I go through systematically and take a few dozen down at a time?
-
Perfect, that's my intent. Thanks so much for your help!! I really appreciate it.
-
Correct.
Also, (i´ve should have said this earlier) there are no effective way to avoid robots crawling the web. The thing here (and what im answering) is to not show (and/or remove if are already in the index) certain pages in google search result.
-
That would happen if I used noindex, correct?
-
Exactly.
Also, remember that disalowing by robots.txt will not remove them from Google´s index.
-
LOL now I'm worried about what you mean by "correctly." Do you mean just making sure that I'm not disallowing valuable pages that should be crawled?
-
Hello Rachel,
If you do it correctly, this is not disallowing useful pages, there is no risk. Go ahead.
Best luck.
GR.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How do I prevent duplicate page title errors from being generated by my multiple shop pages?
Our e-commerce shop has numerous pages within the main shop page. Users navigate through the shop via typical pagination. So while there may be 6 pages of products it's all still under the main shop page. Moz keeps flagging my shop pages as having duplicate titles (ie shop page 2). But they're all the same page. Users aren't loading unique pages each time they go to the next page of products and they aren't pages I can edit. I'm not sure how to prevent this issue from popping up on my reports.
Technical SEO | | NiteSkirm0 -
Canonicalisation and Dynamic Pages
We have an e-commerce single page app hosted at https://www.whichledlight.com and part of this site is our search results page (http://www.whichledlight.com/t/gu10-led-bulbs?fitting_eq=GU10). To narrow down products on the results we make heavy use of query parameters. From an SEO perspective we are telling GoogleBot to not index pages that include these query parameters to prevent duplicate content issues and to not index pages where the combination of query parameters has resulted in no results being returned. The only exception to this is the page parameter. We are posting here to check our homework so to speak. Does the above sound sensible? Although we have told GoogleBot to not index these pages, Moz will still crawl them (to the best of my knowledge), so we will continue to see crawl errors within our Moz reports where in fact these issues don't exist. Is this true? Is there anyway to make Moz ignore pages with certain query parameters? Any other suggestions to improve the SEO of our results pages is most appreciated. Thanks
Technical SEO | | TrueluxGroup0 -
Issue with Cached pages
I have a client who has a three domains:
Technical SEO | | paulbaguley
budgetkits.co.uk
prosocceruk.co.uk
cheapfootballkits.co.uk Budget Kits is not active but Pro Soccer and Cheap Football Kits are. The issue is when you do site:budgetkits.co.uk on Google it brings back results. If you click on the link it goes to page saying website doesn't exist which is correct but if you click on cached it shows you a page from prosocceruk.co.uk or cheapfootballkits.co.uk. The cached pages are very recent by a couple of days ago to a week. The first result brings up www.budgetkits.co.uk/rainwear but the cached page is www.prosocceruk.co.uk/rainwear The third result brings up www.budgetkits.co.uk/kids-football-kits but the cached page is http://www.cheapfootballkits.co.uk The history of this issue is that budgetkits.co.uk was its own website 7 years ago and then it used to point at prosocceruk.co.uk after that but it no longer does for about two months. All files have been deleted from budgetkits.co.uk so it is just a domain. Any help with this would be very much appreciated as I have not seen this kind of issue before.0 -
SEO value of InDesign pages?
Hi there, my company is exploring creating an online magazine built with Adobe's InDesign toolset. If we proceeded with this, could we make these pages "as spiderable" as normal html/css webpages? Or are we limited to them being less spiderable, or not at all spiderable?
Technical SEO | | TheaterMania1 -
New Page Showing Up On My Reports w/o Page Title, Words, etc - However, I didn't create it
I have a WordPress site and I was doing a crawl for errors and it is now showing up as of today that this page : https://thinkbiglearnsmart.com/event-registration/?event_id=551&name_of_event=HTML5 CSS3 is new and has no page title, words, etc. I am not even sure where this page or URL came from. I was messing with the robots.txt file to allow some /category/ posts that were being hidden, but I didn't re-allow anything with the above appendages. I just want to make sure that I didn't screw something up that is now going to impact my rankings - this was just a really odd message to come up as I didn't create this page recently - and that shouldnt even be a page accessible to the public. When I edit the page - it is using an Event Espresso (WordPress plugin) shortcode - and I don't want to noindex this page as it is all of my events. Sorry this post is confusing, any help or insight would be appreciated! I am also interested in hiring someone for some hourly consulting work on SEO type issues if anyone has any references. Thank you!
Technical SEO | | webbmason0 -
Two Domains for the Same Page
We are creating a website for a client that will have hundreds of geographically driven landing pages. These pages will all have a similar domain structure. For example www.domain.com/georgia-atlanta-fastfood-121 We want the domain to be SEO friendly, however it also needs to be print friendly for a business card. (ex www.domain.com/121) The client has requested that we have two domains for each page. One for the Search Engines and then another shorter one for print/advertising purposes. If we do that will search engines the site for duplicate content? I really appreciate any recommendations. Thanks! Anna
Technical SEO | | TracSoft0 -
Are there negative SEO implications to pages without any images?
Hi Mozzers, Do you think there are any negative effects of having no images on a page but several hundreds words of text? (There is a logo image and call to action buttons). Thanks!
Technical SEO | | Charlessipe0 -
Page load speed?
Hi everyone, hope you've all had a good new years celebration. Just a quick one on page load speed? how important do you think it is to organic rankings, especially on competitive terms? Any tips on improving it? do i need to improve just the landing page im trying to rank or every page as its looked at as a qualtiy score as a whole? Kind Regards,
Technical SEO | | pauledwards0