How can i block the below URLs
-
Google indexed plugins pages for my website. Please check below. How can stop them to be indexed on google.?
http://www.ayurjeewan.com/wp-content/plugins/LayerSlider/static/skins/glass/
http://www.ayurjeewan.com/wp-content/plugins/LayerSlider/static/skins/borderlesslight3d/
http://www.ayurjeewan.com/wp-content/plugins/LayerSlider/static/skins/defaultskin/
My robots.txt file is -
User-agent: * Disallow: /wp-admin/ -
robots.txt:
Disallow:
/wp-admin/
/wp-content/plugins/
Then, deindex them as mentioned in the first response.
-
It's also worth remembering that blocking a URL in a robots.txt file does not automatically mean that the URL will be deindexed. The robots.txt file will prevent robots, such as the Googlebot, from seeing or accessing the file and that's all. Now obviously, if a robot repeatedly tries to access a file and gets denied it will eventually stop trying to, which is what leads to the file being deindexed (the same principle applies to 404 and 410 errors).
Therefore, if you want a quicker and more definite deindexing solution, you should use the explicit noindex robot command, as recommended above. This will tell any visiting robot to not index it straight away, which will reduce the number of revisits and drop the page from the index faster.
-
Take a look over here - http://www.robotstxt.org/robotstxt.html and we can't forget the Moz version - http://moz.com/learn/seo/robotstxt
Alternatively you can just add meta noindex on the pages with with the added bonus of letting link juice flow better as well as being a bit more stern to robots ( recommend the noindex tag!)
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Can access my site using www
Hello, when I try to access my website using www i would like it to redirect to non www but instead it shows a sal error message.
On-Page Optimization | | Voopoo2 -
301 and the base URL
Hi, please bear with me as I'm pretty new to all this! I've my base URL but obviously want to add keywords to it for seo purposes. Should I redirect from the base URL to the URL with the keywords appended? So my landing page goes from say www.moz.com to www.moz.com/keywords-here. If I do that, should I replicate all the meta data (descriptions etc) on the original landing page? Or does it not matter? Thanks, Nick
On-Page Optimization | | nickwoodward0 -
Canonical URL Tag Usage
I have a large website, almost 1500 pages that each market different keywords for the trucking logistics industry. I don't really understand the new Canonical URL Tag USAGE. They say to use it so the page is not a duplicate but the page that MOZ is call for to have the tag isn't a duplicate. It promotes 1 keyword that no other page directly promotes. Here is the page address, now what tag would I put up in the HEAD so google don't treat it as a duplicate page. http://www.freightetc.com/c/heavyhaul/heavyhaul.php 1. Number 1 the actual page address because I want it treated like its own page or do I have to use #2 below? 2. I don't know why I would use #2 as I want it to be its own page, and get credit and listed and ranked as its own page. Can anyone clarify this stuff to me as I guess i am just new to this whole tag usage.
On-Page Optimization | | dwebb0070 -
Link flow for multiple links to same URL
Hi there,
On-Page Optimization | | doctecs
my question is as follows: How does Google handle link flow if two links in a given page point to the same URL? (do they flow link individually or not?) This seems to be a newbie question, but actually it seems that there is little evidence and even also little consensus in the SEO community about this detail. Answers should include source Information about the current state of art at Google is preferable The question is not about anchor text, general best practises for linking, "PageRank is dead" etc. We do know that the "historical" PageRank was implemented (a long time ago) without special handling for multiple links, as e.g. last stated by Matt Cutts in this video: http://searchengineland.com/googles-matt-cutts-one-page-two-links-page-counted-first-link-192718 On the other hand, many people from the SEO community say that only the first link counts. But so far I could not find any data to back this up, which is quite surprising.0 -
Best URL structure for my page
Hey everyone, I am wondering what is the best URL structure for my activity booking page in Bali. http://www.thingstodoinbali.com/bali/sports/water-sports/rafting/ or http://www.thingstodoinbali.com/bali/sports/water-sports/rafting-bali/ or would you recommend something else? Most people would google for "activity bali" in this case "rafting bali". Please advice. Thanks for your time and help in advance!
On-Page Optimization | | c.eiermann0 -
URL advice
Hi & thanks for looking, I'm not sure if I've adopted the best SEO URL structure for my site, www.vintageheirloom.com For instance, www.vintageheirloom.com/product-category/authentic-designer-vintage-bags/ Works great for the top level category 'All bags', as I'm trying to keyword authentic designer vintage bags. However the sub categories for instance 'Clutch bags' appears as, www.vintageheirloom.com/product-category/authentic-designer-vintage-bags/vintage-clutch-bags/. As you can see at the moment this URL contains duplicate terms vintage & bags. I'm guessing that duplicate keywords in a url isn't too smart, but should amend with Option 1, 2, 3 or something completely different? Option 1 - keep the top level category url the same, change the subcategory: www.vintageheirloom.com/product-category/authentic-designer-vintage-bags/clutch/ Option 2 - amend the top level category: www.vintageheirloom.com/product-category/authentic-designer/vintage-clutch-bags/ Option 3 - amend the top level category as this: www.vintageheirloom.com/product-category/bags/authentic-designer-vintage-clutch/ By the way I'm using WordPress with Woocommerce. I've asked but it's not possible with some technical issues to remove the /product-category/ section. But each product is for example just: www.vintageheirloom.com/shop/vintage-coach-yellow-duffel-sac-bag/ .... sweet. Thanks again !!
On-Page Optimization | | well-its-1-louder0 -
What are the benefits of the URL meta tag?
We have too many meta tags and want to get rid of all the outdated ones. However, we don't want to eliminate valuable meta tags by mistake. So, before we say goodbye to the URL meta tag, we want to make sure we understand the pros and cons, if any. By the way, we are not referring to canonical URL tags, just URL as in:
On-Page Optimization | | GRIP-SEO0 -
Can embedding videos from other sites help SEO?
I work for an eCommerce site and many of the vendors we get our products from have videos our their site that explain how the products work and the different features of them, ect. We really have done nothing with video marketing, and would like to create some of our own unique videos, but this will be impossible for the vast majority of the products on our site. We know from a content standpoint embedding videos from our vendors to our product pages will be beneficial, but at the same time it shows our competitors and other site visitors where we are getting our products from which we would rather not do. How will this effect us from an SEO standpoint though? Will it help us to have new content, even though it is embedded from another site? Or will it hurt us because it is not unique? Or will it have no effect at all? Thanks for you help!
On-Page Optimization | | ClaytonKendall0
