How can i block the below URLs
-
Google indexed plugins pages for my website. Please check below. How can stop them to be indexed on google.?
http://www.ayurjeewan.com/wp-content/plugins/LayerSlider/static/skins/glass/
http://www.ayurjeewan.com/wp-content/plugins/LayerSlider/static/skins/borderlesslight3d/
http://www.ayurjeewan.com/wp-content/plugins/LayerSlider/static/skins/defaultskin/
My robots.txt file is -
User-agent: * Disallow: /wp-admin/ -
robots.txt:
Disallow:
/wp-admin/
/wp-content/plugins/
Then, deindex them as mentioned in the first response.
-
It's also worth remembering that blocking a URL in a robots.txt file does not automatically mean that the URL will be deindexed. The robots.txt file will prevent robots, such as the Googlebot, from seeing or accessing the file and that's all. Now obviously, if a robot repeatedly tries to access a file and gets denied it will eventually stop trying to, which is what leads to the file being deindexed (the same principle applies to 404 and 410 errors).
Therefore, if you want a quicker and more definite deindexing solution, you should use the explicit noindex robot command, as recommended above. This will tell any visiting robot to not index it straight away, which will reduce the number of revisits and drop the page from the index faster.
-
Take a look over here - http://www.robotstxt.org/robotstxt.html and we can't forget the Moz version - http://moz.com/learn/seo/robotstxt
Alternatively you can just add meta noindex on the pages with with the added bonus of letting link juice flow better as well as being a bit more stern to robots ( recommend the noindex tag!)
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Canonical URL Tag Usage
I have a large website, almost 1500 pages that each market different keywords for the trucking logistics industry. I don't really understand the new Canonical URL Tag USAGE. They say to use it so the page is not a duplicate but the page that MOZ is call for to have the tag isn't a duplicate. It promotes 1 keyword that no other page directly promotes. Here is the page address, now what tag would I put up in the HEAD so google don't treat it as a duplicate page. http://www.freightetc.com/c/heavyhaul/heavyhaul.php 1. Number 1 the actual page address because I want it treated like its own page or do I have to use #2 below? 2. I don't know why I would use #2 as I want it to be its own page, and get credit and listed and ranked as its own page. Can anyone clarify this stuff to me as I guess i am just new to this whole tag usage.
On-Page Optimization | | dwebb0070 -
How can i define a ALT Tag for Youtube videos?
Hello, When i scan with SEMrush it shows all such pages do not have ALT Tags http://img.youtube.com/vi/iJ4pmmRSZxc/hqdefault.jpg What should the ALT be for them and how can i control as we have 1000's of Youtube videos displayed on various parts of our website Thank you
On-Page Optimization | | ktrinkqao120 -
How to block index of link and content
Hi, We have pages where articles are shown and in the sides we have small snippets of Articles which shows the title and close to 25 words and a image. When i search for something in Google the snippet image and content is shown and in Google when clicked it redirects to a page which is not meant to be shown for the keyword the visitor is querying Is there a way i can block all the links and content shown in the right and left side of the page so Google does not get confused with the page content thats not related to that page? thanks
On-Page Optimization | | AlexisWithers0 -
Title tag and URL Optimization
Hello guys, Should the URL reflect the structure of the title of a webpage? This is the old title with the Url: 20mm O/D Black Polypropylene LSZH Flexible Conduit 100m Coil /Product/20mm-o-d-black-polypropylene-lszh-conduit-100m-coil/1352 I changed the keyword position and it looks like this: 20mm Flexible Conduit | O/D Black Polypropylene LSZH | 100m I kept the same Url for now, should I change that too? Thanks
On-Page Optimization | | PremioOscar0 -
Optimize URL
Hello, My website have been running over five years. I have just reviewed and seen some URLs had not good. It is http://www.vietnamvisacorp.com/faqs/who-need-visa-to-vietnam---1.html, containing characters "---1". Should I remove unnecessary characters "---"?. Thanks for any advice!
On-Page Optimization | | JohnHuynh0 -
301 redirects / clean urls
hello everyone, we moved our site to a new platform that has url rewrite feature. we are building out a spreadsheet of the old aspx urls and the corresponding page on the new site. my dev tells me that its not important to redirect the old page to the new "clean url" page which sounds a bit odd to me ..example below. okay. so he says its the same difference to 301 redirect this url: http://72.3.181.97/catalog/DesignerDogBeds.aspx to this urls: https://www.k9electronics.com/product.php?productid=3474 instead of this clean url: http://www.k9electronics.com/designer-dog-beds/ can anyone give me any feedback on this? thanks!
On-Page Optimization | | k9byron0 -
Using magentos own url re-writes
We are changing ecommerce platforms. Is it best to use magentos own url re-writes to redirect every page of a site from its old url to it new one?
On-Page Optimization | | LadyApollo0 -
Can A Blog post impact negatively on Google rankings?
We used to rank fairly highly on Google for our preferred search term, but once we introduced a blog (or rather the blog excerps) onto our home page we dropped from page 1 to page 3 for our preferred search term. We have made the assumption this may be down to diluting of keywords and therefor the key phrase or message to the Search Engines was not as clear as it was before. Is this generally accepted as being the case ? If so - should each and every blog post excerpt start with the key seo search phrase to maintain the integrity of that search phrase ? or perhaps even better still is it best to have the blog on a dedicated page only with no excerpts on the home page ? We are using Wordpress and also using multiple categories within the blog - is it perhaps best to just have all blog postings on the one page and use TAGS more often? Thanksing you in advance.
On-Page Optimization | | kiswebsltd0
