How can i block the below URLs
-
Google indexed plugins pages for my website. Please check below. How can stop them to be indexed on google.?
http://www.ayurjeewan.com/wp-content/plugins/LayerSlider/static/skins/glass/
http://www.ayurjeewan.com/wp-content/plugins/LayerSlider/static/skins/borderlesslight3d/
http://www.ayurjeewan.com/wp-content/plugins/LayerSlider/static/skins/defaultskin/
My robots.txt file is -
User-agent: * Disallow: /wp-admin/ -
robots.txt:
Disallow:
/wp-admin/
/wp-content/plugins/
Then, deindex them as mentioned in the first response.
-
It's also worth remembering that blocking a URL in a robots.txt file does not automatically mean that the URL will be deindexed. The robots.txt file will prevent robots, such as the Googlebot, from seeing or accessing the file and that's all. Now obviously, if a robot repeatedly tries to access a file and gets denied it will eventually stop trying to, which is what leads to the file being deindexed (the same principle applies to 404 and 410 errors).
Therefore, if you want a quicker and more definite deindexing solution, you should use the explicit noindex robot command, as recommended above. This will tell any visiting robot to not index it straight away, which will reduce the number of revisits and drop the page from the index faster.
-
Take a look over here - http://www.robotstxt.org/robotstxt.html and we can't forget the Moz version - http://moz.com/learn/seo/robotstxt
Alternatively you can just add meta noindex on the pages with with the added bonus of letting link juice flow better as well as being a bit more stern to robots ( recommend the noindex tag!)
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How can I check spam score and non indexed links
Hi I am working on a project in which I have a local services website <a target="_self">www.thepcrrg.com</a> I want to be sure its links are getting indexed or not. Please help me.
On-Page Optimization | | Jioiebytt2 -
Weird 404 Error URL's - Where to redirect?
For some reason, I end up with a bunch of weird 404 error URLs like these. /KRVWZ/SVfSZ/ /wp-content/upl /MKWTZ/RKYQZ/ /wp-content/upload What should I do with them? Should I redirect them or is there a better way to do something with them. I do use Yoast Premium if that makes a difference. Thanks in advance.
On-Page Optimization | | photoseo10 -
How defined does the anchor text need to be for a domain url?
Ok, I'm looking to clean up my domain with irrelevant Anchor text linking to specific URL's. Whether, it's my root domain or a particular product URL. I'm finding a wide variety of terms (anchor text) pointing to my homepage, category and finally my product pages. Example, the Anchor text "Paragon Print Systems" is pointing to my homepage "barcodefactory.com" does this hurt my homepage enough to either have the link/anchor text removed from linking to my homepage? thanks much!, Warren
On-Page Optimization | | Warren.W0 -
URL Structure Category Pages -Current Moz Friday-
Hello,
On-Page Optimization | | _Heiko_
regarding #15 of the last moz friday I have a question: http://moz.com/blog/15-seo-best-practices-for-structuring-urls What would you prefer if the lenght of the URL will be still under 60 characters and you have an example like this: Let's call it a specific page in a category. As I like the old shoe examples: You have a page about red shoes in your shoe category. Which URL would you prefer: a) www.mydomain.com/shoes/red-shoe b) www.mydomain.com/shoes/red Personally I would prefer a) or would you already consider this as spammy? My real example is not that trivial like the shoe example and the categories will be in plural and the specific pages always in singular (like in the example shoes vs shoe). c) would be to put it independently from the side structure on www.mydomain.com/red-shoe - but personally I have the experience that a) or b) will help the rankings of the category page if you have the specific pages in the same subfolder. What's your opinion on this?1 -
How do you handle URLs with slashes?
I asked this question before, but with a different scenario. I upgraded my plan to a more advanced cart and all of my URLs changed about 1.5 years ago. I knew nothing about redirects and such, so none of that was done. Basically, let's say my site was: http://www.abc.com, but when people actually visit my site, they are directed to https://www.abc.com/. I have asked my host about redirecting and she that it is not possible. In the past, the link shared has been just www.abc.com . Will this hurt my ranking? My second question is ...let's say I have a link http://www.abc.com/blog , but now, the link is http://www.abc.com/blog/ . Will I be affected, since all my old links omit the slash?
On-Page Optimization | | tiffany11030 -
Can we listed URL on Website sitemap page which are blocked by Robots.txt
Hi, I need your help here. I have a website, and few pages are created for country specific. (www.example.com/uk). I have blocked many country specific pages from Robots.txt file. It is advisable to listed those urls (blocked by robots.txt) on my website sitemap. (html sitemap page) I really appreciate your help. Thanks, Nilay
On-Page Optimization | | Internet-Marketing-Profs0 -
Can bad text URLs hurt pages?
If you have some pages that contain plain text URLs (not anchored links) that used to be good URLs, but are now bad, either because the website shut down or because it has been acquired by someone else and is now parked (or worse) - are those URLs enough to cause quality problems? For example: This information was brought to you by Waymaker http://www.waymaker.net These aren't the only ones. And yes, I know I should fix them, but there are probably 10,000 pages like it. I will fix them, but its not something I can do in a few minutes. (this one is easy to fix programmatically, but others are a lot more complex) So my question is: do you have actual experience that these are bad enough to cause ranking problems (making them low quality)
On-Page Optimization | | loopyal0 -
Maximum length of a URL for good SEO?
Hi there, We have a content database as part of our site and I noticed that the way the database is loaded with new content, it means that the URL’s for these pages are really long, around 100 characters or sometimes more http://www.xxxyy.org/knowledge-base/documents/word1-word2-word3-word4-word5-word6-word7-word8 Is there a suggted maximum character length for a URL? Kind of like for title tag where I max out at 69… Should I truncate the URL’s or at least reduce the numbers of words in them to something less spammy? Does that make a difference? Thanks!
On-Page Optimization | | inhouseninja0