How can i block the below URLs
-
Google indexed plugins pages for my website. Please check below. How can stop them to be indexed on google.?
http://www.ayurjeewan.com/wp-content/plugins/LayerSlider/static/skins/glass/
http://www.ayurjeewan.com/wp-content/plugins/LayerSlider/static/skins/borderlesslight3d/
http://www.ayurjeewan.com/wp-content/plugins/LayerSlider/static/skins/defaultskin/
My robots.txt file is -
User-agent: * Disallow: /wp-admin/ -
robots.txt:
Disallow:
/wp-admin/
/wp-content/plugins/
Then, deindex them as mentioned in the first response.
-
It's also worth remembering that blocking a URL in a robots.txt file does not automatically mean that the URL will be deindexed. The robots.txt file will prevent robots, such as the Googlebot, from seeing or accessing the file and that's all. Now obviously, if a robot repeatedly tries to access a file and gets denied it will eventually stop trying to, which is what leads to the file being deindexed (the same principle applies to 404 and 410 errors).
Therefore, if you want a quicker and more definite deindexing solution, you should use the explicit noindex robot command, as recommended above. This will tell any visiting robot to not index it straight away, which will reduce the number of revisits and drop the page from the index faster.
-
Take a look over here - http://www.robotstxt.org/robotstxt.html and we can't forget the Moz version - http://moz.com/learn/seo/robotstxt
Alternatively you can just add meta noindex on the pages with with the added bonus of letting link juice flow better as well as being a bit more stern to robots ( recommend the noindex tag!)
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Anyone know the benefit of URLs being in the target language?
For example, a page targeting the spanish language, does it make a difference if the URL is site.com/i-don't-speak-spanish vs site.com/no-habla-espanol
On-Page Optimization | | LennyO0 -
301 and the base URL
Hi, please bear with me as I'm pretty new to all this! I've my base URL but obviously want to add keywords to it for seo purposes. Should I redirect from the base URL to the URL with the keywords appended? So my landing page goes from say www.moz.com to www.moz.com/keywords-here. If I do that, should I replicate all the meta data (descriptions etc) on the original landing page? Or does it not matter? Thanks, Nick
On-Page Optimization | | nickwoodward0 -
To update or not to update news URLs ?
We manage a huge daily news website in my small country - keeping this a bit mysterious in case competitors are reading 🙂 Our URL structure is www.companyname.com/news/categoryofnews/title-of-article?id=articleid In this hyperreactive news world, title of articles change frequently (may be ten times a day for the main stories). The question we debate is : should we reflect the modification of the title in the URL or not ? Example : "Trump says he wants to ban search engines" would have URL http://www.companyname.com/news/entertainment/Trump-says-he-wants-to-ban-search-engines?id=12345678 Later in the day the title becomes "Trump denies he suggested banning search engines". Should the URL be modified to http://www.companyname.com/news/entertainment/Trump-denies-he-suggested-banning-search-engines?id=12345678 (option A) or not (option B) ? In Google News it makes no difference because of the sitemap, but in Google organic things are different. At present (option B in place), Google apparently doesn't see that the article has been updated, and shows the initial timestamp which is visually (and presumably SEOwise) not good : our new news looks like old news. Modifiying the URL would solve that issue, but could, may be, create another one : the new URL, being considered a new article, would lose, the acquired weight of the previous one in terms of referrals, social trafic and so on. Or not ? What do you think is the best option ? Thanks for your expertise, Yves
On-Page Optimization | | yves678901 -
Home page keyword in url
I have been looking into SEO for a few weeks now trying to perfect a homepage. Going through various sources on MOZ, and other examples out there on the internet, I keep seeing that you should have your keyword in the URL of the page. The homepage is the page most people want to rank the highest in google searches, however, you cannot put the keyword in the URL as most home page URLs are simply /. Should I actually make the home like this: www.example.com/key-word-example? I would imagine this would not be the normal for many users and would seem like it's not the home page.
On-Page Optimization | | Matthew_smart0 -
301 vs. keeping identical URL
Hey everybody! I have a question pertaining to our redesign. The situation is as follows: /drug-rehab/alcohol-withdrawal-los-angeles gets a decent amount of views on out website, and we would like it to be on our redesigned site. I was curious what impact, if any, I would see given the two scenarios below. 301 to /alcohol-withdrawal make the new page /drug-rehab/alcohol-withdrawal-los-angeles as well The second situation is that there are a serious of other pages which don't seem to be of drastic benefit, which I don't feel NEED to be on the website. For example: /post-acute-alcohol-withdrawal-treatment/drug-los-angeles /rehabs-resources/drug-abuse/sub-acute-alcohol-withdrawal etc It appears to me that the content on these pages is rather similar, and I feel like they don't really say anything special. Can I 301 them to the new page? Should I let them die in the black hat inferno they were made in? Any thoughts are greatly appreciated! Thanks guys!
On-Page Optimization | | HashtagHustler0 -
Can someone take a quick look at my blog?
hey guys, just joined seomoz. was wondering is someone could take a quick look at my blog and see if it is fine to start hitting SEO hard? my top bar title has a few of my targeted keywords, is that a good idea or? http://www.onestopmuscle.co.uk
On-Page Optimization | | FLEAR0 -
Adding keywords to URL's
I understand the importance of having the keyword in the URL (at least now I do). When I created my site (www.enchantingquotes.com), I was completely ignorant about SEO. So....question is...how do I go about adding keywords to already done pages? Do I create a new section and then redirect - or do I have to basically recreate pages? Thx much 🙂
On-Page Optimization | | enchantedgirlz0 -
URL Extensions (with or without??!!)
Hello, SEOers~ Today I have a question about URL extensions. Which one is more search engine friendly between URL with extensions and without extensions? e.g. URL with extension : www.example.com/tv/lcd.jsp URL without extension : www.example.com/tv/lcd I heard that URL without extensions is in trend considering user experience. User experience is also important but I would like to know from SEO perspective. Please people~ Help me out with this~! Thanks.
On-Page Optimization | | Artience0