How can i block the below URLs
-
Google indexed plugins pages for my website. Please check below. How can stop them to be indexed on google.?
http://www.ayurjeewan.com/wp-content/plugins/LayerSlider/static/skins/glass/
http://www.ayurjeewan.com/wp-content/plugins/LayerSlider/static/skins/borderlesslight3d/
http://www.ayurjeewan.com/wp-content/plugins/LayerSlider/static/skins/defaultskin/
My robots.txt file is -
User-agent: * Disallow: /wp-admin/
-
robots.txt:
Disallow:
/wp-admin/
/wp-content/plugins/
Then, deindex them as mentioned in the first response.
-
It's also worth remembering that blocking a URL in a robots.txt file does not automatically mean that the URL will be deindexed. The robots.txt file will prevent robots, such as the Googlebot, from seeing or accessing the file and that's all. Now obviously, if a robot repeatedly tries to access a file and gets denied it will eventually stop trying to, which is what leads to the file being deindexed (the same principle applies to 404 and 410 errors).
Therefore, if you want a quicker and more definite deindexing solution, you should use the explicit noindex robot command, as recommended above. This will tell any visiting robot to not index it straight away, which will reduce the number of revisits and drop the page from the index faster.
-
Take a look over here - http://www.robotstxt.org/robotstxt.html and we can't forget the Moz version - http://moz.com/learn/seo/robotstxt
Alternatively you can just add meta noindex on the pages with with the added bonus of letting link juice flow better as well as being a bit more stern to robots ( recommend the noindex tag!)
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How can i get a higher ranking
Hey All, As a rookie on SEO, I optimised this page http://www.nhad.de/fernstudium/sprachkurse/arabisch/sprachkurse-arabisch/arabisch.aspx for the keywords Arabisch lernen. I saw a huge improvement from position 100+ to position 45. but now it's stuck around this position. Are there any things I missed on this page? Are there still things to increase the ranking? I hope to learn from you. thanks in advance!
On-Page Optimization | | NHA_DistanceLearning0 -
Content with changing URL and duplicate content
Hi everyone, I have a question regarding content (user reviews), that are changing URL all the time. We get a lot of reviews from users that have been dining at our partner restaurants, which get posted on our site under (new) “reviews”. My worry however is that the URL for these reviews is changing all the time. The reason for this is that they start on page 1, and then get pushed down to page 2, and so on when new reviews come in. http://www.r2n.dk/restaurant-anmeldelser I’m guessing that this could cause for serious indexing problems? I can see in google that some reviews are indexed multiple times with different URLs, and some are not indexed at all. We further more have the specific reviews under each restaurant profile. I’m not sure if this could be considered duplicate content? Maybe we should tell google not to index the “new reviews section” by using robots.txt. We don’t get much traffic on these URLs anyways, and all reviews are still under each restaurant-profile. Or maybe the canonical tag can be used? I look forward to your input. Cheers, Christian
On-Page Optimization | | Christian_T2 -
Does a URL forward slash break up an exact match phrase?
I've seen some organisations implementing keyword phrases per URL with forward slashes in-between the keywords. Would this still work for broad AND exact match keywords when the search engine references the URL? Here's an example with the keyword being "scuola di lingue tedesco". http://www.esl.ch/it/adulti/scuola-di-lingue/tedesco/index.htm Thanks in advance.
On-Page Optimization | | featherseo0 -
How do I remove a Canonical URL Tag?
Some of my report cards say I have too many canonical URL tags. However, there is no information no how to delete one. Can someone give me a link or explain? Thanks.
On-Page Optimization | | dealblogger0 -
Can a page lose ranking because it has too many bold tags?
I run a product search website for the Engineering Sector. Each of the companies listed on the website has a profile page. This page used to rank very well in google. Over the last couple of weeks these pages disappeared for google i.e they fell about 20 to 30 spots in the search results for relevant keywords like the Manufacturer Name. I was trying to analyze the page and found that Each of the Categories that the manufacturer supports was listed in "bold" and linked to the page for that category. A manufacturer can support up to 20 or 30 categories so this results in about 20 or 30 Bold keywords. Is this a bad practice? Could this be a reason for a drop in ranking or do bold keywords on a page not matter? What are your views?
On-Page Optimization | | raghavkapur0 -
How do i chaneg a url without losing pre-existing linkjuice?
my client has a page on his site: www.revisitors.com/REF/TEST the page is for a free traffic offer...i have this page currently ranking 25th or so for "free traffic" - a great keyword to rank for to promote this offer with..... i want the url to be www.revisitors.com/free-traffic.html and then change the title tags to free traffic | free targeted traffic or something along those lines....i have hardly put in any linkbuilding work at all to get to 25 - it was a surprise but now that we are there i want to tighten things up and optimize as well as possible.... how do i do this without losing previously built linkjuice and without having a duplicate content issue for having both pages exist with a 301 redirect.... if i am wrong about something please dont hesitate to set me straight...i am only masquerading as an seo expert these days anyway.....thanks.
On-Page Optimization | | Ezpro90 -
Cononical URL problem
Hello, In the keywords difficulty tool, one of my client's sites has a different number for links to root domain and links to page even though they're both the home page. How do I tell if I have a cononical URL problem, and what do I do about it if I do? Thank you.
On-Page Optimization | | BobGW0 -
Maximum length of a URL for good SEO?
Hi there, We have a content database as part of our site and I noticed that the way the database is loaded with new content, it means that the URL’s for these pages are really long, around 100 characters or sometimes more http://www.xxxyy.org/knowledge-base/documents/word1-word2-word3-word4-word5-word6-word7-word8 Is there a suggted maximum character length for a URL? Kind of like for title tag where I max out at 69… Should I truncate the URL’s or at least reduce the numbers of words in them to something less spammy? Does that make a difference? Thanks!
On-Page Optimization | | inhouseninja0