How can i block the below URLs
-
Google indexed plugins pages for my website. Please check below. How can stop them to be indexed on google.?
http://www.ayurjeewan.com/wp-content/plugins/LayerSlider/static/skins/glass/
http://www.ayurjeewan.com/wp-content/plugins/LayerSlider/static/skins/borderlesslight3d/
http://www.ayurjeewan.com/wp-content/plugins/LayerSlider/static/skins/defaultskin/
My robots.txt file is -
User-agent: * Disallow: /wp-admin/ -
robots.txt:
Disallow:
/wp-admin/
/wp-content/plugins/
Then, deindex them as mentioned in the first response.
-
It's also worth remembering that blocking a URL in a robots.txt file does not automatically mean that the URL will be deindexed. The robots.txt file will prevent robots, such as the Googlebot, from seeing or accessing the file and that's all. Now obviously, if a robot repeatedly tries to access a file and gets denied it will eventually stop trying to, which is what leads to the file being deindexed (the same principle applies to 404 and 410 errors).
Therefore, if you want a quicker and more definite deindexing solution, you should use the explicit noindex robot command, as recommended above. This will tell any visiting robot to not index it straight away, which will reduce the number of revisits and drop the page from the index faster.
-
Take a look over here - http://www.robotstxt.org/robotstxt.html and we can't forget the Moz version - http://moz.com/learn/seo/robotstxt
Alternatively you can just add meta noindex on the pages with with the added bonus of letting link juice flow better as well as being a bit more stern to robots ( recommend the noindex tag!)
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Does having a \ on a URL make a difference?
On our website we have CMS pages which end normally without a /. However when linking from web banners e.c.t it always adds a / on the end. Will this have a negative impact on SEO?
On-Page Optimization | | AHF_Furniture0 -
Link flow for multiple links to same URL
Hi there,
On-Page Optimization | | doctecs
my question is as follows: How does Google handle link flow if two links in a given page point to the same URL? (do they flow link individually or not?) This seems to be a newbie question, but actually it seems that there is little evidence and even also little consensus in the SEO community about this detail. Answers should include source Information about the current state of art at Google is preferable The question is not about anchor text, general best practises for linking, "PageRank is dead" etc. We do know that the "historical" PageRank was implemented (a long time ago) without special handling for multiple links, as e.g. last stated by Matt Cutts in this video: http://searchengineland.com/googles-matt-cutts-one-page-two-links-page-counted-first-link-192718 On the other hand, many people from the SEO community say that only the first link counts. But so far I could not find any data to back this up, which is quite surprising.0 -
What word should I use in my URL for my blog
Should I use the word "blog" in my sub folder as in : http://www.mybusiness.com/blog or should I use http://www.mybusiness.com/news. Is there a difference for when my site is crawled. I understand that a blog works a little differently. Can someone explain the basics?
On-Page Optimization | | graemesanderson0 -
Canonical URLs and SEO
After publishing my new wordpress website my on page analysis shows two canonical urls on my homepage. In checking the source code the Yoast SEO plugin states my canoncial url is www.homepage.ca/ This is the only one showing. Having said this I do not add the trailing slash to anything on my url structure. Could this be why the on page analysis tool says there are 2? If so is there any way to resolve this?
On-Page Optimization | | casper4340 -
SEF URLs. Should I use / or - ?
I have o activate SEF URLs in a website. Regarding SEO, is there any difference between using / or - ? I mean, Is it better to write URLs like this: http://www.domain.com/folder/folder/page or like this: http://www.domain.com/folder-folder-page ? Is there any difference? Thanks!
On-Page Optimization | | ociosu0 -
Site URL's
We are redeveloping our website, and have the option to amend URLs (with 301 redirects from old URL to new), so my question is: Would 'golfsite.com/golf-clubs' achieve superior rankings than 'golfsite.com/clubs' for the search term 'golf clubs' if all other factors were the same? Should the URL reflect the intended search term wherever possible?
On-Page Optimization | | swgolf1230 -
How to use good keyword URL to help main site
Hi. I'm a long time ecommerce guy and starting a third business. The main site URL is the name of the new business but I also purchased a .com URL that is our #1 keyword to target. So I need to know the best strategy to use the keyword url for helping with getting a top ranking for that keyword. I'm curious if I can or should build out the keyword URL site for the search engines and use a 301 redirect. Can you get top ranking for a site that just redirects? Anyway, I guess you get my question. This keyword gets a ton of perfectly targeted traffic so seems like a goldmine if I work it right. Thanks very much.
On-Page Optimization | | jimmyseo1
