How can i block the below URLs
-
Google indexed plugins pages for my website. Please check below. How can stop them to be indexed on google.?
http://www.ayurjeewan.com/wp-content/plugins/LayerSlider/static/skins/glass/
http://www.ayurjeewan.com/wp-content/plugins/LayerSlider/static/skins/borderlesslight3d/
http://www.ayurjeewan.com/wp-content/plugins/LayerSlider/static/skins/defaultskin/
My robots.txt file is -
User-agent: * Disallow: /wp-admin/
-
robots.txt:
Disallow:
/wp-admin/
/wp-content/plugins/
Then, deindex them as mentioned in the first response.
-
It's also worth remembering that blocking a URL in a robots.txt file does not automatically mean that the URL will be deindexed. The robots.txt file will prevent robots, such as the Googlebot, from seeing or accessing the file and that's all. Now obviously, if a robot repeatedly tries to access a file and gets denied it will eventually stop trying to, which is what leads to the file being deindexed (the same principle applies to 404 and 410 errors).
Therefore, if you want a quicker and more definite deindexing solution, you should use the explicit noindex robot command, as recommended above. This will tell any visiting robot to not index it straight away, which will reduce the number of revisits and drop the page from the index faster.
-
Take a look over here - http://www.robotstxt.org/robotstxt.html and we can't forget the Moz version - http://moz.com/learn/seo/robotstxt
Alternatively you can just add meta noindex on the pages with with the added bonus of letting link juice flow better as well as being a bit more stern to robots ( recommend the noindex tag!)
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How can i increase DA of my site?
hello everyone, I have my 4-month-old blog, PA of the website is 17 yet DA is as yet 5. I don't have the foggiest idea how to expand the DA of the site. It would be ideal if you propose to me how to expand the DA of Site get best jigsaw Thanks
On-Page Optimization | | Salman425521 -
How would you improve our URL structure?
Hi Mozzers, I have a question about the URL structure on our website (www.ikwilzitzakken.nl). We now have a main category with "zitzakken" (beanbags). We also have different brands, types and colours. Now we have URL's like this: <a>https://www.ikwilzitzakken.nl/zitzakken/vetsak/vetsak-fs600-flokati-zitzak/_381_w_3544_3862_NL_1</a> which seems long and not clean. Please don't look at the query at the end, we can't do anything about that in our CMS. In english this would be: https://www.iwantbeanbags.nl/beanbags/vetsak/vetsak-fs600-flokati-beanbag/_381_w_3544_3862_NL_1 How would you optimise this? We do have good rankings (this one ranks #1 for example), but I think our overall structure could be way better. Would love your thoughts about this.
On-Page Optimization | | TheOnlineWarp0 -
URL / Meta info for Author bio pages
When you create a biography page for authors that point back to their G+ account, do you include any other signals in the URL other than the authors name? For example, would you use www.domain.com/sam-spade or www.domain.com/sam-spade-biography, etc? Similar question for the meta description. Do you add any signals there? Or, is rel=author strong enough. Best,
On-Page Optimization | | ChristopherGlaeser
Christopher0 -
Can you recommend a few resources for Yahoo SEO
We get very little traffic from Yahoo Search and it seems only 15%-20% of our pages are indexed by Yahoo and Bing although we do submit sitemaps to both of them. Do you have some tips or resources (articles, blogs, books) you could recommend for Yahoo Search optimization?
On-Page Optimization | | AEM130 -
How do you block development servers with robots.txt?
When we create client websites the urls are client.oursite.com. Google is indexing theses sites and attaching to our domain. How can we stop it with robots.txt? I've heard you need to have the robots file on both the main site and the dev sites... A code sample would be groovy. Thanks, TR
On-Page Optimization | | DisMedia0 -
Keywords in URL:
what kind of URL should we use? www.keyword.net/keyword-city or www.keyword.net/city which URL you would prefer?
On-Page Optimization | | alibeef0 -
Can someone please help me identify where all these URLS to my homepage are coming from?
Hi. I installed the SEOmoz toolbar for Firefox, and analyzed my home page, then clicked on 'get a full site analysis at Site Explorer'. This is what came up: http://www.opensiteexplorer.org/www.frs-solutions.com%252Fcontent%252Fhome/a!links?src=mb I hope that link works. If not, the URL is www.frs-solutions.com Anyway, there are about 57 different URLS within my site all pointing to my homepage! I have no idea where they are coming from. Can someone with an experienced eye take a quick look and tell me what I might be up against? Thank you!
On-Page Optimization | | aprilm-1890400 -
How to use good keyword URL to help main site
Hi. I'm a long time ecommerce guy and starting a third business. The main site URL is the name of the new business but I also purchased a .com URL that is our #1 keyword to target. So I need to know the best strategy to use the keyword url for helping with getting a top ranking for that keyword. I'm curious if I can or should build out the keyword URL site for the search engines and use a 301 redirect. Can you get top ranking for a site that just redirects? Anyway, I guess you get my question. This keyword gets a ton of perfectly targeted traffic so seems like a goldmine if I work it right. Thanks very much.
On-Page Optimization | | jimmyseo1