How can i block the below URLs
-
Google indexed plugins pages for my website. Please check below. How can stop them to be indexed on google.?
http://www.ayurjeewan.com/wp-content/plugins/LayerSlider/static/skins/glass/
http://www.ayurjeewan.com/wp-content/plugins/LayerSlider/static/skins/borderlesslight3d/
http://www.ayurjeewan.com/wp-content/plugins/LayerSlider/static/skins/defaultskin/
My robots.txt file is -
User-agent: * Disallow: /wp-admin/ -
robots.txt:
Disallow:
/wp-admin/
/wp-content/plugins/
Then, deindex them as mentioned in the first response.
-
It's also worth remembering that blocking a URL in a robots.txt file does not automatically mean that the URL will be deindexed. The robots.txt file will prevent robots, such as the Googlebot, from seeing or accessing the file and that's all. Now obviously, if a robot repeatedly tries to access a file and gets denied it will eventually stop trying to, which is what leads to the file being deindexed (the same principle applies to 404 and 410 errors).
Therefore, if you want a quicker and more definite deindexing solution, you should use the explicit noindex robot command, as recommended above. This will tell any visiting robot to not index it straight away, which will reduce the number of revisits and drop the page from the index faster.
-
Take a look over here - http://www.robotstxt.org/robotstxt.html and we can't forget the Moz version - http://moz.com/learn/seo/robotstxt
Alternatively you can just add meta noindex on the pages with with the added bonus of letting link juice flow better as well as being a bit more stern to robots ( recommend the noindex tag!)
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Anyone know the benefit of URLs being in the target language?
For example, a page targeting the spanish language, does it make a difference if the URL is site.com/i-don't-speak-spanish vs site.com/no-habla-espanol
On-Page Optimization | | LennyO0 -
URL for a new website
Hi, I am creating a new website for a client. Is it best to include the keywords from the most common search in the domain name, they would like: forenamesurname.com but should I be recommending: weddingmakeupbyforename.com Does it make much difference to search rankings if the keyword is in the domain name? Thanks v much
On-Page Optimization | | danieldunn100 -
Tips on URL structure for a site re-design
Wanted to know what you would do with regards to urls – in an ideal world how would you structure them? Keen to know as me and dave are soon to have a meeting about this and were wondering about changing them from the current – http://www.looking4parking.com/airport/gatwick to something like - www.looking4parking.com/gatwick-airport-parking We will soon have pages for the specific parking types that will be a lot more engaging to users with some really useful content on benefits, features, how a certain type of parking works, images, video etc. Currently going to a type of parking, such as meet and greet just brings up a dropdown modal – I was thinking of having the url structure looking like this – www.looking4parking.com/gatwick-airport-meet-and-greet-parking www.looking4parking.com/gatwick-airport-on-site-parking www.looking4parking.com/gatwick-airport-park-and-ride We will then have specific pages for each parking product – in which this product will have unique content built around it – each will have an overview of the product, benefits, features, reviews, images, directions to the car park, find your route and eventually a video on each product So for example we currently have the product “Jet Parks 2” at Manchester airport – the current url is - http://www.looking4parking.com/airport/manchester/park-and-ride/jetparks-2 I would like to change this now we have the opportunity to refresh the whole system, to something along the lines of **domain/location/product title - **www.looking4parking.com/manchester-airport-parking/jetparks-2 or as we have some similar products at certain airports (mainly where the airport has multiple terminals) we would just change it to the following - www.looking4parking.com/manchester-airport-parking/jetparks-3 What are peoples thoughts/opinions on the above?
On-Page Optimization | | RyanCrawf19840 -
How can I find out which filter my site has fallen under?
How can I find out which filter my site has fallen under? there only 22 pages (out of 380) that are not under Google filters Thank you in advance.
On-Page Optimization | | andreysmiling19870 -
Which URL Structure is best for New Website?
Hello All, I have one client and he want to develop his website but the he want the URL structure for his website page like below: http://www.example.com/category/xxproduct.html But I have suggest him below URL Structure http://www.example.com/category/xx-product.html So Can you people please suggest me that from above which URL structure is better from SEO side as well as from Visual side..? Is using - is better to separate the words in URL?
On-Page Optimization | | jemindesai0 -
Why do I hear that it can be a bad thing to have too many content pages?
My site has lots of long tail opportunity and so intention is to produce as much content as possible to target these keywords effectively. However, over the last 6 months I have come across numerous sources suggesting that Google can/does look down on a site having too many content pages. Is there any truth in this? Thanks.
On-Page Optimization | | Clicksjim0 -
Wordpress pages URL's redirection.
I was checking W3C Markup Validation and in report it was shown that that pages (not post or any other URL's just PAGES) at investmentcontrarians.com are 301 redirected. e.g. original URL "http://www.investmentcontrarians.com/debt-crisis" which is redirected to "http://www.investmentcontrarians.com/debt-crisis/" I know that its not that serious issue, but still want to know why only pages are being redirected and how can we avoid it.
On-Page Optimization | | NumeroUnoWebSolutions0 -
Redirecting URLS on windows
Could anyone help out here please. A client of ours have reveloped their website from HTML to ASP (helpful!). They have 60 odd pages indexed in Google with the .html extension. We need to do a redirect on these pages so that all link juice is passed to the new pages. What would be the best way to do this please?
On-Page Optimization | | Grumpy_Carl0
