How can i block the below URLs
-
Google indexed plugins pages for my website. Please check below. How can stop them to be indexed on google.?
http://www.ayurjeewan.com/wp-content/plugins/LayerSlider/static/skins/glass/
http://www.ayurjeewan.com/wp-content/plugins/LayerSlider/static/skins/borderlesslight3d/
http://www.ayurjeewan.com/wp-content/plugins/LayerSlider/static/skins/defaultskin/
My robots.txt file is -
User-agent: * Disallow: /wp-admin/
-
robots.txt:
Disallow:
/wp-admin/
/wp-content/plugins/
Then, deindex them as mentioned in the first response.
-
It's also worth remembering that blocking a URL in a robots.txt file does not automatically mean that the URL will be deindexed. The robots.txt file will prevent robots, such as the Googlebot, from seeing or accessing the file and that's all. Now obviously, if a robot repeatedly tries to access a file and gets denied it will eventually stop trying to, which is what leads to the file being deindexed (the same principle applies to 404 and 410 errors).
Therefore, if you want a quicker and more definite deindexing solution, you should use the explicit noindex robot command, as recommended above. This will tell any visiting robot to not index it straight away, which will reduce the number of revisits and drop the page from the index faster.
-
Take a look over here - http://www.robotstxt.org/robotstxt.html and we can't forget the Moz version - http://moz.com/learn/seo/robotstxt
Alternatively you can just add meta noindex on the pages with with the added bonus of letting link juice flow better as well as being a bit more stern to robots ( recommend the noindex tag!)
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How do you handle URLs with slashes?
I asked this question before, but with a different scenario. I upgraded my plan to a more advanced cart and all of my URLs changed about 1.5 years ago. I knew nothing about redirects and such, so none of that was done. Basically, let's say my site was: http://www.abc.com, but when people actually visit my site, they are directed to https://www.abc.com/. I have asked my host about redirecting and she that it is not possible. In the past, the link shared has been just www.abc.com . Will this hurt my ranking? My second question is ...let's say I have a link http://www.abc.com/blog , but now, the link is http://www.abc.com/blog/ . Will I be affected, since all my old links omit the slash?
On-Page Optimization | | tiffany11030 -
URL Structure
What's the best way to set up a url structure? When a user goes through the funnel should it show it in the url? Like this: domain.com/thickness/high-density/1-mil-plastic-bags (1 mil plastic bags is a subcategory - when the user is at this page they will see many products. When they select one - it brings them to a product detail page which I think should be done like this: domain.com/product-name regardless of the funnel that brought them there. Does this make sense?) or **domain.com/1-mil-plastic-bags ** Also, is there a limit of how many "/" could be used?
On-Page Optimization | | EcomLkwd0 -
Duplicate URL for homepage
Hi Gurus, Thank you for reading this question My site is developed in Classic ASP How can i make sure the homepage is not duplicated for http://www.partyrama.co.uk/ http://www.partyrama.co.uk/default.asp http://partyrama.co.uk/ http://partyrama.co.uk/default.asp Regards Sri
On-Page Optimization | | partyrama0 -
Can I force an update of Grade Reports?
It looks like my weekly crawl has finished, but my Grade Reports still reflect last week. Is there a way to manually update them, or do I just have to wait it out?
On-Page Optimization | | FDAitsupport0 -
Blog URL
I know that this question has been asked in the past, and that website.com/blog is better for seo purposes than blog.website.com. We want to setup a custom blog on our site, using Wordpress. Our designers/host are telling us that buy using website.com/blog can causes issues b/c Wordpress is open source, and our site could be hacked? Is there anything we should do about this? Any suggestions? Any Advice appreciated!!! Thanks!
On-Page Optimization | | TP_Marketing0 -
Tag-URLs in Magento
Hello, I have got a problem concerning Tag-URLs in Magento (the URLs mentioned are just fictitious 😞 At the moment, they look something like this: (1) http://store.com/tag/product/list/tagId/1/ ... so these URLs are not search engine friendly at all. Using a Magento extension you could transform them in speaking URLs: (2) http://store.com/tag/digital-cameras What would you do if you sold, say, digital cameras and your online shop ranked high for the keyword "digital camera" with URL No. 1 (not search engine friendly). Would you transform (1) in (2) and 301 all non speaking URLs? But would you keep the high ranking for "digital camera" when 301 to URL No. (2). But, what I'm most concerned of is : There is actually a landing page (category page) for the keyword "digital camera" : http://store.com/digital-cameras. Shouldn't the last URL rank high for "digital camera"? (instead of the tag URLs). But given the situation above, does it make sense now to 301 the tag URL to the category page? I would perhaps lose my good ranking, wouldn't I? Thanks a lot for your help! Martin
On-Page Optimization | | SmartyMarty810 -
How long is too long for domain URL length?
I noticed one of the negatively correlated ranking factors was length of URL. I'm building a page from scratch, we are trying to rank for 'Minneapolis Fitness' and 'Minneapolis Massage'. Is www.minnnepolismassageandfitness.com just ridiculously long? Or does the exact match outweigh the penalty for URL length?
On-Page Optimization | | JesseCWalker2 -
Best URL Structure For Products That Are The Same
I know that the url structure is very important for seo preferably using the keyword. But is it okay to have the same url with the product number at the end ? Each of our products have a name with a product number. Or will this cause to many similar urls? or if the folder is the name of the product that needs to be optimized, can the page just be called the product number? Example: Say you have a 20 different product lines and they are all catagorized in the appropriate folders, and need to be optimized for the actual product name. XXX (folder name ) WWW-PR-123 WWW-PR-1234 WWW-PR-12345 WWW-PR-123456 what would be the best url structure? Can they have the same begining? The product name? something like: www.example.com/xxx/www-pr-123.php www.example.com/xxx/www-pr-1234.php or www.example.com/xxx/pr-123.php www.example.com/xxx/pr-1234.php
On-Page Optimization | | hfranz0