How can i block the below URLs
-
Google indexed plugins pages for my website. Please check below. How can stop them to be indexed on google.?
http://www.ayurjeewan.com/wp-content/plugins/LayerSlider/static/skins/glass/
http://www.ayurjeewan.com/wp-content/plugins/LayerSlider/static/skins/borderlesslight3d/
http://www.ayurjeewan.com/wp-content/plugins/LayerSlider/static/skins/defaultskin/
My robots.txt file is -
User-agent: * Disallow: /wp-admin/ -
robots.txt:
Disallow:
/wp-admin/
/wp-content/plugins/
Then, deindex them as mentioned in the first response.
-
It's also worth remembering that blocking a URL in a robots.txt file does not automatically mean that the URL will be deindexed. The robots.txt file will prevent robots, such as the Googlebot, from seeing or accessing the file and that's all. Now obviously, if a robot repeatedly tries to access a file and gets denied it will eventually stop trying to, which is what leads to the file being deindexed (the same principle applies to 404 and 410 errors).
Therefore, if you want a quicker and more definite deindexing solution, you should use the explicit noindex robot command, as recommended above. This will tell any visiting robot to not index it straight away, which will reduce the number of revisits and drop the page from the index faster.
-
Take a look over here - http://www.robotstxt.org/robotstxt.html and we can't forget the Moz version - http://moz.com/learn/seo/robotstxt
Alternatively you can just add meta noindex on the pages with with the added bonus of letting link juice flow better as well as being a bit more stern to robots ( recommend the noindex tag!)
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Using the Same Word in Every URL
Just looking to get some opinions on this. Some coupon sites use "coupon" in all of their URLs - this is a practice I would avoid, as to me it is a little spammy. For example:
On-Page Optimization | | vanessakohl
.com/amazon-coupons/
.com/ebay-coupons/
.com/walmart-coupons/
... and so on for thousands of other brands. I don't think this is necessary, as Google will understand from the content, backlinks and the domain name (including the word "coupon") that brand pages are coupon-focused. Any other thoughts on this?0 -
Long url links
Just wondering about creating links.
On-Page Optimization | | Robotnik
Is it ok to have very long links?
Like: http://www.robotnik.com/computer-hardware-ram/8gb-ddr3-1600-desktop Is the above too long, is it better for SEO to be more to the point? Also For better SEO, is it better to use hyphens in a domain name or not?0 -
URL advice
Hi & thanks for looking, I'm not sure if I've adopted the best SEO URL structure for my site, www.vintageheirloom.com For instance, www.vintageheirloom.com/product-category/authentic-designer-vintage-bags/ Works great for the top level category 'All bags', as I'm trying to keyword authentic designer vintage bags. However the sub categories for instance 'Clutch bags' appears as, www.vintageheirloom.com/product-category/authentic-designer-vintage-bags/vintage-clutch-bags/. As you can see at the moment this URL contains duplicate terms vintage & bags. I'm guessing that duplicate keywords in a url isn't too smart, but should amend with Option 1, 2, 3 or something completely different? Option 1 - keep the top level category url the same, change the subcategory: www.vintageheirloom.com/product-category/authentic-designer-vintage-bags/clutch/ Option 2 - amend the top level category: www.vintageheirloom.com/product-category/authentic-designer/vintage-clutch-bags/ Option 3 - amend the top level category as this: www.vintageheirloom.com/product-category/bags/authentic-designer-vintage-clutch/ By the way I'm using WordPress with Woocommerce. I've asked but it's not possible with some technical issues to remove the /product-category/ section. But each product is for example just: www.vintageheirloom.com/shop/vintage-coach-yellow-duffel-sac-bag/ .... sweet. Thanks again !!
On-Page Optimization | | well-its-1-louder0 -
Post url not matching with post title ( wordpress)
I have this site called searchoflife.com on which I have noticed the post urls are not matching with the post title. For Example:Post Title - A Dialogue With NaturePost URL - http://searchoflife.com/dialogue-nature-2013-09-12 Words like 'A' and 'with' are not present in the post URL. This has been the trend since a few days. After investigating I found out that it was due to my plugin SEO ultimate which is actually creating post slugs automatically for the post urls. So my question is whether it is advisable to use post slugs instead of the full post url. Does it affect the SERPS for my site?
On-Page Optimization | | toxicpls0 -
Schema business address - can this affect me in the national rankings
Looking at using Schema microdata on our new website, we currently rank quite decently in Google UK. If I was to add schema mark up for our business address could this effect us in Google UK rankings wise i.e. our rankings for national search (for want of a better word!) suffer but our local rankings increase?
On-Page Optimization | | Jon-C0 -
Long URLs
Many URLs of my site are long due to long navigation paths. Here is an example: http://tinyurl.com/6qc4syb. My question is, if I shorten the urls (which I probably should do), does it matter that they no longer follow the navigation path?
On-Page Optimization | | rdreich490 -
Is it good to have dashes in url's
When using keywords in url's for internal pages, isn't it a good idea to use dashes or underscores in the url between the keywords?
On-Page Optimization | | BradBorst0 -
2 URLs, same content, 1 with keywords. Does this hurt me?
I'm in the process of adding some new features to our site and have a question about our URLs. Most of our URLs consist of either sitename.com/contentname or sitename.com/content/contentid I'm in the process of building a directory to those pages. The directory has a number of filters which will ultimately point to the destination page. sitename.com/filter1/filter2/contentid or sitename.com/filter1/filter2/contentname The destinations will have references. From an SEO perspective, I would think I want the filter1/filter2 version of the link indexed since this will add keywords that someone might search for. However, since the filters are dynamic, if someone just searches for contentname I would want to have sitename.com/contentname returned in the search results. Do I get any SEO benefit out of building those filter links as described if they are not the canonical links?
On-Page Optimization | | JoeCotellese810