How can i block the below URLs
-
Google indexed plugins pages for my website. Please check below. How can stop them to be indexed on google.?
http://www.ayurjeewan.com/wp-content/plugins/LayerSlider/static/skins/glass/
http://www.ayurjeewan.com/wp-content/plugins/LayerSlider/static/skins/borderlesslight3d/
http://www.ayurjeewan.com/wp-content/plugins/LayerSlider/static/skins/defaultskin/
My robots.txt file is -
User-agent: * Disallow: /wp-admin/ -
robots.txt:
Disallow:
/wp-admin/
/wp-content/plugins/
Then, deindex them as mentioned in the first response.
-
It's also worth remembering that blocking a URL in a robots.txt file does not automatically mean that the URL will be deindexed. The robots.txt file will prevent robots, such as the Googlebot, from seeing or accessing the file and that's all. Now obviously, if a robot repeatedly tries to access a file and gets denied it will eventually stop trying to, which is what leads to the file being deindexed (the same principle applies to 404 and 410 errors).
Therefore, if you want a quicker and more definite deindexing solution, you should use the explicit noindex robot command, as recommended above. This will tell any visiting robot to not index it straight away, which will reduce the number of revisits and drop the page from the index faster.
-
Take a look over here - http://www.robotstxt.org/robotstxt.html and we can't forget the Moz version - http://moz.com/learn/seo/robotstxt
Alternatively you can just add meta noindex on the pages with with the added bonus of letting link juice flow better as well as being a bit more stern to robots ( recommend the noindex tag!)
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Is it better to shorten my existing url to use only keyword after domain with a 301 redirect from existing url
I have a long existing URL that has included my key word but the url has about 5 additional words in the text ( eg url would have " /super widgets in stock at the widget store " as url text after domain. keywords is super widget The URL was at the top of search results for my keyword for many years until recently. Is it better to shorten my url text to now use only my keyword " /super-widgets " after the domain with a 301 direct from my existing url to optimize it Thanks
On-Page Optimization | | mrkingsley2 -
can we include iframe content if we properly reference the source in the url?
can we safely include iframe content if we properly source the source url. eg we have a travel company we are looking to produce another website, but use the same hotel descriptions in the hotel directory. can we do this?
On-Page Optimization | | Direct_Ram0 -
SEO can id and class be used in H1?
Can ID and class be used in my H1 tag. I realize best case would be to change it, but it's going to require a change order from the ecommerce company to fix their sloppy code. Will this hurt seo? Example:
On-Page Optimization | | K-WINTER0 -
URL Question
This url looks bad: http://www.patrickmunoz.com/#!classes/c1vw1 And when you click around the page change doesn't actually occur, it's a fade into the next page. I think this is a major problem for rankings. Although pages are crawled: https://www.google.com/search?q=site%3Ahttp%3A%2F%2Fwww.patrickmunoz.com%2F&oq=site%3A&aqs=chrome.2.69i57j69i58j69i59l3j69i61.3548j0j7&sourceid=chrome&espv=210&es_sm=122&ie=UTF-8 When I search for a simple page - "patrick munoz FAQs" nothing comes up:
On-Page Optimization | | tylerfraser
https://www.google.com/search?q=site%3Ahttp%3A%2F%2Fwww.patrickmunoz.com%2F&oq=site%3A&aqs=chrome.2.69i57j69i58j69i59l3j69i61.3548j0j7&sourceid=chrome&espv=210&es_sm=122&ie=UTF-8#q=patrick+munoz+|+FAQs Do you think this is a bad url configuration? Thanks! Tyler0 -
How can I nofollow my affiliate links?
I have a lot of affiliate links and I need to find an efficient way to nofollow them. I have over 500 blog posts and most have an affiliate link. I use Wordpress and Genesis. Any advice?
On-Page Optimization | | 2bloggers0 -
Is it possible to have the crawler exclude urls with specific arguments?
Is it possible to exclude specific urls in the crawl that contain certain arguments - like you can do in google webmaster tools?
On-Page Optimization | | djangojunkie0 -
Is it better to embed my longtail article or give it a separate url?
Within my e-commerce site I have jewelers who using uncommon techniques and maetrials. I have a few long tail type article ready to publish about these niche topics. My site navigation has each jeweler as a category with their often changing products within their category. I am thinking I would add an article to the artist-category content. But in the past, I have put "how to" or "what is" content in an article section of the site. This way I could link to it from several places. With the long tail in mind, would I be better off adding the article to the jeweler's category page? If I have a 2nd jeweler using this same technique, I am thinking I would rewrite the 1st article including different long tail phrases. Thank You for Helping- Handcrafter
On-Page Optimization | | stephenfishman0 -
URL length does it really make a difference
I am currently working redevloping my site, I have terribly long URL's according to SEOMoz, can someone cover the real benefit of my editing my URLs, and then putting in all of those redirects. Should I edit old material? My site has been in wordpressformat for a couple of years now.
On-Page Optimization | | copykatrecipes0
