Block web archieve/way back machine
-
Hi i want to block web archive/wayback machine from indexing my site and creating a record of it on their database.
Any ideas on how to do this?
Cheers,
Superpak -
You can block Wayback Machine from crawling and creating a record of your site by adding the following to your Robots.txt file:
User-agent: ia_archiver
Disallow: /This will not only stop new records from being created but also stop people viewing what had previously been indexed by Wayback Machine.
More information about this can be found here: https://archive.org/about/exclude.php
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Hreflang : mixing with/without country code for same language
Hello, I would like to display 3 different english versions of my website : 1 for UK, 1 for CA and 1 for other english users. It would look like this for a page: . (english content with £ prices) <link rel="alternate" href="https: xxx.com="" en-ca" hreflang="en-CA">(english content with $CA prices)</link rel="alternate" href="https:> <link rel="alternate" href="https: xxx.com="" en="" " hreflang="en">(english content without currency)</link rel="alternate" href="https:> I wonder if I can mix this hreflang without country code with hreflangs with country code for the 2 other specific versions... or if the hreflang without country code version will appear whatever the country, even if i specified it . In other terms, is hreflang="en" > hreflang="en-CA" + hreflang="en-GB" if tagged together on a same page? Thank you
Intermediate & Advanced SEO | | AlexisH0 -
Pages blocked by robots
**yazılım sürecinde yapılan bir yanlışlıktı.** Sorunu hızlı bir şekilde nasıl çözebilirim? bana yardım et. ```[XTRjH](https://imgur.com/a/XTRjH)
Intermediate & Advanced SEO | | mihoreis0 -
2015/2016 Sitemaps Exclusions
Hello fellow mozrs!
Intermediate & Advanced SEO | | artdivision
Been working on a few Property (Real Estate for our American friends) websites recently and and two questions that constantly come up as we spec the site are: 1. What schema (schema.org) should the website use (throughout all pages as well as individual pages). Did anyone found that schema actually helped with their ranking/CTR?
2. Whilst setting up the sitemaps (usually Yaost is our preferred plugin for the job), what page would you EXCLUDE from the site map? Looking forward to some interesting comments.
Dan.0 -
Google v bing/yahoo
Had a penguin unnatural link building penalty was lifted last June. Since then no significant recovery in rankings. Saying that why do we still rank on page 1 on Bing and Yahoo but rank nowhere in Google. Any suggestions. Thanks David
Intermediate & Advanced SEO | | Archers0 -
Should Site Search results be blocked from search engines?
What are the advantages & disadvantages of letting Google crawl site search results? We currently have them blocked via robots.txt, so I'm not sure if we're missing out on potential traffic. Thanks!
Intermediate & Advanced SEO | | pbhatt0 -
Does having a file type on the end of a url affect rankings (example www.fourcolormagnets.com/business-cards.php VS www.fourcolormagnets.com/business-cards)????
Does having a file type on the end of a url affect rankings (example www.fourcolormagnets.com/business-cards.php VS www.fourcolormagnets.com/business-cards)????
Intermediate & Advanced SEO | | JHSpecialty0 -
Problem w/ 301 Redirect
Here is how I did the configuration of the redirects: I don’t understand why the destination page is different from the one is configured in the apache server. Any ideas? For example: http://www.meliacaribetropical.com/spanish/entertainment/ is being already being 301 redirected to a 404 page (http://www.meliacaribetropical.com/es/index.htmlentertainment/) that does not exist on the Apache server. As you can see, the url was incorrectly written. Another occurrence from the spreadsheet is http://www.meliacaribetropical.com/spanish/gallery/beach.html, which is also being 301 redirected to a 404 page (http://www.meliacaribetropical.com/es/index.htmlgallery/beach.html). This is causing a hard 404 page. Here is my .httpaccess file: <virtualhost 192.168.200.25:80=""></virtualhost> ServerAdmin ecommerce@sol-group.com DocumentRoot /home/www/solgroup/americas/meliacaribetropical.com ServerName www.meliacaribetropical.com ServerAlias meliacaribetropical.com Redirect permanent /spanish/services/ http://www.meliacaribetropical.com/en/services.html Redirect permanent /entertainment/ http://www.meliacaribetropical.com/en/services.html Redirect permanent /press/ http://www.meliacaribetropical.com/en/index.html Redirect permanent /spanish/ http://www.meliacaribetropical.com/es/index.html Redirect permanent /es/restaurantes/ http://www.meliacaribetropical.com/es/gastronomia.html Redirect permanent /spanish/entertainment/ http://www.meliacaribetropical.com/es/servicios.html Redirect permanent /spanish/services/ http://www.meliacaribetropical.com/es/servicios.html Redirect permanent /es/spa/ http://www.meliacaribetropical.com/es/servicios.html Redirect permanent /spanish/accommodations/ http://www.meliacaribetropical.com/es/habitaciones.html Redirect permanent /spanish/spa/ http://www.meliacaribetropical.com/es/servicios.html Redirect permanent /spanish/royal/ http://www.meliacaribetropical.com/es/servicio-real.html Redirect permanent /spanish/dining/ http://www.meliacaribetropical.com/es/gastronomia.html Redirect permanent /spanish/flintstones/ http://www.meliacaribetropical.com/es/index.html Redirect permanent /es/galeria/ http://www.meliacaribetropical.com/es/visor.html Redirect permanent /spanish/gallery/ http://www.meliacaribetropical.com/es/visor.html Redirect permanent /es/reuniones-eventos/ http://www.meliacaribetropical.com/es/grupos.html Redirect permanent /lowest-rate.php http://www.meliacaribetropical.com/en/index.html Redirect permanent /es/los-picapiedra/ http://www.meliacaribetropical.com/es/index.html Redirect permanent /gallery/beach.html http://www.meliacaribetropical.com/en/index.html Redirect permanent /gallery/dining.html http://www.meliacaribetropical.com/en/gastronomy.html Redirect permanent /gallery/pools.html http://www.meliacaribetropical.com/en/index.html Redirect permanent /spanish/sitemap.html http://www.meliacaribetropical.com/es/index.html Redirect permanent /es/galeria/playa.html http://www.meliacaribetropical.com/es/index.html Redirect permanent /es/galeria/restaurantes.html http://www.meliacaribetropical.com/es/gastronomia.html Redirect permanent /es/galeria/piscinas.html http://www.meliacaribetropical.com/es/index.html Redirect permanent /es/prensa/ http://www.meliacaribetropical.com/es/index.html Redirect permanent /spanish/gallery/beach.html http://www.meliacaribetropical.com/es/index.html Redirect permanent /spanish/gallery/pools.html http://www.meliacaribetropical.com/es/index.html Redirect permanent /spanish/gallery/dining.html http://www.meliacaribetropical.com/es/gastronomia.html Redirect permanent /spanish/press/ http://www.meliacaribetropical.com/es/index.html Redirect permanent /en/groups.html http://www.meliacaribetropical.com/en/groups.html Redirect permanent /terms-condition.php http://www.meliacaribetropical.com/en/index.html Redirect permanent /es/all_inclusive.html http://www.meliacaribetropical.com/es/index.html Redirect permanent /es/terms-condition.php http://www.meliacaribetropical.com/es/index.html Redirect permanent /es/prensa/family-facilities-amenities.html http://www.meliacaribetropical.com/es/index.html Redirect permanent /es/outside-us-telephone-listing.php http://www.meliacaribetropical.com/en/index.html Redirect permanent /spanish/press/melia-international-brand-overhaul.html http://www.meliacaribetropical.com/es/index.html Redirect permanent /es/prensa/melia-international-brand-overhaul.html http://www.meliacaribetropical.com/es/index.html Redirect permanent /press/family-facilities-amenities.html http://www.meliacaribetropical.com/en/index.html Redirect permanent /press/melia-international-brand-overhaul.html http://www.meliacaribetropical.com/en/index.html Redirect permanent /es/prensa/melia-caribe-tropical-announces-fall-promotion.html http://www.meliacaribetropical.com/es/index.html Redirect permanent /spanish/press/melia-caribe-tropical-announces-fall-promotion.html http://www.meliacaribetropical.com/es/index.html Redirect permanent /press/melia-caribe-tropical-announces-fall-promotion.html http://www.meliacaribetropical.com/en/index.html
Intermediate & Advanced SEO | | Melia0
