Welcome to the Q&A Forum

Browse the forum for helpful insights and fresh discussions about all things SEO.

Your browser does not seem to support JavaScript. As a result, your viewing experience will be diminished, and you have been placed in read-only mode.

Please download a browser that supports JavaScript, or enable it if it's disabled (i.e. NoScript).

Block web archieve/way back machine

Intermediate & Advanced SEO

930

Mikey008 last edited by

Hi i want to block web archive/wayback machine from indexing my site and creating a record of it on their database.

Any ideas on how to do this?

Cheers,
Superpak
1 Reply Last reply
Reply Quote 2
davebuts last edited by

You can block Wayback Machine from crawling and creating a record of your site by adding the following to your Robots.txt file:

User-agent: ia_archiver
Disallow: /

This will not only stop new records from being created but also stop people viewing what had previously been indexed by Wayback Machine.

More information about this can be found here: https://archive.org/about/exclude.php
1 Reply Last reply
Reply Quote 9

Got a burning SEO question?

Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.

Start my free trial

Browse Questions

View

From

Sorted by

With category

Explore more categories

Related Questions

Current Sitelinks Refresh/Removal timeframe?

Hi All, Did good chunk of research here prior to posting on sitelinks. Tried the Webmaster Console to remove the links Tried removing via sitemap, WP SEO plugin removal Now removed unnecessary pages, and redirected to homepage. Re petitioned for removal via Webmaster Google Console area. Whats been the recent conservative realistic estimate of when sitelinks will be removed/refreshed? Also any tricks to expedite recrawl or re build of sitelinks. The site has very low traffic SMB that uses site purely for portfolio and no ranking/traffic purposes. so maybe the priority is very low for this site. Back in the day sitelinks were a lot easier to manage or so it seems. Thoughts?
Intermediate & Advanced SEO | | vmialik

1
How to get back links with higher rank ?

Hi All , These days I am finding new ways of creating back links. Could any one tell me how to get backlinks with higher DA ?
Intermediate & Advanced SEO | | mozentution

2
Two Domains, Same Products/Content

We're an e-commerce company with two domains. One is our original company name/domain, one is a newer top-level domain. The older domain doesn't receive as much traffic but is still searched and used by long-time customers who are loyal to that brand, who we don't want to alienate. The sites are both identical in products and content, which creates a duplicate content issue. I have come across two options so far: 1. a 301 redirect from the old domain to the new one. 2. Optimize the content on the newer domain (the strongest of the two) and leave the older domain content as is. Does anyone know of a solution better than the two I listed above or have experience resolving a similar problem in the past?
Intermediate & Advanced SEO | | ilewis

0
Does anyone know how to appear with snippet that says something like: Jobs 1-10 of 80 in the beginning of the description on Google? e.g. like on: https://www.google.co.za/#q=pickers+and+packers

Does anyone know how to appear with snippet that says something like: Jobs 1-10 of 80 in the beginning of the description on Google? e.g. like on: https://www.google.co.za/#q=pickers+and+packers Any markup that could be used to be listed like this. Why is some sites listed like this and some not. Why is the adzuna.co.za page listed with Results 1-10 while some other with Jobs 1-10 ?
Intermediate & Advanced SEO | | classifiedtech

0
How can this site rank post panda/penguin?

I am doing link building for an adult dating comparison website. One of the main competitors though, having checked their backlink profile have anchor text that is not varied at all. In fact many, many links that are all the same. How can they possibly rank in the post panda/penguin era? In fact they're at number 2! The site is an adult site and it www.f hypen buddy.co.uk if anyone wants to runa backlink check on OSE. Any help greatly appreciated!
Intermediate & Advanced SEO | | SamCUK

0
Could a HTML <select>with large numbers of <option value="<url>">'s affect my organic rankings</option></select>

Hi there, I'm currently redesigning my website, and one particular pages lists hotels in New York. Some functionality I'm thinking of adding in is to let the user find hotels close to specific concert venues in New York. My current thinking is to provide the following select element on the page - selecting any one of the options will automatically redirect to my page for that concert venue. The purpose of this isn't to affect the organic traffic - I'm simply introducing this as a tool to help customers find the right hotel, but I certainly don't want it to have an adverse effect on my organic traffic. I'd love to know your thoughts on this. I must add that in certain cities, such as New York, there could be up to 450 different options in this select element. | <select onchange="location=options[selectedIndex].value;"> <option value="">Show convenient hotels for:</option> <option value="http://url1..">1492 New York</option> <option value="http://url2..">Abrons Arts Center</option> <option value="http://url3..">Ace of Clubs New York</option> <option value="http://url4..">Affairs Afloat</option> <option value="http://url5..">Affirmation Arts New York</option> <option value="http://url6..">Al Hirschfeld Theatre</option> <option value="http://url7..">Alice Tully Hall</option> .. .. ..</select> Many thanks Mike |
Intermediate & Advanced SEO | | mjk26

0
Internal Search / Faceted Navigation

Hi there, I'm working on an e-learning site with the following content pages: main page, category pages, course pages, author pages, tag pages. We will also have an internal search for users to search by keyword for courses & authors & categories. Is it still recommend to "noindex, follow" and disallow in robots.txt internal search results? Or for a site like this, is it better to use faceted navigation? It seems that faceted navigation is mostly for e-commerce sites. What is the latest thinking on SEO best practices for internal search result pages?
Intermediate & Advanced SEO | | mindflash

0
Seo advice / plan ? penalized ?

I built an ecommerce site for a client of mine just over 9 months ago now. To begin with the serps were great, everything was listed in the results but for some reason a few weeks in all the results vanished from google and now we're lucky to find anything. I've been as far as page 200 and havent found any results. Its been like this for a solid 8 months so i can only presume that the site has been penalised in some form. Searching for unique phrases from the site doesnt even return results. The website in question is = http://goo.gl/A6Gz2
keywords we're aiming for = coloured contact lenses, fashion contact lenses
Target = Google UK Now im not really an seo guy but regardless of this my client has hired me to see whats going on and correct it. I've been scratching my head thinking all sorts of things but none of which im certain about so i'm looking for someone to point me in the right direction before i do anything drastic. So to begin with here are some of my suspicions which i personally think are affecting ranking and possible penalisation. #1 - Too many links on the page
#2 - Possibly over optimised
#3 - Lack of content on the product and category pages
#4 - Lack of backlinks and links in general coming from other sites My main concern is the lack of links from other sites and the odd link coming from low quality sites. I've also just found out that my client has been using an automatic link submitter which i've always thought of as a big no no. Some of the sites these links have been submitted to have nothing to do with the keyword we are targetting and are sort of spammy sites containing all sorts of links. Im wondering if these poor quality links could have caused the site to be penalised, google may be seeing it as a spammy site due to this. Whats your opinions on the above, are my suspicions correct and can this be recovered ? My planned course of action is to be as follows: #1 - Re write the content currently on the site so that it is better written and include more keywords, especially long tails since i think these will help bring the serps up.
#2 - Write detailed category and product descriptions as well as making sure every page has some well written content with links and keywords.
#3 - Keep the above pages to one main subject / keyword so that google doesnt get confused.
#4 - Get some links on popular and relevant sites, the only problem here is the lack of fashion contact lens sites. Does anyone have any advice on how to find these or where i should be getting links placed. Are directories worth while ?
#5 - Get more involved in the social side ie facebook, twitter I will be building on the above over time, aswell as running google ads moderately for our chosen keywords. Is there anything i have missed, anything i shouldnt be doing. Please advise. Thanks.
Intermediate & Advanced SEO | | gfxpixeldesigns

0