Welcome to the Q&A Forum

Browse the forum for helpful insights and fresh discussions about all things SEO.

Your browser does not seem to support JavaScript. As a result, your viewing experience will be diminished, and you have been placed in read-only mode.

Please download a browser that supports JavaScript, or enable it if it's disabled (i.e. NoScript).

Block web archieve/way back machine

Intermediate & Advanced SEO

929

Mikey008 last edited by

Hi i want to block web archive/wayback machine from indexing my site and creating a record of it on their database.

Any ideas on how to do this?

Cheers,
Superpak
1 Reply Last reply
Reply Quote 2
davebuts last edited by

You can block Wayback Machine from crawling and creating a record of your site by adding the following to your Robots.txt file:

User-agent: ia_archiver
Disallow: /

This will not only stop new records from being created but also stop people viewing what had previously been indexed by Wayback Machine.

More information about this can be found here: https://archive.org/about/exclude.php
1 Reply Last reply
Reply Quote 9

Got a burning SEO question?

Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.

Start my free trial

Browse Questions

View

From

Sorted by

With category

Explore more categories

Related Questions

Page auto directing to /#/id0 but no 301 in place?

I'm a little perplexed and hope someone technically savvy can help. Wordpress site. Our page: www.curveball-media.co.uk/animation Redirects to: www.curveball-media.co.uk/animation/#/id0 I cannot see any reason for this. No 301s, nothing.
Intermediate & Advanced SEO | | curveballmedia

0
What would be best way to transition from mobile website to responsive

We have a mobile website (mobile.website.com) that mirror our desktop site (www.website.com) with +100 000 pages. We have an alternate tag on our desktop to our mobile site and a user agent detect that redirect mobile traffic to our mobile site Our mobile site is no index and has a canonical to our desktop. Everything works pretty well, the mobile website is not index and only show up in SERP when a user make a search from a mobile. Our main website is now responsive and we would like to kill our mobile site without compromising our traffic. We know that a slight speed change or content change can affect our traffic, what would be the best way to do that? Big bang: redirect all mobile URL to desktop, remove user agent detect and remove alternate tag on desktop Semi Big bang: remove user agent detect and remove alternate tag on desktop and see how the traffic react before redirecting Progressive: remove the user agent detect and the alternate tag on some section of the website to see how the traffic react Other ? Anyone has any experience with that? Thanks and let me know if anything is not clear.
Intermediate & Advanced SEO | | Digitics

0
Https://www.mywebsite.com/blog/tag/wolf/ setting tag pages as blog corner stone article?

We do not have enough content rich page to target all of our keywords. Because of that My SEO guy wants to set some corner stone blog articles in order to rank them for certain key words on Google. He is asking me to use the following rule in our article writing(We have blog on our website):
For example in our articles when we use keyword "wolf", link them to the blog page:
https://www.mywebsite.com/blog/tag/wolf/
It seems like a good idea because in the tag page there are lots of material with the Keyword "wolf" . But the problem is when I search for keyword "wolf" for example on the Google, some other blog pages are ranked higher than this tag page. But he tells me in long run it is a better strategy. Any idea on this?
Intermediate & Advanced SEO | | AlirezaHamidian

0
Thousands of Web Pages Disappered from Google Index

The site is - http://shop.riversideexports.com We checked webmaster tools, nothing strange. Then we manually resubmitted using webmaster tools about a month ago. Now only seeing about 15 pages indexed. The rest of the sites on our network are heavily indexed and ranking really well. BUT the sites that are using a sub domain are not. Could this be a sub domain issue? If so, how? If not, what is causing this? Please advise. UPDATE: What we can also share is that the site was cleared twice in it's lifetime - all pages deleted and re-generated. The first two times we had full indexing - now this site hovers at 15 results in the index. We have many other sites in the network that have very similar attributes (such as redundant or empty meta) and none have behaved this way. The broader question is how to do we get the indexing back ?
Intermediate & Advanced SEO | | suredone

0
Login required pages that redirect back to the post

Hi, Login required pages that redirect back to the post get the same description Example http://www.Somesite.com/modal/register?destination=post title Every post when a non member sees it has links that are as above post title is the only difference in the URL's So these Meta descriptions 1,000's of them are same. What can we do Thanks
Intermediate & Advanced SEO | | mtthompsons

0
Could a HTML <select>with large numbers of <option value="<url>">'s affect my organic rankings</option></select>

Hi there, I'm currently redesigning my website, and one particular pages lists hotels in New York. Some functionality I'm thinking of adding in is to let the user find hotels close to specific concert venues in New York. My current thinking is to provide the following select element on the page - selecting any one of the options will automatically redirect to my page for that concert venue. The purpose of this isn't to affect the organic traffic - I'm simply introducing this as a tool to help customers find the right hotel, but I certainly don't want it to have an adverse effect on my organic traffic. I'd love to know your thoughts on this. I must add that in certain cities, such as New York, there could be up to 450 different options in this select element. | <select onchange="location=options[selectedIndex].value;"> <option value="">Show convenient hotels for:</option> <option value="http://url1..">1492 New York</option> <option value="http://url2..">Abrons Arts Center</option> <option value="http://url3..">Ace of Clubs New York</option> <option value="http://url4..">Affairs Afloat</option> <option value="http://url5..">Affirmation Arts New York</option> <option value="http://url6..">Al Hirschfeld Theatre</option> <option value="http://url7..">Alice Tully Hall</option> .. .. ..</select> Many thanks Mike |
Intermediate & Advanced SEO | | mjk26

0
Duplicate Content/ Indexing Question

I have a real estate Wordpress site that uses an IDX provider to add real estate listings to my site. A new page is created as a new property comes to market and then the page is deleted when the property is sold. I like the functionality of the service but it creates a significant amount of 404's and I'm also concerned about duplicate content because anyone else using the same service here in Las Vegas will have 1000's of the exact same property pages that I do. Any thoughts on this and is there a way that I can have the search engines only index the core 20 pages of my site and ignore future property pages? Your advice is greatly appreciated. See link for example http://www.mylvcondosales.com/mandarin-las-vegas/
Intermediate & Advanced SEO | | AnthonyLasVegas

0
Seo advice / plan ? penalized ?

I built an ecommerce site for a client of mine just over 9 months ago now. To begin with the serps were great, everything was listed in the results but for some reason a few weeks in all the results vanished from google and now we're lucky to find anything. I've been as far as page 200 and havent found any results. Its been like this for a solid 8 months so i can only presume that the site has been penalised in some form. Searching for unique phrases from the site doesnt even return results. The website in question is = http://goo.gl/A6Gz2
keywords we're aiming for = coloured contact lenses, fashion contact lenses
Target = Google UK Now im not really an seo guy but regardless of this my client has hired me to see whats going on and correct it. I've been scratching my head thinking all sorts of things but none of which im certain about so i'm looking for someone to point me in the right direction before i do anything drastic. So to begin with here are some of my suspicions which i personally think are affecting ranking and possible penalisation. #1 - Too many links on the page
#2 - Possibly over optimised
#3 - Lack of content on the product and category pages
#4 - Lack of backlinks and links in general coming from other sites My main concern is the lack of links from other sites and the odd link coming from low quality sites. I've also just found out that my client has been using an automatic link submitter which i've always thought of as a big no no. Some of the sites these links have been submitted to have nothing to do with the keyword we are targetting and are sort of spammy sites containing all sorts of links. Im wondering if these poor quality links could have caused the site to be penalised, google may be seeing it as a spammy site due to this. Whats your opinions on the above, are my suspicions correct and can this be recovered ? My planned course of action is to be as follows: #1 - Re write the content currently on the site so that it is better written and include more keywords, especially long tails since i think these will help bring the serps up.
#2 - Write detailed category and product descriptions as well as making sure every page has some well written content with links and keywords.
#3 - Keep the above pages to one main subject / keyword so that google doesnt get confused.
#4 - Get some links on popular and relevant sites, the only problem here is the lack of fashion contact lens sites. Does anyone have any advice on how to find these or where i should be getting links placed. Are directories worth while ?
#5 - Get more involved in the social side ie facebook, twitter I will be building on the above over time, aswell as running google ads moderately for our chosen keywords. Is there anything i have missed, anything i shouldnt be doing. Please advise. Thanks.
Intermediate & Advanced SEO | | gfxpixeldesigns

0