Block web archieve/way back machine
-
Hi i want to block web archive/wayback machine from indexing my site and creating a record of it on their database.
Any ideas on how to do this?
Cheers,
Superpak -
You can block Wayback Machine from crawling and creating a record of your site by adding the following to your Robots.txt file:
User-agent: ia_archiver
Disallow: /This will not only stop new records from being created but also stop people viewing what had previously been indexed by Wayback Machine.
More information about this can be found here: https://archive.org/about/exclude.php
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Is there a limit on back linking per week
I know the ratio matters of follow/no follow when it comes to backlinks/inbound but also if there is a high volume of them per week, it can be flagged as spammy. Is there a limit on how many backlinks can be done in a week for a domain? I am talking backlinks from high domain authority low spam links. Media/blog sites.
Intermediate & Advanced SEO | | SeobyKP0 -
Urgent: Any point having /au version of the website for Australia?
Hi, We just migrated our website from /uk to the global one (but we still kept /us). We are expanding our business to Australia. Is there any point having the global .com site duplicated as .com/au provided the content will be identical? What's the /au impact on the domain strength and rank in Australia in comparison to having just .com. Is there any point? Anyone has direct experience? What's the best practice? Many thanks for the answers. Katarina
Intermediate & Advanced SEO | | Katarina-Borovska1 -
Regret changing the URL structure, Would it be appropriate to change it back?
Hi Moz Community, We changed the URL structure 6 months ago for our new site, and we experienced a ranking drop since then. From my understanding, changing URL structure and using 301 redirects will lose link juice, more or less. We think the ranking drop is because of the loss of link juice, assuming other factors remain constant. Here are my questions: How do those link juice losses have an impact on our ranking? Would changing URL structure back to original version regain the lost link juice, with all the redirects done properly? Would it take a lot of efforts? Is it recommended to change it back? Thank you so much in advance. Any thoughts and opinions are appreciated! Best, Raymond
Intermediate & Advanced SEO | | raymondlii0 -
Language/Country Specific Pages All in English
Hi Folks, I have been checking how many pages our competitors have indexed in Google compared to our website and I noticed that one of our main competitors has over 2 million indexed pages and I have figured out that it is because they have language/country specific pages for every page on their website. That being said, these pages contain all of the same content and the language doesn't actually change, it remains in English. Now my question is this. Will this not in fact hurt their rankings, in terms of duplicate content? Or am I missing something here? The URL's essentially do something like www.competitor.com/fr/ for France for example but as I say the content is in English, and duplicates their main website. Seems odd to me but would love your opinions on this. Thanks. Gaz
Intermediate & Advanced SEO | | PurpleGriffon0 -
Old/wrong meta-titles in index
Hi, We have problems with old Meta titles in the index of google.nl. If you look for example at this wine: https://www.wijnvoordeel.nl/Italie/Just-Hugo::5460.html The Meta tile is: **Just Hugo | Heerlijke Hugo | Het zomerdrankje van 2014 | Wijnvoordeel ** If you look at the results in Google: https://www.google.nl/webhp?sourceid=chrome-instant&ion=1&espv=2&ie=UTF-8#safe=active&q=just hugo The Meta tile is: Just Hugo - Wijnvoordeel(this is an old/automatic generated Meta tile). I already added the code "", but I don't see any progress. Does anybody knows what could be the problem? Thanks for the help! Douwe Veldstra
Intermediate & Advanced SEO | | Eluscious.com0 -
Canonical and Rel=next/prev Implementation
Hi, I have an ecommerce site that allows users to view numerous pages and sort by a number of options on categories. I've read numerous posts around my issue but am still a little confused on what is best practice with regards to the canonical tag and rel=next and prev. Below is an example of the various page/sort by URL's: Paginated URL: http://www.example.co.uk/category/subcategory.html?p=3 Sort by URL: http://www.example.co.uk/category/subcategory.html?dir=desc&order=price Paginated & Sort by URL: http://www.example.co.uk/category/subcategory.html?dir=desc&order=price&p=3 It is not viable for us to use a canonical tag to the view all page as some of the categories contain a large number of products and therefore would not have the best load speeds. Is it best to use the below structure when it comes to the canonical tag and rel=next and prev? Paginated URL: http://www.example.co.uk/category/subcategory.html?p=3 Sort by URL: http://www.example.co.uk/category/subcategory.html?dir=desc&order=price Paginated & Sort by URL: http://www.example.co.uk/category/subcategory.html?dir=desc&order=price&p=3 http://www.example.co.uk/category/subcategory.html?dir=desc&order=price&p=2" /> Thanks
Intermediate & Advanced SEO | | GrappleAgency0 -
Scraping / Duplicate Content Question
Hi All, I understanding the way to protect content such as a feature rich article is to create authorship by linking to your Google+ account. My Question
Intermediate & Advanced SEO | | Mark_Ch
You have created a webpage that is informative but not worthy to be an article, hence no need create authorship in Google+
If a competitor comes along and steals this content word for word, something similar, creates their own Google+ page, can you be penalised? Is there any way to protect yourself without authorship and Google+? Regards Mark0 -
How Important is Domain Authority in Back-Link Audit
First off I just want to say thanks Penguin! Now I get to start the joyous experience doing a back-link audit, and removing all the negative links. Also I now have to be on constant alert for Black SEO tactics targeted at my domain due to the cut throat business I am in. I think it can only be a matter of time before Google says all backlinks do not matter. Unfortunately, I need rank now!! So I have a couple of questions: First how important is domain rank in a back-link audit? Should I remove myself from indexes with low domain rank, and leave ones with high? Should I remove myself from as many indexes as possible? What about obvious paid blog posts that have high domain rank? Do you leave those? What is considered a low Domain Rank for back-links, under 35 - 40? Second, what is a good success rate for a back link audit. How can you measure improvement, other than waiting for your PR or SERP to go up? Third, in some situations it looks like back-links are legitimate, but they all point to my home page. Is it worth pursuing for example asking these people to link to the specific product they are referring to for example children picnic tables instead of just our home page? And, lastly what legal rights do I have to get back-links removed? Is it only on sites that copy my content that I have copy written? Is it possible to prevent Google from counting these back-links through an .htaccess file? Thanks in advance for all of the help. I hope to take what I learn and put it into a guide of some capacity as I am sure many people are going through this same situation at the moment.
Intermediate & Advanced SEO | | fifthroommarkets0