Welcome to the Q&A Forum

Browse the forum for helpful insights and fresh discussions about all things SEO.

Your browser does not seem to support JavaScript. As a result, your viewing experience will be diminished, and you have been placed in read-only mode.

Please download a browser that supports JavaScript, or enable it if it's disabled (i.e. NoScript).

How can I exclude display ads from robots.txt?

Technical SEO

822

GregBeddor last edited by

Google has stated that you can do this to get spiders to content only, and faster. Our IT guy is saying it's impossible.
Do you know how to exlude display ads from robots.txt?

Any help would be much appreciated.
1 Reply Last reply
Reply Quote 0
EricaMcGillivray last edited by

You'd want to make the URL paths where the display ads live to have the crawl disallowed in your robots.txt, just like any other section of your site. Here's some basics on robots.txt.

Hope this helps!
1 Reply Last reply
Reply Quote 1

Got a burning SEO question?

Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.

Start my free trial

Browse Questions

View

From

Sorted by

With category

Explore more categories

Related Questions

How to use robots.txt to block areas on page?

Hi, Across the categories/product pages on out site there are archives/shipping info section and the texts are always the same. Would this be treated as duplicated content and harmful for seo? How can I alter robots.txt to tell google not to crawl those particular text Thanks for any advice!
Technical SEO | | LauraHT

0
Can hreflang replace canonicalisation ?

Hi Im working with a site that has ALOT of duplicate content and have recommended developer fix via correct use of Canonicalisation i.e the canonical tag. However a US version (of this UK site) is about to be developed on a subfolder (domain.com/uk/ & domain.com/US/ etc so also looking into adopting the hreflang attribute on these. Upon reading up about the hreflang attribute i see that it performs a degree of canonicalisation too. Does that mean that developing the international versions with hreflang means there's no need to apply canonicalistion tags to deal with the dupe content, since will deal with the original dupe content problems as well as the new country related dupe content, via the hreflang ? I also understand that hreflang and canonicalisation can conflict/clash on different language versions of international subfolders etc as per: http://www.youtube.com/watch?v=Igbrm1z_7Hk In this instance we are only looking at US/UK versions but very likely will want to expand into non english countries too in the future like France for example. So given both the above points if you are using hreflang is it advisable (or even best) to totally avoid the canonical tag ? I would be surprised if the answers yes, since whilst makes logical sense given the above (if the above statements are correct), that seems strange given how important and standard best practice canonical usage seems to be these days. What best ? Use the Hreflang alone, or the Canonical tag alone or both ? What does everyone else do in similar situation ? All Best Dan
Technical SEO | | Dan-Lawrence

0
How can i do SEO For Ecommerce site

I am doing SEO for my WP blog but now I am starting my recently launch an eCommerce site where I am selling electronics products. I want to know how can I do the SEO so at least I can top 10 position for my google India. Second how can i avoid duplicate content about copying manufacture contents. Please help
Technical SEO | | chandubaba

0
Site blocked by robots.txt and 301 redirected still in SERPs

I have a vanity URL domain that 301 redirects to my main site. That domain does have a robots.txt to disallow the entire site as well. However, for a branded enough search that vanity domain still shows up in SERPs and has the new Google message of: A description for this result is not available because of this site's robots.txt I get why the message is there - that's not my , my question is shouldn't a 301 redirect trump this domain showing in SERPs, ever? Client isn't happy about it showing at all. How can I get the vanity domain out of the SERPs? THANKS in advance!
Technical SEO | | VMLYRDiscoverability

0
How can I redirect old URLs to new ones?

We are trying to optimize our website for search engines and one of the things we have noticed that needs to be changed are the URLs. I know how to modify the URLs but have no idea how I can get the old redirected to the new ones using 301. We have some pages ranking pretty high on Google and we can't loose this traffic. Our website is hosted on Plesk and it was coded using Cold Fusion. I do have access to the .htaccess file. Any help would be highly appreciated.
Technical SEO | | personalproducts

0
How can I add an additional user

I want to give access to this account to an additional user. Is this possible?
Technical SEO | | MishconAdmin

0
Robots.txt Showing in SERP Results

Currently doing a technical audit for a website and when I search "Site:website.com -www" the only result is website.com/robots.txt I was wondering if anyone else has come across this before -- or what this may mean from a technical audit standpoint. Thank you!
Technical SEO | | vectormedia

0
How to block google robots from a subdomain

I have a subdomain that lets me preview the changes I put on my site. The live site URL is www.site.com, working preview version is www.site.edit.com The contents on both are almost identical I want to block the preview version (www.site.edit.com) from Google Robots, so that they don't penalize me for duplicated content. Is it the right way to do it: User-Agent: * Disallow: .edit.com/*
Technical SEO | | Alexey_mindvalley

0