Welcome to the Q&A Forum

Browse the forum for helpful insights and fresh discussions about all things SEO.

Your browser does not seem to support JavaScript. As a result, your viewing experience will be diminished, and you have been placed in read-only mode.

Please download a browser that supports JavaScript, or enable it if it's disabled (i.e. NoScript).

How can I exclude display ads from robots.txt?

Technical SEO

817

GregBeddor last edited by

Google has stated that you can do this to get spiders to content only, and faster. Our IT guy is saying it's impossible.
Do you know how to exlude display ads from robots.txt?

Any help would be much appreciated.
1 Reply Last reply
Reply Quote 0
EricaMcGillivray last edited by

You'd want to make the URL paths where the display ads live to have the crawl disallowed in your robots.txt, just like any other section of your site. Here's some basics on robots.txt.

Hope this helps!
1 Reply Last reply
Reply Quote 1

Got a burning SEO question?

Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.

Start my free trial

Browse Questions

View

From

Sorted by

With category

Explore more categories

Related Questions

What is this SERP feature called? Organic ads...?

serp features

What are these called (screenshot attached)? They are organic product listings in the body of the SERP that look just like paid ads, but are placed below paid ads. I would like to read up on how to optimize for them, but am having trouble finding information.Capture.PNG
Technical SEO | | LivDetrick

0
Little confused regarding robots.txt

Hi there Mozzers! As a newbie, I have a question that what could happen if I write my robots.txt file like this... User-agent: * Allow: / Disallow: /abc-1/ Disallow: /bcd/ Disallow: /agd1/ User-agent: * Disallow: / Hope to hear from you...
Technical SEO | | DenorL

0
Easy Question: regarding no index meta tag vs robot.txt

This seems like a dumb question, but I'm not sure what the answer is. I have an ecommerce client who has a couple of subdirectories "gallery" and "blog". Neither directory gets a lot of traffic or really turns into much conversions, so I want to remove the pages so they don't drain my page rank from more important pages. Does this sound like a good idea? I was thinking of either disallowing the folders via robot.txt file or add a "no index" tag or 301redirect or delete them. Can you help me determine which is best. **DEINDEX: **As I understand it, the no index meta tag is going to allow the robots to still crawl the pages, but they won't be indexed. The supposed good news is that it still allows link juice to be passed through. This seems like a bad thing to me because I don't want to waste my link juice passing to these pages. The idea is to keep my page rank from being dilluted on these pages. Kind of similar question, if page rank is finite, does google still treat these pages as part of the site even if it's not indexing them? If I do deindex these pages, I think there are quite a few internal links to these pages. Even those these pages are deindexed, they still exist, so it's not as if the site would return a 404 right? ROBOTS.TXT As I understand it, this will keep the robots from crawling the page, so it won't be indexed and the link juice won't pass. I don't want to waste page rank which links to these pages, so is this a bad option? **301 redirect: **What if I just 301 redirect all these pages back to the homepage? Is this an easy answer? Part of the problem with this solution is that I'm not sure if it's permanent, but even more importantly is that currently 80% of the site is made up of blog and gallery pages and I think it would be strange to have the vast majority of the site 301 redirecting to the home page. What do you think? DELETE PAGES: Maybe I could just delete all the pages. This will keep the pages from taking link juice and will deindex, but I think there's quite a few internal links to these pages. How would you find all the internal links that point to these pages. There's hundreds of them.
Technical SEO | | Santaur

0
Robots.txt best practices & tips

Hey, I was wondering if someone could give me some advice on whether I should block the robots.txt file from the average user (not from googlebot, yandex, etc)? If so, how would I go about doing this? With .htaccess I'm guessing - but not an expert. What can people do with the information in the file? Maybe someone can give me some "best practices"? (I have a wordpress based website) Thanks in advance!
Technical SEO | | JonathanRolande

0
Can backlinks from advertising cause a traffic drop?

Hi, I recently noticed that our organic traffic has started to drop and maybe coincidently our adwords traffic has increased. I was asked to investigate the drop. I know that from the google update that unnatural backlinks would be penalized so I thought it might be the backlinks from a site that we advertise on because of the sheer number we have required from them in the last month. Would you think that would be the cause? if not, what could it be? and if it is, how do I go about correcting it as fast as possible? Any Help with this would be greatly appreciated. Many Thanks, Colin
Technical SEO | | digital.moretogether.com

0
Adding more content to an old site

We have a site which was de-moted from PR4 to PR3 with the latest Google update. We have not done any SEO for a long time for the site and the content is the same with over 100 page. My question is, in order to update the site, which is the best to do it, do we: 1. re-introduced new content to replace old once 2. re-write old content 3. Add new pages Many thanks in advance.
Technical SEO | | seomagnet

0
Using robots.txt to deal with duplicate content

I have 2 sites with duplicate content issues. One is a wordpress blog. The other is a store (Pinnacle Cart). I cannot edit the canonical tag on either site. In this case, should I use robots.txt to eliminate the duplicate content?
Technical SEO | | bhsiao

0
Browser Pop Ups - Can it be SEO Friendly and how?

Hi The designer of the company I work for is re-designing Pop Up browsers as well as inline Pop up and Drop down menus. He needs SEO requirements - how can they be SEO friendly? Thanks a lot for your help! SL. Please see below the detail: Browser Pop Ups all include:
• a browser title,
• a logo and title in the title bar,
• a close window button and
• a call to action (that closes pop-up when clicked). Usage:
Use when you'd like to offer additional information to the
user but, not take the user away from the main page. Inline Pop up and Drop down menus. The inline pop-up & drop down is used to display additional menu options, functionality
or content on the page without dedicating real estate in the page layout. It's a part of the page HTML to retain SEO value and thus does not trigger pop-up blockers. A title bar displays when content of the pop-up or drop down is not in context of
the trigger. When used as a drop down, it is attached to the the bottom of it's trigger and left-aligned (unless it would exceed beyond the browser chrome, then it's right-aligned). When used as a pop-up, it is centered vertically/horizontally in the users browser window.
The inline pop-up/drop down can be triggered differently per instance (e.g. onclick, onhover with delay). It can be closed by: clicking on link/location that triggered the pop-up/drop down (a.k.a. close icon) clicking anywhere outside the pop-up/drop down There are 5 widths to choose from, based on the needs of the content: 196px (3 columns) 266px (4 columns) 406px (6 columns) 546px (8 columns) 658px (10 columns)
Technical SEO | | charsimona

0