How can I exclude display ads from robots.txt?
-
Google has stated that you can do this to get spiders to content only, and faster. Our IT guy is saying it's impossible.
Do you know how to exlude display ads from robots.txt?Any help would be much appreciated.
-
You'd want to make the URL paths where the display ads live to have the crawl disallowed in your robots.txt, just like any other section of your site. Here's some basics on robots.txt.
Hope this helps!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Blocking subdomains with Robots.txt file
We noticed that Google is indexing our pre-production site ibweb.prod.interstatebatteries.com in addition to indexing our main site interstatebatteries.com. Can you all help shed some light on the proper way to no-index our pre-prod site without impacting our live site?
Technical SEO | | paulwatley0 -
Adding your sitemap to robots.txt
Hi everyone, Best practice question: When adding your sitemap to your robots.txt file, do you add the whole sitemap at once or do you add different subcategories (products, posts, categories,..) separately? I'm very curious to hear your thoughts!
Technical SEO | | WeAreDigital_BE0 -
My SERP meta description is displaying 315 characters...
Hi Mozzers, We have recently taken the #2 spot for our main keyword in Google UK serp. I just checked again and we have dropped to #4 and our meta description is no longer there as it has been replaced with some homepage content... 315 characters of homepage content right up to the full stop. I'm a little confused. A couple of our competitors meta descriptions are showing the same, extra long homepage text instead. Is there something totally normal and harmless causing this or do I need to be monitoring/changing something? Has Google made an update to allow for longer meta decs? Any advice appreciated! sWrBcuB.png
Technical SEO | | SanjidaKazi0 -
Can you do a 301 redirect without a hosting account?
Trying to retire domain1 and 301 it to domain2 - just don't want to get stuck having to pay the old hosting provider simply to serve a .htaccess file with the redirect rule.
Technical SEO | | TitanDigital0 -
How can I redirect old URLs to new ones?
We are trying to optimize our website for search engines and one of the things we have noticed that needs to be changed are the URLs. I know how to modify the URLs but have no idea how I can get the old redirected to the new ones using 301. We have some pages ranking pretty high on Google and we can't loose this traffic. Our website is hosted on Plesk and it was coded using Cold Fusion. I do have access to the .htaccess file. Any help would be highly appreciated.
Technical SEO | | personalproducts0 -
Subdomain mozTrust - does other parkd domains can affect that ?
Hi , I have my domain www.mydomain.com and it have dpmain authority 26 , domain mozRank around 3 , domain mozTrust 1.63 , page authority 31, Google PR 2.0 etc etc So I am not in very bottom of scores, but my SUBDOMAIN MOZTRUST is only 0.961 and I've checked other websites that I've made some time ago and they have it like 4.0. So it is quite bad. I am having some domain parked within my hosting package. they have different names like www.mydomain2.co.uk , www.mydomain3.com etc. I can acces those domains as wel by typing : mydomain2.mydomain.com mydomain3**.mydomain.com** and have some testing subdomains there as well (just if I need to test something like drupal , wordpress or testing shoping cart etc.) Can that fact affect my subdomain rank ? Because I am having those domains parked there or I've made some subdomains that are not in use and nobody is linking to them and they are visible in Google ?
Technical SEO | | sever3d0 -
Use of Robots.txt file on a job site
We are performing SEO on a large niche Job Board. My question revolves around the thought of no following all the actual job postings from their clients as they only last for 30 to 60 days. Anybody have any idea on the best way to handle this?
Technical SEO | | WebTalent0 -
Blocking other engines in robots.txt
If your primary target of business is not in China is their any benefit to blocking Chinese search robots in robots.txt?
Technical SEO | | Romancing0