Why specify robots instead of googlebot for a Panda affected site?
-
Daniweb is the poster child for sites that have recovered from Panda. I know one strategy she mentioned was de-indexing all of her tagged content, fo rexample: http://www.daniweb.com/tags/database
Why do you think more Panda affected sites specifying 'googlebot' rather than 'robots' to capture traffic from Bing & Yahoo?
-
Hi there,
I'm not sure of the percentage of sites who have specified all bots as opposed to just Google, but I also have to assume that the percentage depends on the sites' territory. For instance, there are few sites in the UK who would think of Bing or Yahoo, due to those search engines having a tiny market share. In the US, Japan and several other places where non-Google search engines have more share, the consideration will be much higher.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Technical Site Migration
Hello All- I have been in the SEO industry for about 4 years and feel fairly confident in my technical SEO; however, I am being asked to conduct a migration that is both a platform change (step 1) then a consolidation change (think of combining websites when a company is acquired) all within a 4-5 month time span. I feel like as I begin to create this multi-step checklist I would love to hear of others that have gone through something similar or have resources you could point me to form more of a process/procedure standpoint. Again - I feel confident in the technical output but I have a fairly junior team and want to execute this smoothly. Feel free to add recommendations or ask for clarification if my discussion question doesn't make sense.
Intermediate & Advanced SEO | | Instructure0 -
Twitter Robots.TXT
Hello Moz World, So, I trying to wrap my head around all of the different robots.txt. I decided to dive into a site like Twitter, and look at their robot text. And now, I'm super confused. What are they telling the search engines with /hasttag/*src=. Why don't they just use: Useragent: * Disallow: But, they address each search engine. Is there any benefit to this? Thanks for all of the awesome responses!!! B/R Will H.
Intermediate & Advanced SEO | | MarketingChimp100 -
Mobile Googlebot vs Desktop Googlebot - GWT reports - Crawl errors
Hi Everyone, I have a very specific SEO question. I am doing a site audit and one of the crawl reports is showing tons of 404's for the "smartphone" bot and with very recent crawl dates. If our website is responsive, and we do not have a mobile version of the website I do not understand why the desktop report version has tons of 404's and yet the smartphone does not. I think I am not understanding something conceptually. I think it has something to do with this little message in the Mobile crawl report. "Errors that occurred only when your site was crawled by Googlebot (errors didn't appear for desktop)." If I understand correctly, the "smartphone" report will only show URL's that are not on the desktop report. Is this correct?
Intermediate & Advanced SEO | | Carla_Dawson0 -
How to make sure dev site is not index in wordpress and how would it be affected?
hi guys! I'm currently having a dev version of my site (dev.website.com) that once everything is done i would move the dev to the public domain (website.com). But since is a total duplicate content of my real site would it affect the seo? if so, i tried setting the reading privacy in wordpress so google would not index it but im afraid when i live it in the future and revert the setting back to normal it would affect the site seo. any opinion and suggestion on this?
Intermediate & Advanced SEO | | andrewwatson920 -
301 from one site to another
I have two e-commerce websites and i'm going to remove some products from website as requested by a supplier and sell them only on one site. Is it a good idea to 301 redirect the pages from site 1 to site 2?? Thanks for your help
Intermediate & Advanced SEO | | Aikijeff0 -
Recovering from a site migration
Hi. I've been working on http://www.alwayshobbies.com/ for a number of months. All was fine, but then we had a site migration which involved a huge number of redirects. There's been a couple of similar moves in the past. As a result, rankings have plummeted. To resolve this, we're considering letting all the old pages 404 by turning of the redirects, and removing all links to them where we can. Some key pages could have canonicals added, but basically we're looking to purge as much as possible. Does this sound like a reasonable tactic?
Intermediate & Advanced SEO | | neooptic0 -
Re-Direct Users But Don't Affect Googlebot
This is a fairly technical question... I have a site which has 4 subdomains, all targeting a specific language. The brand owners don't want German users to see the prices on the French sub domain and are forcing users into a re-direct to the relevant subddomain, based on their IP address. If a user comes from a different country, (ie the US) they are forced on the UK sub domain. The client is insistent on keeping control of who sees what (I know that's a debate in it's own right), but these re-directs we're implementing to make that happen, are really making it difficult to get all the subdomains indexed as I think googlebot is also getting re-directed and is failing to do it's job. Is there are a way of re-directing users, but not Googlebot?
Intermediate & Advanced SEO | | eventurerob0 -
Getting rid of a site in Google
Hi, I have two sites, lets call them site A and site B, both are sub domains of the same root domain. Because of a server config error, both got indexed by Google. Google reports millions of inbound links from Site B to Site A I want to get rid of Site B, because its duplicate content. First I tried to remove the site from webmaster tools, and blocking all content in the robots.txt for site B, this removed all content from the search results, but the links from site B to site A still stayed in place, and increased (even after 2 months) I also tried to change all the pages on Site B to 404 pages, but this did not work either I then removed the blocks, cleaned up the robots.txt and changed the server config on Site B so that everything redirects (301) to a landing page for Site B. But still the links in Webmaster Tools to site A from Site B is on the increase. What do you think is the best way to delete a site from google and to delete all the links it had to other sites so that there is NO history of this site? It seems that when you block it with robots.txt, the links and juice does not disappear, but only the blocked by robots.txt report on WMT increases Any suggestions?
Intermediate & Advanced SEO | | JacoRoux0