Why do we have so many pages scanned by bots (over 250,000) and our biggest competitors have about 70,000? Seems like something is very wrong.
-
We are trying to figure out why last year we had a huge (80%) and sudden (within two days) drop in our google searches. The only "outlier" in our site that we can find is a huge number of pages reported in MOZ as scanned by search engines. Is this a problem? How did we get so many pages reported? What can we do to bring the number of searched pages back to a "normal" level?
BT
-
Hi. A mystery indeed! Have you recently upgraded or changed Web platforms or changed or upgraded what you are using for your site navigation?
-
Stewart_SEO
Thanks for your quick response. We did review the robots.txt of the competitors. Not line by line - they took surprisingly different approaches to the robots.txt. But there were the usual exclusions for wish lists, etc. We've gone back and tightened up our robots.txt and haven't yet seen any changes. Several months ago we were at about 600,000 pages and it is dropping. Very mysterious.
-
Have you looked at your competitors robots.txt file? they are probably blocking the very same searches you are talking about. if there is a particular bot like a Chinese crawler for example baidu that you don't want to come to your site you can block them via the command: User-agent: Baiduspider
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Key webpage fluctuating between page 2 and page 6 of Google SERP
Hi, We have found that one of our key webpages has been fluctuating between page 2 and page 6 of Google SERP for around 2 weeks. Some days it will be on page 6 in the morning and then page 2 in the afternoon. We have recently updated some copy on the page and wondered if this could be the cause. Has anyone else experienced this? If so how long was it before the page settled? https://www.mrisoftware.com/uk/products/property-management-software/ Thanks.
Algorithm Updates | | nfrank0 -
Anything wrong with multiple meta descriptions and multiple title tags? We have 2 by mistake
Hi, As I stated in the we have 2 meta description and title tags. Will this hurts? How Google handles this? Thanks
Algorithm Updates | | vtmoz0 -
Latest Best Practices for Single Page Applications
What are the latest best practices for SPA (single page application) experiences? Google is obviously crawling Javascript now, but is there any data to support that they crawl it as effectively as they do static content? Considering Bing (and Yahoo) as well as social (FB, Pinterest, etc) - what is the best practice that will cater to the lowest-common denominator bots and work across the board? Is a prerender solution still the advised route? Escaped fragments with snapshots at the expanded URLs, with SEO-friendly URL rewrites?
Algorithm Updates | | edmundsseo2 -
Links to category pages unnatural?
If people are linking to your site, it would seem natural that the vast majority of those links would point to the homepage, product page, or a article/content page. Let's say you have 100 links pointing to your site, and 40 of them are pointing to category pages. Would this seem unnatural? Does Google or other search engines have a way of determining this as a factor in ascertaining whether the links are natural or not? Is there a rule of thumb when it comes to the pages that are linked to on your site?
Algorithm Updates | | inhouseseo0 -
Duplicate pages in language versions, noindex in sitemap and canonical URLs in sitemap?
Hi SEO experts! We are currently in the midst of reducing our amount of duplicate titles in order to optimize our SEO efforts. A lot of the "duplicate titles" come from having several language versions of our site. Therefore, I am wondering: 1. If we start using "" to make Google (and others) aware of alternative language versions of a given site/URL, how big a problem will "duplicate titles" then be across our domains/site versions? 2. Is it a problem that we in our sitemap include (many) URL's to pages that are marked with noindex? 3. Are there any problems with having a sitemap that includes pages that includes canonical URL's to other pages? Thanks in advance!
Algorithm Updates | | TradingFloor.com0 -
How do I figure out what's wrong with my site?
I'm fairly new to SEO and can't pinpoint what's wrong with my site...I feel so lost. I am working on revamping www.RiverValleyGroup.com and can't figure out why it's not ranking for keywords. These keywords include 'Louisville homes', 'Homes for sale in Louisville KY', etc. Any suggestions? I write new blog posts everyday so I feel there's no shortage of fresh content. I'm signed up with Moz Analytics and Google analytics
Algorithm Updates | | gohawks77900 -
How come google image search doesn't link to the right page?
For one site I work with the images link to the home page of the site rather than the page the image lives on. I think this is hurting my bounce rate quite a bit. Thoughts?
Algorithm Updates | | NetvantageMarketing0 -
How do these people have so many external back links?
A few of the sites that I 'compete' with have blogger accounts...and according to the research I've done they have over 500 million external, followed, links! These are mom blogs....and they aren't trying to do any SEO. I've noticed that if you are using blogger (as opposed to wordpress) and adding embed youtube videos to your blog you can set it to auto populate to youtube and also get a link in return. From what I can gather this just happens with blogger accounts since bloggers...and youtube...and google are all related. Does that have something to do with the crazy amount of followed external links they are getting? They also have a domain authority of 96 (which is because blogger is a trusted domain.... Is the sub-domain a better number to look at since their blog is a sub domain on blogger? Also, does anyone know who to get the same sort of exposure in youtube as blogger blogs get...or if it's even possible? Thanks.
Algorithm Updates | | NoahsDad0