Why do we have so many pages scanned by bots (over 250,000) and our biggest competitors have about 70,000? Seems like something is very wrong.
-
We are trying to figure out why last year we had a huge (80%) and sudden (within two days) drop in our google searches. The only "outlier" in our site that we can find is a huge number of pages reported in MOZ as scanned by search engines. Is this a problem? How did we get so many pages reported? What can we do to bring the number of searched pages back to a "normal" level?
BT
-
Hi. A mystery indeed! Have you recently upgraded or changed Web platforms or changed or upgraded what you are using for your site navigation?
-
Stewart_SEO
Thanks for your quick response. We did review the robots.txt of the competitors. Not line by line - they took surprisingly different approaches to the robots.txt. But there were the usual exclusions for wish lists, etc. We've gone back and tightened up our robots.txt and haven't yet seen any changes. Several months ago we were at about 600,000 pages and it is dropping. Very mysterious.
-
Have you looked at your competitors robots.txt file? they are probably blocking the very same searches you are talking about. if there is a particular bot like a Chinese crawler for example baidu that you don't want to come to your site you can block them via the command: User-agent: Baiduspider
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Have AMP pages on Chrome disappeared?
Hi, We amplified our site the other day and Google was quick to find our AMP pages. Today, we can´t seem to find any AMP pages on Chrome and it looks like we can´t find any other pages from sites like Wapo and so forth. They´re still visible on Safari. We are searching from Iceland. Is anyone experiencing this and if so, could you please share a link to the news? Thanks and have a nice day!
Algorithm Updates | | guidetoiceland0 -
Can we ignore "broken links" without redirecting to "new pages"?
Let's say we have reaplced www.website.com/page1 with www.website.com/page2. Do we need to redirect page1 to page2 even page1 doesn't have any back-links? If it's not a replacement, can we ignore a "lost page"? Many websites loose hundreds of pages periodically. What's Google's stand on this. If a website has replaced or lost hundreds of links without reclaiming old links by redirection, will that hurts?
Algorithm Updates | | vtmoz0 -
Sitemaps for landing pages
Good morning MOZ Community, We've been doing some re-vamping recently on our primary sitemap, and it's currently being reindexed by the search engines. We have also been developing landing pages, both for SEO and SEM. Specifically for SEO, the pages are focused on specific, long-tail search terms for a number of our niche areas of focus. Should I, or do I need to be considering a separate sitemap for these? Everything I have read about sitemaps simply indicates that if a site has over 50 thousand pages or so, then you need to split a sitemap. Do I need to worry about a sitemap for landing pages? Or simply add them to our primary sitemap? Thanks in advance for your insights and advice.
Algorithm Updates | | bwaller0 -
Do we take a SEO hit for having multiple URLs on an infinite scroll page vs a site with many pages/URLs. If we do take a hit, quantify the hit we would suffer.
We are redesigning a preschool website which has over 100 pages. We are looking at 2 options and want to make sure we meet the best user experience and SEO. Option 1 is to condense the site into perhaps 10 pages and window shade the content. For instance, on the curriculum page there would be an overview and each age group program would open via window shade. Option 2 is to have an overview and then each age program links to its own page. Do we lose out on SEO if there are not unique URLS? Or is there a way using metatags or other programming to have the same effect?
Algorithm Updates | | jgodwin0 -
Why do I have 7 URLs from the same domain ranking on the 1st page?
I have a client that has individual pages for authorized dealers of their product (say "Car Dealers"). When you search for "brand name + location", Google returns 7 "dealership" pages from the parent company's domain as the first 7 results, but there is one that gets pushed off to the 5th page of the SERPs. The formatting of content, geo-targeting, and meta data on the page is identical on every single one. None of them have external links and there is not one extremely distinguishable thing to assess why the one page doesn't get placed on that first SERP. Why is the one getting pushed so far down? I know this may be a bit confusing, but any thoughts would be greatly appreciated. Thanks!
Algorithm Updates | | MichaelWeisbaum0 -
Will too many [img no alt-text] links harm a link profile?
Hi everyone. I have a client who has a lot of sponsorships etc and therefore a lot of inbound image links (many of them sitewide). Unfortunately most of these don't have alt text, and [img no alt-text] links now make up over 50% of their link profile. Should I be trying to correct this and requesting updates from the people who are linking? Obviously I wouldn't want loads of keyword stuffed alt texts, but maybe I should request alt texts on the brand name or URL instead. Do you think this would make a significant difference and be worth the time it will take to contact all these webmasters? Thanks in advance.
Algorithm Updates | | QubaSEO0 -
Site name appended to page title in google search
Hi there, I have a strange problem concerning how the search results for my site appears in Google. The site is Texaspoker.dk and for some strange reason that name is appended at the end of the page title when I search for it in Google. The site name is not added to the page titles on the site. If I search in Google.dk (the relevant search engine for the country I am targeting) for "Unibet Fast Poker" I get the following page title displayed in the search results: Unibet Fast Poker starter i dag - få €10 og prøv ... - Texaspoker.dk If you visit the actual page you can see that there is no site name added to the page title: http://www.texaspoker.dk/unibet-fast-poker It looks like it is only being appended to the pages that contains rich snippets markup and not he forum threads where the rich snippets for some reason doesn't work. If I do a search for "Afstemning: Foretrukne TOPS Events" the title appears as it should without the site name being added: Afstemning: Foretrukne TOPS Events Anybody have any experience regarding this or an idea to why this is happening? Maybe the rich snippets are automatically pulling the publisher name from my Google+ account... edited: It doesn't seem to have anything to do with rich snippets, if I search for "Billeder og stuff v.2" the site name is also appended and if I search for "bedste poker bonus" the site name is not.
Algorithm Updates | | MPO0 -
How did my Page Authority and Page Rank disappear?
I've hit a problem. A couple days ago my site's page authority was 51 and the PR was 3 and now they're 1 and 0 respectively. The developer did adjust some of the code in the site in the past couple days but that shouldn't have affected this. It was last cached by Google on the 5th. Can anyone offer some good advice? If it helps the page is www.duracard.com
Algorithm Updates | | Andrea.G0