Mysterious drop in the Number of Pages Crawled
-
The # of crawled pages on my campaign dashboard has been 90 for months.
Approximate a week ago it dropped down to 25 crawled pages, and many links went with it.
I have checked with my web master, and he said no changes have been made which would cause this to happen.
I am looking for suggestions on how I can go about trouble shooting this issue, and possible solutions.
Thanks in advance!
-
shouldn't be hard to check the robots file.
go to www.yourdomain.com/robots.txt and tell us what that says. could be a server side issue as well even if they said "no changes.."
looking at the sitemap might help you figure out what's happening. like, which of the 65 pages are missing. Are they all pages located in the same sub-directory? etc.
-
I just ran my sitemap, and came up with 25 pages, which corresponds exactly with my Dash Board in SEOMOZ...I am still having trouble wrapping my head around why it has said 90 for months.
Still checking robots.txt
Thank you for the quick response!
-
did you have a sitemap on this site?
if so, run another one now and compare them to look for answers.
if not, run one anyway and see if that sheds any light on things.
any changes to the robots.txt file?
is this roger-bot's crawl you're referring to?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Does a number of products in anchor text (link to product list page) have any influence on SEO?
For example: shower cabins (660), used in our onpage-navigation which links to a product list page.
Technical SEO | | Maxaro.nl0 -
Unavoidable duplicate page
Hi, I have an issue where I need to duplicate content on a new site that I am launching. Visitors to the site need to think that product x is part of two different services. e.g. domain.com/service1/product-x domain.com/service2/product-x Re-writing content for product x for each service section is not an option but possibly I could get over that only one product-x page is indexed by search engines. What's the best way to do this? Any advice would be appreciated. Thanks, Stuart
Technical SEO | | Stuart260 -
50,000 pages or a page with parameters
I have a site with about 12k pages on a topic... each of these pages could use another several pages to go into deeper detail about the topic. So, I am wondering, for SEO purposes would it be better to have something like 50,000 new pages for each sub topic or have one page that I would pass parameters to and the page would be built on the fly in code behind. The drawback to the one page with parameters is that the URL would be static but the effort to implement would be minimal. I am also not sure how google would index a single page with parameters. The drawback to the 50k pages model is the dev effort and possibly committed some faux pas by unleashing so many links to my internal pages. I might also have to mix aspx with html because my project can't be that large. Anyone here ever have this sort of choice to make? Is there a third way I am not considering?
Technical SEO | | Banknotes0 -
Rankings drop after Panda
Hi All, My site dropped completely out of the SERPS on September 27th. I've tried everything I know to do (re-wrote all content, disavow links tool, filed DCMA complaints, de-optimized on-page content, made anchor text less aggressive, etc). Can you all please take a look at www.doctorloanusa.com and let me know what you think the problem is and how much you'd charge to help? Keywords used to be: doctor loans, physician loans. I ranked 2 or 3 for those keywords consistently for over 4 years. I know I need more content, but I feel like it's a waste of time creating it. If a thin site was the issue, wouldn't I at least rank SOMEWHERE in the 1000 results? Thanks for your consideration. At my wits end.
Technical SEO | | Cary_Forest0 -
Top pages give " page not found"
A lot of my top pages point to images in a gallery on my site. When I click on the url under the name of the jpg file I get an error page not found. For instance this link: http://www.fastingfotografie.nl/architectuur-landschap/single-gallery/10162327 Is this a problem? Thanks. Thomas. JkLej.png
Technical SEO | | thomasfasting0 -
SEOMoz Crawl Diagnostic indicates duplicate page content for home page?
My first SEOMoz Crawl Diagnostic report for my website indicates duplicate page content for my home page. It lists the home page URL Page Title and URL twice. How do I go about diagnosing this? Is the problem related to the following code that is in my .htaccess file? (The purpose of the code was to redirect any non "www" backlink referrals to the "www" version of the domain.) RewriteCond %{HTTP_HOST} ^whatever.com [NC]
Technical SEO | | Linesides
RewriteRule ^(.*)$ http://www.whatever.com/$1 [L,R=301] Should I get rid of the "http" reference in the second line? Related to this is a notice in the "Crawl Notices Found" -- "301 Permanent redirect" which shows my home page title as "http://whatever.com" and shows the redirect address as http://http://www.whatever.com/ I'm guessing this problem is again related to the redirect code I'm using. Also... The report indicates duplicate content for those links that have different parameters added to the URL i.e. http://www.whatever.com?marker=Blah Blah&markerzoom=13 If I set up a canonical reference for the page, will this fix this? Thank you.0 -
404 - page authority?
If in open site explorer my 404 pages have a higer page authority - what benefit would i see in rankings if I 301 redirected those pages to the right page. For example www.site.com/widget is a 404 but has authority according to open site explorer - but the page i see in the serps is www.site.com/widget/ with the / at the end. so what benefit would i see in rankings if I 301 redirected those pages to the right page?
Technical SEO | | DavidS-2820610 -
Can Search engines crawl this page
Hi guys, To put a long story short we have had to make a copy of our site and put it on another domain so in essence there are 2 copies of our site on the web.What we have done is put a username and password on the homepage - http://www.ughhwiki.co.uk/ now i just want to be 100% sure that the search engines cannot crawl this? Thank you Jon
Technical SEO | | imrubbish0