Increase number of pages crawled
-
Only one page is being crawled, how do I increase the number to include most of our site?
-
Hi! I am guessing that you're referring to SEOmoz only crawling one page of your site? The list of possible reasons for that is at https://seomoz.zendesk.com/entries/409821-why-isn-t-my-site-being-crawled-you-re-not-crawling-all-my-pages. If none of those are applicable, go to the help hub at http://www.seomoz.org/help and contact our help team and we'll work on figuring it out with you.
-
make sure you are not unintentionally blocking pages in your robots.txt or meta robots tags and that the links on your site are search engine friendly links that the crawlers can follow.
submit the URL(s) to be crawled in google webmaster tools to tell the bots to come around again.
increase the crawlrate manually in WMT
-
Hi Lisa,
OSE will crawl enough pages of your site that are found/relevant at the time of the last crawl.
OSE is not up to date at any one time.
This FAQ shows that some data may be delayed up to 60 days, however in my experience listing new websites, this has been more like a couple of weeks.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How do I create a segment that shows me all pages using a certain keyword? But nothing that doesn't have that keyword?
There must be an easy answer to this, but I can't seem to find it. All I want to do is create a segment in Google Analytics that shows all pages and search strings with "orthopaedics" in the title, with pageviews, uniques etc. If I simply navigate to "All Pages" in Google Analytics and then click Advanced Filters and do an Include Page Contains "orthopaedics" it works just fine. (See attached Screen Shot) But when I try to recreate this as a segment, it pulls in all other pages the users visited before arriving on the orthopaedics page I want to include, which I don't want. I can manually exclude each URL I don't want, but this is tedious and I feel there must be a simpler method I'm just missing. At the end of the day, I'm trying to create a list of every page and dynamically created query string that includes the word "orthopaedics" to say doctor X, your orthopaedics section generated X views, and here's a list of the pages. Mm6YTKa
Reporting & Analytics | | Patrick_at_Nebraska_Medicine0 -
Drop in indexation but increase in organic traffic
We've had a puzzling drop in indexed pages on our ecommerce website. My crawl returns just over 25k items. Until 19/6 we had about 23-24k indexed. Then we experienced a sudden drop from 19/6 to 26/6: from 23,400 to 18,999, losing 4.4k pages from one week to the next. At the same time, our organic traffic has not decreased, it actually increased, however, it's only been a couple of weeks so that may be coincidence. A few things that have happened during the past few weeks: 31/5: we implemented pagination on category pages to avoid issues with duplicate content - could it be that this led to a decrease in indexed pages 3 weeks later? However, I can only find about 1.5k pages in my crawl that are page 2+ 18-19/6: we had some website outages over the weekend; as a B2B business, we don't get much traffic over the weekend, so I can't see an impact to traffic. However, the following week, indexation dropped by another 250 (then stayed the same this past week), so I don't think this was a factor. 21/6: we retired another website and migrated it to our main website. However, all pages were redirected to existing pages so no new pages were created for the migration. This doesn't really explain a decrease in indexation, but may account for some of the increase in organic traffic; however not all as the retired website hardly got any organic traffic. So, should we be worried? As our website is quite large, it would probably be quite difficult to pin point exactly which pages dropped off the index, but a loss of 19% of pages is quite significant. Then again, it doesn't appear to have negatively impacted organic traffic... Have you got any suggestions for what I should be looking at to find out what happened? Should I be worried at this point? I will definitely continue to have an eye on how our organic traffic (and indexation) develops but I am not sure if there is anything I can do at this point. I'd appreciate your advice on this, to make sure I am not missing something blindingly obvious. Thanks! RmWaNib JJm4tC3
Reporting & Analytics | | ViviCa10 -
Why am i getting a flux of increase in Impressions on my site & then it decreases
They guys. Hope everyone is having a great week. I wanted to get some inputs from you guys in regards to what is happening to my site that i quite don't understand. Every month or so i get this influx of high visibility with impressions for my keywords and then the impressions go away but my rankings still keep going up. Has anyone experienced this before and can give me some insight on what is going . Why do i get such a big jump and then it dies off only to return again a month later or 2 months later. I know you guys want probably some info from my site or from analytics or webmaster tools so i will provide as much as i can . For now i have included a screen shot. ScreenShot2013-06-04at31220PM_zps0d02f5fc.png ScreenShot2013-06-04at31134PM_zps5bb81b68.png ScreenShot2013-06-04at31134PM_zps5bb81b68.png ScreenShot2013-06-04at31220PM_zps0d02f5fc.png
Reporting & Analytics | | BizDetox0 -
Does anyone know of a way to do a profile level filter to exclude all traffic if it enters the site via certain landing pages?
Does anyone know of a way to do a profile level filter to exclude all traffic if it enters the site via certain landing pages? The problem I have is that we have several pages that are served to visitors of numerous other domains but are also served to visitors of our site. We end up with inflated Google Analytics numbers because people are viewing these pages from our partners' domains but never actually entering our site. I've made an advanced segment that serves the purpose but I'd really like to filter it at the profile level so the numbers across the board are more accurate without having to apply an advanced segment to every report. The advanced segment excludes visits that hit these pages as landing pages but includes visits where people have come from other pages on our domain. I know that you can do profile filters to exclude visits to pages or directories entirely but is there a way to filter them only if they are a landing pages? Any other creative thoughts? Thanks in advance!
Reporting & Analytics | | ATIseo0 -
I have few similar job forms that were created for different positions. SEOMoz says, its "duplicate pages". So how do I resolve it? I want my jobs to be searchable in Search Engines.
Hi There, I have few similar job forms that were created for different positions. SEOMoz says, its "duplicate pages". So how do I resolve it? I want my jobs to be searchable in Search Engines. Thanks !
Reporting & Analytics | | pointstar0 -
Why such a high page rank with so low metrics in OSE
Hi,
Reporting & Analytics | | sebastiankoch
The website is:
www.s123parka123ble.com (remove the 123) All websites I have ever seen with similar metrics in OSE have normally a PR of 2 or 3 or max 4. This one is PR 5 and I would like to know why. I also noticed that they do not redirect the urls without www. to www. Which is normally bad ... can it be good for some reason in this case? It is PR 5 since a long time I just can´t get why it is PR 5. Please have a look. Thanks!0 -
Magic UVs - PPC landing pages delivering organic traffic by magic...
I have checked and double checked this. GA is showing over the last couple of weeks mysite.com/ppc/landingpage1 as a landing page for organic traffic, where it shouldn't. Main facts: The entire /ppc/ folder is blocked from the googlebot, and doesn't appear on any internal site maps. As far as I can tell, these pages have never been cached for the main index. I cannot recreate any of the organic searches myself (i.e. typing in keywords that triggered the traffic, even the almost unique long-tail ones). We just don't appear in the organic listings with these pages. The analytics and adwords accounts are linked. We are not paying for this mystery traffic through our PPC - these keywords are not appearing in our AdWords account (though other keywords / traffic are). The traffic is real - we have received phone calls from these pages, tracked to the visits recorded as organic These pages should only receive PPC traffic. They are receiving organic traffic also, but I can't recreate it. Can anyone suggest what's going on? I'm concerned about duplicate content issues and also skewing the analysis of the PPC campaign. Thanks
Reporting & Analytics | | RobPell0 -
What does "on first page" mean in seomoz ranking reports?
Hi - When reports here show numbers of keywords appearing "on first page", there must be some implicit assumption made about the number of results listed per page. 1. Can anyone tell me what that assumption is? Is it 10? 20? 2. What about universal results Local links? If the answer to number one is, for instance, 20 results per page, then are there any assumptions made about the number of universal results Local links included? I'm just trying to understand what the reports mean. Thanks, Tim
Reporting & Analytics | | tcolling0