Crawl test. Bot crawled only 200 or so links when it should have crawled thousands
-
Hi everyone,
I just recieved my crawl test report and its only given me 200 or so URL's when my site has thousands, any thoughts?
-
Hi Ryan,
I am the site owner and this is the precise reason im trying to take matters into my own hands.
<meta name="keywords" content="E60,Rear,lamp,set" /> I see what you mean, because this is actually ridiculous, not quite sure how it got into this state either. Whats that saying, "if you want something done you have to do it yourself" Looks like i have to take a crash course in SEO to sort it all out. Thanks very much for all your help.
-
I realize you may not have full control over the site. What I would share is:
"That's how the site is" is not an acceptable response, unless the site owner is satisfied with their current SEO ranking.
The keywords have NOTHING to do with the product being displayed on the page. The link I offered is for a Hella e60 Rear Lamp. The only related in the keyword section is "rear". I am quite certain that is by coincidence.
Your keywords are not dynamically generated to vary with the pages content, nor were they manually altered to fit the pages content. The keyword selection is awful. The numbers "3", "5", and "7" are listed as 3 of the key words.
I want to help you, so don't take this the wrong way. The best thing about that site is it probably qualifies as a textbook case of what NOT to do from a SEO perspective. Perhaps you can appeal to a SEO company to use the site in a case study and turn it around.
-
Thank you very much Ryan, the columns on two sides of the page cant be helped as thats, how the site is, only the central content changes. However the duplicate keywords are for the products themselves, for example i sell 50 different BMW oil filters. Theres not much i can do about duplicating keywords as all of the products are very very similar.
I think you might be right about the site redesign....
-
A few notes about your site:
-
you are using meta keyords in your header. It offers no benefit and I would suggest removing it. It's not related to your inquiry but is something I noticed.
-
your site has a 50 keyword TAG block with the same keywords on every page. This isn't good from a SEO perspective on many levels. You want your keywords to focus the unique content on a given page
-you site pages are likely viewed as all duplicates. I can recognize the main item in the center of the page changes, but would a crawler? Your left and right sidebars are identical on all pages, along with most of your header. The actual content you offer is only a small percentage of the total page.
The large image of the various car parts is not considered as part of the content, aside from the ALT tag.
Look at a random page from your site: http://www.incarmotorfactors.co.uk/content/16-hella-bmw-e60-rear-lamp-set
According to the Analyze tool there are 5975 words on the page. I estimate about 100 of them are unique words addressing your Rear Lamp product, and the remaining 5800+ words are exactly the same as every other product page.
A crawler will see your pages as 98% duplicated data and the result will likely be your site isn't going to be listed. I would recommend a site re-design. Before taking that advice, it would probably be best to hear from others who have a lot more SEO expertise then myself.
-
-
-
What is different about your site? Is it flash or javascript based? Can you share your site URL?
-
Hi Ryan,
I used
On-Page Optimization Tools: Crawl Test. However this problem may be deeper than i first thought, as SEOMoz is not able to read any of my site info properly.
Open site explorer cant read it
Linkscape cant read the links. Crawl test isnt read properly, however my server and robots.txt files are fine, theres no blocking attempts from the server. Very strange.
-
What Crawl Test tool did you use?
Depending on the crawl tool, it may not look at content blocked from your Robots.txt file. You may want to ensure it is configured correctly.
Are there any permission issues? The crawler will look at your site the way a guest would. Any content which requires users to log in would be hidden to the crawler.
Are there any other issues regarding your site's accessibility? Connection or firewall issues? Could a server admin have seen a server performance issue and kicked the crawler before it finished? You can check server logs for this information.
If you check everything and do not locate a definitive cause, I would suggest running the crawl once more and checking the results before pursuing the matter further.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How to remove broken links from our wordpress site?
Hello! How are you? We just signed up to Moz.com. Moz link tool. It gave us many broken links with 404's and 302's. Could you please help me with deleting the links? Thanks!
Moz Pro | | hsma0 -
Link Analysis-Moz analytics
Good afternoon, I relay hope someone can help me out with this report In link analysis I have viewed inbound links where I see many links linking to our site but among them I see also many times our site listed with various anchor text. How is this possible. Our site is linking to our site or what? It realy does not make sense to me. Can somebody explain me this please?
Moz Pro | | Rebeca10 -
No follow links also been reported in SEOmoz crawl diagnostics
Hi, Why does SEOmoz reports links which has been marked as 'nofollow'. I am getting 'Overly-Dynamic URL' reports on links which I have designated as nofollow which means Google will discount them. So why does SEOmoz still report them. Thanks.
Moz Pro | | malpani0 -
Competitive Link Analysis Tool?
Hi, I ran a competitive link analysis report today and back came quite a few domains that 2 or more of my 5 main competitors link from. Is it worth me submitting links to these sites? And would i be best served submitting my homepage URL or submitting a brand page such as Creative Recreation Trainers? I want to target that brand but don't want to do it if my main URL is better? Any ideas? See below my report. | Subdomain | Subdomain mR | Subdomain mT | # Competitors | # Linking Pages | Link Acquired |
Moz Pro | | YNWA
| t.co/ | 8.05 | 8.04 | 2 | <a>2</a> | |
| ww2.cox.com/ | 5.99 | 6.50 | 2 | <a>3</a> | |
| www.littlewebdirectory.com/ | 5.90 | 5.59 | 2 | <a>2</a> | |
| www.amazines.com/ | 5.69 | 5.66 | 2 | <a>3</a> | |
| svpply.com/ | 5.66 | 5.53 | 3 | <a>20</a> | |
| www.jayde.com/ | 5.64 | 5.68 | 3 | <a>4</a> | |
| www.pearltrees.com/ | 5.58 | 5.81 | 2 | <a>2</a> | |
| www.businessseek.biz/ | 5.52 | 5.51 | 2 | <a>3</a> | |
| www.a1articles.com/ | 5.50 | 5.22 | 3 | <a>9</a> | |
| www.linksilo.de/ | 5.48 | 5.23 | 2 | <a>15</a> | |
| www.alistsites.com/ | 5.46 | 5.24 | 2 | <a>38</a> | |
| www.the-free-directory.co.uk/ | 5.37 | 5.07 | 2 | <a>20</a> | |
| www.walhello.com/ | 5.30 | 4.97 | 2 | <a>2</a> | |
| www.quarkbase.com/ | 5.14 | 5.12 | 2 | <a>2</a> | |
| snipsly.com/ | 5.13 | 5.20 | 2 | <a>21</a> | |
| www.counterdeal.com/ | 5.12 | 5.07 | 2 | <a>2</a> | |
| www.01webdirectory.com/ | 5.03 | 5.03 | 2 | <a>2</a> | |
| www.2addlink.info/ | 4.92 | 4.58 | 3 | <a>4</a> | |
| www.fuk.co.uk/ | 4.64 | 5.00 | 3 | <a>20</a> | |
| www.final-fantasy.us/ | 4.63 | 4.77 | 2 | <a>2</a> | |
| oyax.com/ | 4.42 | 4.61 | 2 | <a>4</a> | |
| www.touchretail.co.uk/ | 4.33 | 4.21 | 2 | <a>4</a> | |
| tptbtv.cold10.com/ | 4.27 | 4.86 | 3 | <a>1</a> | |
| www.mastbusiness.com/ | 4.23 | 4.34 | 2 | <a>2</a> | |
| www.competitionhunter.com/ | 4.16 | 4.21 | 2 | <a>6</a> | |0 -
Issue in number of pages crawled
i wanted to figure out how our friend Roger Bot works. On the first crawl of one of my large sites, the number of pages crawled stopped at 10000 (due to the restriction on the pro account). However after a few weeks, the number of pages crawled went down to about 5500. This number seemed to be a more accurate count of the pages on our site. Today, it seems that Roger Bot has completed another crawl and the number is up to 10000 again. I know there has been no downtime on our site, and the items that we fixed on our site did not reduce or increase the number of pages we had. Just making sure there are no known issues with Roger Bot before I look deeper into our site to see if there is an issue. Thanks!
Moz Pro | | cchhita0 -
Links listed in MozPro Crawl Diagnostics
Ok, seeing as I'm getting to the end of my first week as a Pro Member, I'm getting more and more feedback regarding the pages on my site. I'm slightly concerned though that, having logged in this morning, I'm being shown 407 warnings for pages with 'Too Many On Page Links.' According to the blurb at the top of the page, 'Too Many' is generally defined as being over 100 links on a page ... but when I look at the pages which are being thrown up in the report, none of them contain anywhere near 100 links. I seriously doubt there is a glitch with the tool which has led me to think that maybe there's an issue with the way my site is coded. Is anyone aware of a coding problem that may lead Google and SEOMoz to suspect that I have a load of links across my site? P.S. As an aside, when this tool mentions 'Too Many Links' is it referring purely to OBL or does it count links to elsewhere on my domain too? Cheers,
Moz Pro | | theshortstack0 -
Can you change crawl day of week?
Can I somehow sync the day of the week for each of my campaigns' crawls, so that all campaigns are updated on the same day?
Moz Pro | | ATShock0 -
Seo moz crawl is not updating
When I check our seo campaign I can see that the report was not updated. It still show that the next crawl is Nov 1 but it is already Nov 3.
Moz Pro | | shebinhassan0