Where does the crawler find the urls?
-
The SEO Moz crawler has found a number of 500 error pages, and 404s etc which is very useful
however some of the urls are weird/broken formats we don't recognise and nobody remembers ever using - not weird enough to imply hacking, but something broken in the CMS
Is there anyway to find out where the crawler found these urls? I can patch up and redirect the end result as best I can but I would prefer to fix plug the leak
thanks
-
If you export the crawl diagnostics to a CSV, we do have this information in the last column.
-
thanks for the tips. It is a little frustrating that the information I need has passed through seomoz's system but I guess they don't have the inclination or resources to show us the info
Xenu reckons it can handle 1m urls, we are in the position of not really knowing how many pages our site has!
-
You can pop the links into the free Xenu Link Sleuth* - after you've done a crawl just right-click on the URL you're interested in and click 'URL Properties' - you'll see any inlinks it finds listed there. Depending on the size of your site, it could take a while for the crawl to complete.
You could try the link: property in Google first, though it won't be as thorough as Xenu.
*If you haven't seen it before, don't worry about how the Xenu website looks - the software is kosher - as recommended by many SEOmoz staff. Screaming Frog is a paid alternative (with a limited free version).
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Facebook URLs, Anchor Text
I have a client that is considering a facebook url change. For ease of explanation, let's say their currently existing URL is facebook.com/Company123. I've googled their currently existing facebook url and found a dozen or so websites that include the text, "facebook.com/Company123". But, these results don't include websites that have an anchor text of, for example, "Facebook" and a link pointing to facebook.com/Company123. Has anybody had success tracking down any/all websites that point to a specific Facebook url? I've tried Open Site Explorer, OpenLinkprofiler, RankSignals, and SEO SpyGlass to no avail. Thank you!
Moz Pro | | OMTAnno0 -
Page Rank Report says #6 in Google but I can't find the page anywhere
So SEOMoz says that I've consistently ranked #6 for a certain keyword. But when I search I'm no where to be found. I've done regular searches, incognito and some non-seomoz reports and all come up with nothing in Google. I noticed it a week or two ago, but didn't think it would continue. This is no bueno. I wouldn't be surprised if I got penalized (luckily my homepage relatively well for similar keywords), an old seo consultant used very spammy tactics. I recently removed them, but not before I started to notice that I fell off the map. Why would SEOMoz not recognize this, and continue to say I'm ranking well? The keyword is bpi building analyst the page is http://www.cleanedison.com/courses/bpi-building-analyst
Moz Pro | | CleanEdisonInc0 -
Find New Keywords Tool - Information is not accurate
When using the Find New Keywords tool, the information listed for Rank is often incorrect. I attached an image with two screen shots. One from the Find New Keywords Tool and the other is a Google search results page. In Incognito mode, I searched for the third item on the list, "Speak Creative Memphis." The results put us at #1 for that term, but the Keyword Tool shows that we're #4. Can you help me understand why there is a difference? keywordtool-ss.jpg
Moz Pro | | speakcreative1 -
Canonical link on canonical url
This might seem a bit of an odd one, but we seem to be going around in circles on this when using the on page optimizer tool. We have an ecommerce site (magento) which by default is putting a canonical link in the header on every product page. For example; www.example.com/product1.html has the But when we run the on page optimiser tool, we're losing points on the critical section for not having canonical set correctly. If we remove the tag, we get the tick and the a grade, but then further down the report we lose a tick for not using canonical links. What are we missing here?
Moz Pro | | andyjsi0 -
Link buying: finding sites based on criteria
Hi. I'm looking to get links to my site from blogs/sites/pages that fit this criteria: niche: IT, electronics, product reviews Minimum PR: 2 Maximum 20 external links Is there an automated tool that can help me discover these site?
Moz Pro | | seo_marker0 -
Why are SEOmoz Pro Keyword Ranking reports different between two 301-linked URLs?
Hi, My main domain is www.dancenut.com. I have this 301 URL redirected to www.dancenut.com/boston/. I set up an SEOmoz Pro campaign for each of these URLs in order to see if they were being treated differently in any way. In most cases the report results are identical, or the small differences are understandable. However, there is one big difference between the two sets of campaign reports. In the Keyword Ranking reports, the data for the Bing and Yahoo! reports are identical, but the data for the Google reports are dramatically different. Out of 21 keywords, 9 are listed in the top 50 for www.dancenut.com, but only 2 are listed in the top 50 for www.dancenut.com/boston/ (the specific positions are the same for the 2 keywords that are listed for both). Does this make any sense? Could the SEOmoz Pro data be wrong? If not, then I'm suspicious that Google may not be interpreting the 301 redirect properly. I don't think this could be fully explained by a 1-10% reduction in link juice due to the 301 because I have one keyword for which my site ranks #1 in Google, Bing, and Yahoo! with www.dancenut.com, but it doesn't even rank in the top 50 in Google with www.dancenut.com/boston/. And why would these differences only exist for Google? Any insight would be much appreciated! Andrew
Moz Pro | | dancenut0 -
How to find highest PR pages by search term?
is there a search operator in google to sort by highest PR, or an alternate search engine (besides Google directory) to find high PR sites by keyword? I am sending free product out to webmasters to use as a giveaway for their audience, and would like to find the highest PR sites/pages to partner with. I want the highest SEO value out of the products I send out. I'm trying to find outdoor sports related sites, etc, which have done contests/giveaways in the past and linked well to the supplier of the prize. I've been just googling phrases like "backpack giveaway", etc. and this has worked ok, but I'd like to find a tool to search for the highest PR sites around each of the phrases.
Moz Pro | | rakesh_patel0