Screaming frog, Xenu, Moz giving wrong results
-
Hello guys and gals,
This is a very odd one, I've a client's website and most of the crawlers I'm using are giving me weird/ wrong results.
For now lets focus on screaming frog, when I crawl the site it will list e.g. meta titles as missing (not all of them though), however going into the site the title is not missing, and Google seems to be indexing the site fine. The robots.txt are not affecting the site (I've also tried changing the user agent).
The other odd thing is SF gives a 200 code but as a status tells me "connection refused" even though it's giving me data.
I'm unable to share the clients site, has any one else seen this very odd issue? And solutions for it?
Many thanks in advanced for any help,
-
Thanks Dan!
-
^ That's customer service!
-
Cheers for the tweet Andy.
Chris, I'd recommend running through this list here in our FAQ -
http://www.screamingfrog.co.uk/seo-spider/faq/#17
In particular, I'd recommend reducing the crawl speed & adjusting the user-agent.
You can also pop it through to our support (http://www.screamingfrog.co.uk/seo-spider/support/) & we can advise. All confidential etc.
Cheers
Dan
-
I have shared this question with Dan from Screaming Frog for you
-Andy
-
Hi Chris
This is hard to answer without an example - do you have a URL or two that you can share with the community?
Thanks so much!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Moz Crawl Report more urls?
Hi. I have used Moz Crawl Test and get my 3,000 urls crawled no issue. However, my site has more than that, is it possible to crawl the entire website? Alot of the crawl urls in the Moz test are search string urls and filters so Ive probably wasted about 2,500 urls on filter urls. Any advise or alternative software that wont cost a fortune?
Moz Pro | | YNWA
Thanks0 -
Moz shows an F grade and 0 keywords on a previously high ranking page - Please help
Hi all, Moz is showing an F grade and 0 keywords across all URL, Title, Meta, H1, Meta and IMG ALT. My page has also dropped from 1st page rankings for numerous pages to 2nd and 3rd page. The page I am talking about is: http://year13.com.au/10-highest-paying-jobs-you-can-get-without-a-degree/ The keywords/phrases we were previously ranking very well for are: High paying jobs in Australia High paying jobs Highest paying jobs you can get with no degree Etc Etc I have also seen reduction in rankings on other keywords/ pages as well. Does anyone know why this might have happened? And what might be happening? Thanks in advance!
Moz Pro | | Stubs0 -
Keyword Stuffing - MOZ On-Page Grader
We sell a great number of insulation products, many of which are produced by individual manufacturers. On the page identified below the Keyword "Kingspan" is repeated numerous times as these items are included in our online shop. However, the many mentions of Kingspan are recorded in the HTML5 Source Code, rather than an external database. When I used the MOZ On-Page Grader, using the keywords "Kingspan" I was surprised to achieve an "A" Grade! I know I shouldn't be complaining, but I am wondering why the significant repetition of the word "Kingspan" has not negatively impacted my score? http://www.just-insulation.com/001-eshop/buy-kingspan-thermapitch-thermawall-thermafloor-insulation-boards.html
Moz Pro | | JustInsulation0 -
Moz vs google data conflict?
Hi there, I am doing an SEO site audit for a client(giveaway, and here is the problem: when performing site:domain.com on google --> 13,800 pages were found When I see this number it seems to be a bit too much compare to the links i checked on integrity(link check for broken links) which gave me a result of 1291. I digged in more into the Google results and saw hundreds(maybe thousands) of pages that are blocked by robots.txt. So I am thinking, ok this is it, thousands of pages can't be crawled by the search engines. Here is the big BUT though, then I check at my moz crawl (see attachment) and no pages are blocked by the SEs, and then look at the dups, only 23 recorded?? Is Moz not crawling properly the 13,800 results that google finds or is this some magical phenomenon happening here? I am really confused here that is why I need some help here! Thank you guys! A990Hu4.png k842AOn.png
Moz Pro | | Ideas-Money-Art0 -
When is the new SEO moz Rank Tracker tool coming?
Anyone know when is the new SEO moz Rank Tracker tool coming? Getting a little flustered with the rank tracker tool, its bit annoying doing one by one. 😞
Moz Pro | | waheeed0 -
Campaign 4XX error gives duplicate page URL
I ran the report for my site and had many more 4xx errors than I've had in the past month. I updated my .htaccess to include 301 statements based on Google Webmaster Tools Crawl Errors. Google has been reporting a positive downward trend in my errors, but my SEOmoz campaign has shown a dramatic increase in the 4xx pages. Here is an example of an 4xx URL page: http://www.maximphotostudio.net/engagements/266/inniswood_park_engagements/http:%2F%2Fwww.maximphotostudio.net%2Fengagements%2F266%2Finniswood_park_engagements%2F This is strange because URL: http://www.maximphotostudio.net/engagements/266/inniswood_park_engagements/ is valid and works great, but then there is a duplicate entry with %2F representing forward slashes and 2 http statements in each link. What is the reason for this?
Moz Pro | | maximphotostudio1 -
Is there a beginners guide to using the tools and what do the results mean.
Great set of tools! But what would make them better is some type of guide to what the tools do (remember I am a new user) and how to maximize the results. Oh and while your at it what do the results mean.
Moz Pro | | marcreece0 -
Any SEO moz users notice a HUGE change in OSE (Open Site Explorer) link data numbers?
Hi All, I am having some serious concern with OSE data recently for numerous clients, one client I want to talk about today has the following data from OSE for the month of August 2011 compared with July 2011: Total links to the domain: (decrease of around 100,000+)
Moz Pro | | ColumbusAustralia
External Followed links: (decrease by around 5,000)
**Linking Root domains: (decrease of over 60) ** The crazy thing is that the domain authority has actually gone up by around 5 points for this client even though every thing has suddenly gone down? Also funny thing is we have been link building quite strong for this client over the last 12 months using only high quality sources from out niche. I am worried that their is serious issues with the data, I realise we saw some updates to OSE recently yet I am suprised it can be this drastic. Kind Regards. PSV1