Screaming frog, Xenu, Moz giving wrong results
-
Hello guys and gals,
This is a very odd one, I've a client's website and most of the crawlers I'm using are giving me weird/ wrong results.
For now lets focus on screaming frog, when I crawl the site it will list e.g. meta titles as missing (not all of them though), however going into the site the title is not missing, and Google seems to be indexing the site fine. The robots.txt are not affecting the site (I've also tried changing the user agent).
The other odd thing is SF gives a 200 code but as a status tells me "connection refused" even though it's giving me data.
I'm unable to share the clients site, has any one else seen this very odd issue? And solutions for it?
Many thanks in advanced for any help,
-
Thanks Dan!
-
^ That's customer service!
-
Cheers for the tweet Andy.
Chris, I'd recommend running through this list here in our FAQ -
http://www.screamingfrog.co.uk/seo-spider/faq/#17
In particular, I'd recommend reducing the crawl speed & adjusting the user-agent.
You can also pop it through to our support (http://www.screamingfrog.co.uk/seo-spider/support/) & we can advise. All confidential etc.
Cheers
Dan
-
I have shared this question with Dan from Screaming Frog for you
-Andy
-
Hi Chris
This is hard to answer without an example - do you have a URL or two that you can share with the community?
Thanks so much!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Is Search Visibility actually weighted by search volume at all? Everyone seems to think so, but that seems completely wrong?
As I've been exploring Moz more, I've been looking into search visibility and what it actually represents. While searching for answers on Moz, I've seen several posts that all seem to think the search volume for keywords matters at all for the search visibility metric, and I'm wondering why anyone thinks that, when everything from Moz seems to indicate otherwise. https://moz.com/help/guides/moz-pro-overview/rankings/search-visibility describes the calculation as taking the CTR for each keyword, based on rank, summing those, and dividing by the number of keywords. This describes a simple, and completely unweighted by search volume, average. This means that if you are tracking 10 keywords, and rank #1 on 5 that have a search volume of 0-10, and are unranked on 5 that have a search volume of 70,801-118,000, that would generate exactly the same search visibility score as if you rank #1 on the 5 high volume, and are unranked on the 5 low volume. The above all makes search visibility look like a much less valuable metric than if it were appropriately weighted by volume, as well as potentially very misleading, since the % relates solely to the number of keywords in your analyzed segment, rather than to the % of search volume that you're obtaining (unless a) all of your keywords are the same rank, or b) all of your keywords have the same volume). Am I missing something, or is everyone just giving Moz way too much credit when it comes to the value of the search visibility metric?
Moz Pro | | JoelSharp1 -
What SEO tools do you use in conjunction with Moz?
It seems like most people use multiple SEO tools. I am interested in hearing what you use in conjunction with Moz and why. -Stephen
Moz Pro | | martechwiz2 -
Will moz crawl pages blocked by robots.txt and nofollow links?
i have over 2,000 temporary redirects in my campaign report redirects are mostly events like being redirected to a login page before showing the actual data im thinking of adding nofollow on the link so moz wont crawl the redirection to reduce the notification will this solve my problem?
Moz Pro | | WizardOfMoz0 -
Error in Moz duplicate content reports
Hi - I've run the Moz campaign on a client's site. Moz is saying that there are duplicate content errors, and when I look at the errors it is showing that they are all to do with the non-www URLs having being duplicated in the www form of the URLs. However this is not the case - all the non-www URLs are all 301 redirected to the www URLs. Is this an error in the Moz tool? Has anybody experienced something similar?
Moz Pro | | rorynatkiel0 -
Can not start a SEO MOZ campaign?
I keep getting this message, I have a 301 from my domain to my domain with the www prefix. What's the deal? We have detected that the domain www.photosbykristopher.com and the domain photosbykristopher.com both respond to web requests and do not redirect. Having two "twin" domains that both resolve forces them to battle for SERP positions, making your SEO efforts less effective. We suggest redirecting one, then entering the other here.
Moz Pro | | KrisPhoto0 -
My Moz domain authority fell last week but so did all 3 of my competitors. What could cause that?
As a loyal Moz Pro subscriber I track my site's authority, trust and links against 3 similar competitors. Last week my authority fell from 61 to 60 but all 3 competitors saw a drop in their authority that week too. Was the Moz domain authority calculation changed? Did anyone else see a drop or has something odd happened to just our market? Any ideas? Not sure if this is something I should address or just shrug and ignore. Cheers
Moz Pro | | SteveBrumpton0 -
Moz Trust
I saw a severe drop in MozTrust for multiple sites I am working on. We have added some links this month, but would be considered 'spammy' or to be low quality. No other links appear to have been lost / removed. If we had some very high quality links before and have added some mid-level links can this be what is causing the drop? From the description on SEOmoz, it seemed to me that trust could only go up as links were added. It appears that I am missing something.
Moz Pro | | DigitalDiameter0
