Screaming frog, Xenu, Moz giving wrong results
-
Hello guys and gals,
This is a very odd one, I've a client's website and most of the crawlers I'm using are giving me weird/ wrong results.
For now lets focus on screaming frog, when I crawl the site it will list e.g. meta titles as missing (not all of them though), however going into the site the title is not missing, and Google seems to be indexing the site fine. The robots.txt are not affecting the site (I've also tried changing the user agent).
The other odd thing is SF gives a 200 code but as a status tells me "connection refused" even though it's giving me data.
I'm unable to share the clients site, has any one else seen this very odd issue? And solutions for it?
Many thanks in advanced for any help,
-
Thanks Dan!
-
^ That's customer service!
-
Cheers for the tweet Andy.
Chris, I'd recommend running through this list here in our FAQ -
http://www.screamingfrog.co.uk/seo-spider/faq/#17
In particular, I'd recommend reducing the crawl speed & adjusting the user-agent.
You can also pop it through to our support (http://www.screamingfrog.co.uk/seo-spider/support/) & we can advise. All confidential etc.
Cheers
Dan
-
I have shared this question with Dan from Screaming Frog for you
-Andy
-
Hi Chris
This is hard to answer without an example - do you have a URL or two that you can share with the community?
Thanks so much!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Unsolved Why does Moz Crawler start with HTTP//1.1 version??
We have run Moz Pro's Site Crawl for example-domain.com Why has Moz's crawler decided to site http://www.example-domain.com (ie the non-secure version) as zero crawl depth page and report the actually live https://www.example-domain.com (ie secure HTTP2 version) as a crawl depth of 2? Surely the main live page should be the first crawled and reported as crawl depth = 0?
Moz Pro | | AKCAC0 -
How long does Moz takes to update DA and PA value?
I used to work daily on my office website https://quintdaily.com/ and follow link building strategy. however, since more than 40-50 days, there is no movement in my DA, as the PA has just lifted up. However, there is no changes in Domain Authority so far. Besides, other than this website, im working on location wise SEO projects, but, i can find those websites Domain Authority changed in less 20 days. Is there is time period in changing the Domain Authority once it touched DA 30+?
Moz Pro | | seochris172 -
MOZ not accepting recent changes
We have made some changes on our website according to the SEO standards per Moz report and did some crawler tests but it seems that MOZ is not crawling our recent changes. It is showing the results according to the previous data. What can be the problem here?
Moz Pro | | torbett0 -
Does Moz recognize rel next prev tags? Magento question
Howdy Mozzers! We are running a store in magento where we have many products in each category. Hence view all for category pages is not an option. We have applied rel next prev tags to our paginated pages in the following manner Example for page 2 in a category: The issue we are facing is that Moz suggests www.domain.com/category and www.domain.com/category?p=1 as duplicates, even though rel next prev tags are implemented. 1. Does nel next prev consolidate link juice?
Moz Pro | | MozAddict
2. Does Moz recognize the tags?
3. Will this work for us or should we implement canonical tags as well?0 -
Simply Report for new requests with Moz Pro Plus
Hi all I'd like to know how can I construct a brief report to send to my potential customers when they first write me asking for general informations and evaluation about their websites and certain keywords for which they want to have a great rank. I'd like to reply them with a short email, with a basic report attached, in which showing them some little details about their (and eventually their competitors) ranking and/or their website situation. It's just a brief professional introduction to our seo and web marketing services. Now I'm doing it manually or with other seo tools, but I have just bought a Moz Pro Plus annually version and I'd like to use Moz for this activity. For example, when I first insert a new campaign, I don't see ranking for days so I can't send rankinh report. Do you have any advice? Thanks.
Moz Pro | | Penny10 -
Rank Tracking results
Hi, Some problem with rank tracking from research tools: I rank #1 for this keyword "w immo" for URL : http://www.wimmobiliere.com in my campaign monitor (ranking tab) In rank tracking for the same Search Engine (google.be) i am not in the 50. Do you know why? Some images that might help... Rank_1<a class="attachment-title">Rank 1</a>Rank_2<a class="attachment-title">Rank 2</a>
Moz Pro | | VOUS0 -
UK Google Results .com V's .co.uk
In the UK lots of people use google.com rather than google.co.uk. See: http://weblogs.hitwise.com/robin-goad/2008/04/how_popular_is_googles_pages_from__the_uk_search_option.html The above data is a bit out of date but still I know lots of UK people who search on google.com. When I depersonalise the results and search there is a difference between what I see on google.com V's what I see on google.co.uk. Indeed one of the companies I work for ranks ok on google.co.uk, but not so well on google.com Questions: How would I best track ranking for UK based users who use google.com AND/OR google.co.uk using the SEOmoz tools (or other tools if appropriate)? I note that in "Google Webmaster Tools" you can change the setings to "Target users in United Kingdom". Given the fact that UK users use both google.com AND/OR google.co.uk would it be better to change this to unlisted or deselect the "Target users in" option? Thanks in advance. Justin
Moz Pro | | GrouchyKids0 -
Press Release - using moz bar/OSE is reading domain not page? How? Why?
A question posed by Christopher Glaeser from early today:low PA high DA, had a follow up response from him providing 2 urls from PR WEB for separate press releases: http://www.prweb.com/releases/2011/11/prweb8923419.htm (HP White) On moz bar Page Analysis/Link Data = PA - 47 DA - 36 http://www.prweb.com/releases/2011/12/prweb9051351.htm (Golfer's Advice) On moz bar Page Analysis/Link Data = PA - 1 DA - 96 I kept scratching my head as to how a press release of 6 weeks ago had garnered such attention from a company that would not seem to have a huge traffic due to more obscure product offering and scientific subject (Analyses of Armor Industry versus Golf Advice).
Moz Pro | | RobertFisher
Then I realized that for HP White, Link Data was not about the PR. The url from mozbar was HPWhite.com not the above, I did not notice until I used OSE where same thing was happening. When I cut and pasted the above press release url for HP White and placed it in OSE this changed: PA - 49 DA - 96 (2 links 2 linking root domains) For Golfers advice (0 links from 0 linking domains) Note to all: the links to the PR WEB release for the HP included a low end directory type link and a link from PR WEB (. For Golfer's Advice there was not a link back to the release from PR WEB: Note that Golfer's Advice is a newer release (6 weeks). So, any link from HPWhite release would equal more juice to HP White and PR Web and Vocus. Any link to Golfer's Advice from release offers......???? to Golfer's Advice and who cares to Vocus and PR Web. So, I guess this begs a couple of questions: Why the mozbar link analysis difference for one versus the other? Does PR Web treat some differently than others? Who benefits most from me paying a PR Web to do press releases for a client, PR Web and Vocus or my client and I???? I have tried to order the images to make sense: L to R top, then bottom is last. [
](<a href=)" target="_blank">a> [
](<a href=)" target="_blank">a> [](<a href=)" target="_blank">a> [
](<a href=)" target="_blank">a> [](<a href=)" target="_blank">a> [](<a href=)" target="_blank">a>
0