Screaming frog, Xenu, Moz giving wrong results
-
Hello guys and gals,
This is a very odd one, I've a client's website and most of the crawlers I'm using are giving me weird/ wrong results.
For now lets focus on screaming frog, when I crawl the site it will list e.g. meta titles as missing (not all of them though), however going into the site the title is not missing, and Google seems to be indexing the site fine. The robots.txt are not affecting the site (I've also tried changing the user agent).
The other odd thing is SF gives a 200 code but as a status tells me "connection refused" even though it's giving me data.
I'm unable to share the clients site, has any one else seen this very odd issue? And solutions for it?
Many thanks in advanced for any help,
-
Thanks Dan!
-
^ That's customer service!
-
Cheers for the tweet Andy.
Chris, I'd recommend running through this list here in our FAQ -
http://www.screamingfrog.co.uk/seo-spider/faq/#17
In particular, I'd recommend reducing the crawl speed & adjusting the user-agent.
You can also pop it through to our support (http://www.screamingfrog.co.uk/seo-spider/support/) & we can advise. All confidential etc.
Cheers
Dan
-
I have shared this question with Dan from Screaming Frog for you
-Andy
-
Hi Chris
This is hard to answer without an example - do you have a URL or two that you can share with the community?
Thanks so much!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Moz shows duplicate content, but URL's are tagged with campaign tags
Crawl diagnostics shows a lot of pages with duplicate content, but when I check the details, I see that it lists the same page but the url contains a campaign tag, so it's not really another page that is serving identical content... Is there a way to remove these pages out of the Crawl Diagnostics?
Moz Pro | | jorisbrabants0 -
Moz scores
Hi All, Could anyone please explain for me why the moz scores that show under SERPs in Google search show one set of values, but when i click on the SERP listing and go to that page, the Moz scores at the bottom of the tab the page is open in are completely different. I.E - SERP result Wireless Alarms <cite>www.wireless-alarms.net/</cite>Honeywell, Friedland, Response, Wireless house home intruder burglar alarms and wirefree doorbells chimes and accessories.<iframe class="SEOmoz-iframe" src="chrome-extension://eakacpaijcpapndcfffdgphdiccmpknp/html/serpbar.html#{" settings":{"dmr":false,"dmt":false,"mr":true,"mt":true,"serp-overlay":true,"toolbar-enabled":true,"subdomain-metrics":false,"serp-panel-open":true,"toolbar-position":"bottom"},"lsdata":{"feid":3,"fipl":3,"fmrp":2.0446037910058426,"fmrr":5.396862303226778e-12,"ftrp":1.8723654310590483,"ftrr":9.21604483751679e-12,"fuid":3,"pda":8.903939958502189,"peid":3,"pid":3,"pmrp":1.6026230610187469,"pmrr":1.3566326316606153e-10,"ptrp":1.025503845909542,"ptrr":1.4972548621304158e-11,"puid":3,"ueid":2,"uemrp":2.577887966559377,"uemrr":5.40110633420739e-13,"uid":2,"uipl":2,"ujid":2,"umrp":2.577887966559377,"umrr":5.40110633420739e-13,"upa":23.634692786039714,"utrp":4.9331532541404854,"utrr":1.9684091945588755e-16},"user":{"level":"pro","options":{},"proenabled":true,"aclurl":"http:="" moz.com="" users="" level?src="mozbar","loginURL":"https://moz.com/login","logoutURL":"https://moz.com/logout","apiRequest":"url-metrics","cols":"133146078688","expired":false,"accessId":"pro-RGFkZHlTbXVyZg%3D%3D","expires":1374925378,"signature":"tBSBXpLbBmk073ofsfBwnilmODw%3D","displayName":"DaddySmurf"},"serp":4,"host":"www.wireless-alarms.net"}"" scrolling="no"></iframe> Page result <label>Page:</label><label id="Page Authority" class="key lsdata">PA:</label> <label id="SEOmoz-data-PA" class="val lsdata">27</label><label id="mozrank" class="key lsdata" title="MozRank">mR:</label> <label id="SEOmoz-data-mR" class="val lsdata">5.30</label><label id="moztrust" class="key pro-only lsdata" title="MozTrust">mT:</label> <label id="SEOmoz-data-mT" class="val pro-only lsdata">5.61</label><label id="SEOmoz-data-uid">1,950</label> links from
Moz Pro | | DaddySmurf
<label id="SEOmoz-data-uipl">2</label> Root Domains<label>Root Domain:</label><label id="dom-pageauthority" class="key lsdata" title="Domain Authority">DA:</label> <label id="SEOmoz-data-DA" class="val lsdata">24</label><label id="SEOmoz-data-puid">100,262</label> links from
<label id="SEOmoz-data-pid">3</label> Domains<label>Subdomain:</label><label id="SEOmoz-data-fuid">19,953</label> links from
<label id="SEOmoz-data-fipl">3</label> DomainsThank you.Si0 -
Does Anyone Know The Moz Analytics Release Date?
I know it's coming soon, but does anyone know the actual release date for Moz Analytics? Thanks!
Moz Pro | | eglassman0 -
Rogerbot's crawl behaviour vs google spiders and other crawlers - disparate results have me confused.
I'm curious as to how accurately rogerbot replicates google's searchbot I've currently got a site which is reporting over 200 pages of duplicate/titles content in moz tools. The pages in question are all session IDs and have been blocked in the robot.txt (about 3 weeks ago), however the errors are still appearing. I've also crawled the page using screaming frog SEO spider. According to Screaming Frog, the offending pages have been blocked and are not being crawled. Webmaster tools is also reporting no crawl errors. Is there something I'm missing here? Why would I receive such different results. Which one's should I trust? Does rogerbot ignore robot.txt? Any suggestions would be appreciated.
Moz Pro | | KJDMedia0 -
Poor OSE results on my links
I am really disappointed with the "return on investment" I am getting from being a Pro member and using Open Site Explorer to analyze and remedy my inbound links post-penguin. My site has been crawled for over 4 months now by seomoz and OSE reprots that I have "external links" from just 25 or so domains - whereas Google webmaster tools has accurately identified well over 270 domains that link back to my site. How can do any sort of link removal strategy using open site explorer? I am paying $100/mo for SEomoz, and nothing for Google. What is the problem?
Moz Pro | | veezer0 -
Why does SEO moz show so many more duplicate page titles than webmaster
My SEO moz duplicate page titles are more than double the webmaster duplicate page titles. Plus, some of the items appearing in SEO moz are not really duplicates. Anyone else seeing this?
Moz Pro | | Stamats0 -
Why Do Links, DA and PA have no affect on this search result?
When I type 'plumbing courses' into Google, why does Able Skills with these statistics: Page Authority: 57,
Moz Pro | | MartinHof
2,752 Links from 330 Root Domains
Domain Authority: 52,
575,594 Links from 1,211 Root Domains rank below other pages with inferior statistics?0 -
Seo moz crawl is not updating
When I check our seo campaign I can see that the report was not updated. It still show that the next crawl is Nov 1 but it is already Nov 3.
Moz Pro | | shebinhassan0