Screaming frog, Xenu, Moz giving wrong results
-
Hello guys and gals,
This is a very odd one, I've a client's website and most of the crawlers I'm using are giving me weird/ wrong results.
For now lets focus on screaming frog, when I crawl the site it will list e.g. meta titles as missing (not all of them though), however going into the site the title is not missing, and Google seems to be indexing the site fine. The robots.txt are not affecting the site (I've also tried changing the user agent).
The other odd thing is SF gives a 200 code but as a status tells me "connection refused" even though it's giving me data.
I'm unable to share the clients site, has any one else seen this very odd issue? And solutions for it?
Many thanks in advanced for any help,
-
Thanks Dan!
-
^ That's customer service!
-
Cheers for the tweet Andy.
Chris, I'd recommend running through this list here in our FAQ -
http://www.screamingfrog.co.uk/seo-spider/faq/#17
In particular, I'd recommend reducing the crawl speed & adjusting the user-agent.
You can also pop it through to our support (http://www.screamingfrog.co.uk/seo-spider/support/) & we can advise. All confidential etc.
Cheers
Dan
-
I have shared this question with Dan from Screaming Frog for you
-Andy
-
Hi Chris
This is hard to answer without an example - do you have a URL or two that you can share with the community?
Thanks so much!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Block Moz (or any other robot) from crawling pages with specific URLs
Hello! Moz reports that my site has around 380 duplicate page content. Most of them come from dynamic generated URLs that have some specific parameters. I have sorted this out for Google in webmaster tools (the new Google Search Console) by blocking the pages with these parameters. However, Moz is still reporting the same amount of duplicate content pages and, to stop it, I know I must use robots.txt. The trick is that, I don't want to block every page, but just the pages with specific parameters. I want to do this because among these 380 pages there are some other pages with no parameters (or different parameters) that I need to take care of. Basically, I need to clean this list to be able to use the feature properly in the future. I have read through Moz forums and found a few topics related to this, but there is no clear answer on how to block only pages with specific URLs. Therefore, I have done my research and come up with these lines for robots.txt: User-agent: dotbot
Moz Pro | | Blacktie
Disallow: /*numberOfStars=0 User-agent: rogerbot
Disallow: /*numberOfStars=0 My questions: 1. Are the above lines correct and would block Moz (dotbot and rogerbot) from crawling only pages that have numberOfStars=0 parameter in their URLs, leaving other pages intact? 2. Do I need to have an empty line between the two groups? (I mean between "Disallow: /*numberOfStars=0" and "User-agent: rogerbot")? (or does it even matter?) I think this would help many people as there is no clear answer on how to block crawling only pages with specific URLs. Moreover, this should be valid for any robot out there. Thank you for your help!0 -
Links not appearing in Moz tool
Hey Guys I am finding that my Moz tool isn't showing links that are definitely there like from social media etc. Also links that are there about 4-5 months are not showing either! am i doing something wrong?
Moz Pro | | Johnny_AppleSeed0 -
We need your expertise! Do you have a tool hack or trick you use with Moz Pro?
Hey friends! We're looking for any tricks and tips you have for other people using Pro. Everyone has their own way of doing things. We'd love to see if you have any hacks you've created. You can send over just a screenshot, or a blog post, or anything you've got! Here are a couple examples: [blog post] Did you know you can use OSE's anchor text export to see if you are getting setup for a penguin penalty? This is a blog post about it: http://moz.com/academy/google-penguin-and-you [screenshot] Did you know you can get all your keyword ranking historical data by clicking this one little button? http://i.imgur.com/l4KxqBt.jpg Do you have anything you'd like to add? We're hoping to help spread these to others to help people utilize the tools to their potential. Thanks!
Moz Pro | | jennita2 -
How to Learn a Lot from Moz Q&A
You can learn a lot from Moz Q&A without asking any questions. The way to do that is to watch for topics that you are uncertain about, then subscribe to that question by clicking the "Email Updates" button. Then, when people reply to that topic, you will be able to read what they have to say and will likely learn many things (and see a few entertaining arguments 🙂 Give it a try. You might subscribe to more questions than you think and learn many things that will improve your skills and expertise.
Moz Pro | | EGOL8 -
Poor quality PDF exports from analytics.moz.com
The new reports look great http://analytics.moz.com/dashboard/overview/ But when I export to PDF, the text is low quality and images are low quality. Not as good as the old report. In fact, the PDF is such low quality we are not providing them to our customers. We're using the old reports. ETA for HQ PDF Export please?
Moz Pro | | OPUS-Cambodia0 -
Universal results not in pdf or csv reports
Right now, the universal results show up when I look at tankings on the site, but do not show up at all in the weekly pdf reports. They also did not appear in the full rankings history .csv report. They only appear in the weekly csv report, and then, the information is not as detailed as it is on the website (it only shows whether a universal exists, but doesn't indicate at what position it shows up). This limits the usefulness of these reports to my customers. I need to be able to show them rankings histories, including universal results, in their monthly or weekly reports. Including the charts would be nice as well. Since I can individually download the .pdfs for these for each keywords, why can't we just include it in the monthly reports. Also, would like to be able to view rankings history for all three search engines at once for a keyword, or multiple keywords to reduce the number of pages in a report. Can you add this feature?
Moz Pro | | HMS0 -
On Page Ranking Tool Giving Weird Reports
My on page ranking tool is giving two entirely different reports for my website - I get one report for my domain name and a different report for the index.html even though essentially both of those pages are...well...the same. Not sure what's going on, hoping it's not an indication of a more serious issue? I appreciate any help!
Moz Pro | | Virage0 -
Why does my crawl report show just one page result?
I just ran a crawl report on my site: http://dozoco.com The result report shows results for just one page - the home page, but no other pages. The report doesn't indicate any errors or "do not follows" so I'm unclear on the issue, although I suspect user error - mine.
Moz Pro | | b1lyon0