Possible Crawling Problem with Screaming Frog and Moz Crawlers
-
So I'm not sure if what I'm seeing is a problem or not.
As of about two weeks ago the Moz crawler has only been able to see www.mysite.com, and none of the links, content, title, ect associated with the page. Essentially the report has one line, what should be the homepage, but it's not able to pull any information from the page but does show a 200 http status code. The report shows nothing blocked by robots or any errors.
When I use screaming frog to crawl the site about 75% of the time it just reports one line www.mysite.com with a 200 status code, but again the crawler is not able to actually see the html. The other 25% of the time it works perfectly fine, crawls all pages and sees all meta info and content.
There are no errors in Google WMT and everything looks ok there. We have seen a traffic drop the last two weeks but I don't know if this is the reason for it.
I can't publicly post the page but if someone has an idea of what might be going on I'd be happy to PM them.
Thanks
-
Thank you for the response.
I've ran two MOZ crawl reports today, one with mysite.com and one www.mysite.com. Both returned 1 result for mysite.com and www.mysite.com respectively, with a 200 status code, but no meta data. I know that I've successfully crawled www.mysite.com about a month ago with no problems. I have made small changes here and there but nothing is jumping out at me as wrong.
Screaming Frog is currently crawling my site successfully about 1/10 tries. The successful tries it sees 163 Total URL Encountered (its a small site) and the other 9/10 times it shows exactly 1 URL (the one i entered) and no meta data. There doesn't seem to be any pattern when it successfully crawls and when it doesn't make it past the first page.
Google WMT is currently showing No Data Available for both internal links and links to your site which is a little concerning. Everything else in WMT looks ok.
-
Two possible simple key-in items to consider: make sure the URL is inputted w/ the full url (not just mysite.com) and/or ensure to click any options for including root or sub-domains so its not just looking at a single page.
-
If you PM me the domain I can take a look myself.
Does the robots.txt have anything funny in there?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Competitive Analysis Problem
I just discovered that, for my competitive analysis, it is set to judge my naked domain instead of www. My rel="canonical" is set to point to www., thus, www. has a much higher domain authority than my naked domain. However, I don;t want to lose all of the history I have accumulated over the years for this domain. How to I make it so SEOmoz matches against www. instead of my naked domain? Which is wiser to gauge by?
Moz Pro | | bks_seo0 -
Is Anyone Else Having Problems With The Ranking On Pro Tools?
After checking them from the report I was emailed, some of them seem to be incorrect, or is it something my end? To be fair the majority of them are correct, I'm just querying it.
Moz Pro | | JonathanRolande0 -
Only one page has been crawled
I am running a campaing for three weeks now and first two crawls was ok but the last one is showing only one page crawled. the subdomain I am tracking is: www.cubaenmiami.com I have everything correct in my site. Regards Alex
Moz Pro | | esencia0 -
Google Peru rank and SEO Moz
Hello, I wanna track the google Peru rank but the tool say me that im not in the top 50 and i'm sure that I'm, how to get a real ranking?
Moz Pro | | Kuna0 -
Creating a campaign for a folder is not possible
Hi There! I just signed up for a free pro account to see whether it is something for us as a company 🙂 I tried to set up two campaigns. With the first for our root domain (koopgoedkoop.nl) it all went perfect. I am now trying to set up a campaign for our BBQ page (http://www.koopgoedkoop.nl/tuin/barbecues.html) but I can't? Is it because our URL ends with .html? Here's a screenshot: http://bit.ly/x2zdCd Thanks for your help in advance! Regards, Do
Moz Pro | | vidaXL0 -
Different Moz Scores Between Page and Subdomain
The Moz scores for a page-specific website:: www.website.com are different than the subdomain: www.website.com/. www.website.com: Page MozRank: 4.58
Moz Pro | | DanSpeicher
Page MozTrust: 4.9 www.website.com/ Subdomain MozRank: 5.21
Subdomain MozTrust 5.18 Does that have any significance? Should I be routing pages to the " / " version? Thanks, Daniel0 -
Anyone having problems with the SEOmoz KWD Tool??
Just a quicky: Tried to run keyword difficulty tool reports last couple of days but keep getting error messages regardless of how many words/phrases I request. Is this a general outage or just me? Cheers
Moz Pro | | lovealbatross
James0 -
Crawl complete, but nothing changed?
Hi everyone, According to my account, the crawl diagnostics were completed yesterday. However, the duplicate page titles that it mentions aren't correct. The changes that I implemented several days ago are not being shown in the report. When I click the duplicate page title links, the latest date in the graph is 3/26. However, it says the crawl was completed on 3/30. Does it take a few days for the reports to match what the crawl actually discovered?
Moz Pro | | beeneeb0
