Possible Crawling Problem with Screaming Frog and Moz Crawlers
-
So I'm not sure if what I'm seeing is a problem or not.
As of about two weeks ago the Moz crawler has only been able to see www.mysite.com, and none of the links, content, title, ect associated with the page. Essentially the report has one line, what should be the homepage, but it's not able to pull any information from the page but does show a 200 http status code. The report shows nothing blocked by robots or any errors.
When I use screaming frog to crawl the site about 75% of the time it just reports one line www.mysite.com with a 200 status code, but again the crawler is not able to actually see the html. The other 25% of the time it works perfectly fine, crawls all pages and sees all meta info and content.
There are no errors in Google WMT and everything looks ok there. We have seen a traffic drop the last two weeks but I don't know if this is the reason for it.
I can't publicly post the page but if someone has an idea of what might be going on I'd be happy to PM them.
Thanks
-
Thank you for the response.
I've ran two MOZ crawl reports today, one with mysite.com and one www.mysite.com. Both returned 1 result for mysite.com and www.mysite.com respectively, with a 200 status code, but no meta data. I know that I've successfully crawled www.mysite.com about a month ago with no problems. I have made small changes here and there but nothing is jumping out at me as wrong.
Screaming Frog is currently crawling my site successfully about 1/10 tries. The successful tries it sees 163 Total URL Encountered (its a small site) and the other 9/10 times it shows exactly 1 URL (the one i entered) and no meta data. There doesn't seem to be any pattern when it successfully crawls and when it doesn't make it past the first page.
Google WMT is currently showing No Data Available for both internal links and links to your site which is a little concerning. Everything else in WMT looks ok.
-
Two possible simple key-in items to consider: make sure the URL is inputted w/ the full url (not just mysite.com) and/or ensure to click any options for including root or sub-domains so its not just looking at a single page.
-
If you PM me the domain I can take a look myself.
Does the robots.txt have anything funny in there?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Moz and HubSpot SSL - crawl error?
I'm getting an error message when Moz tries to crawl my site, however when I check in Google Search Console, they return no errors. Our site is hosted on HubSpot. Is Moz still having trouble crawling HubSpot sites that have enabled their SSL? I read an article that this should have been corrected in early 2017, but I'm getting an error.
Moz Pro | | jennygriffin0 -
Understanding Moz Opportunity and Potential
Hello Friends - I am just learning Moz tool via Udacity I was doing some analysis using Moz for the following keyword "digital marketing courses" This is the result what I get. Moz tells me the opportunity is 83. This is a high number and does not reflect what I see when I search in google. The page is full of ads. A organic click is going to be very difficult.
Moz Pro | | udacity-singhvishu
Moz potential also says 79 - again a high number which does not seem to be the reality. It is going to be very difficult for a new service to rank higher. I have fundamental doubt on whether SEO will work on keywords which are in high demand by paid ads. more that 30% of the real estate on screen gets taken by paid ads. Even if one searches for long tail key word the story remains same. Does SEO really have a chance to compete with paid ads for business driving keywords? pub0 -
Is it possible to track rank history
I would like to know is it possible to track a keywords past ranking history? We would like to check the ranking history of some potential vendors, but only get data from the point I add them to Alabs, moz etc. Id like to go back a year to see the trend in their rankings. Is this possible and what tool or process can be used to achieve this. Thanks in advance.
Moz Pro | | anthonytjm0 -
Sub-accounts on Moz
Can you create sub-user accounts on Moz?... Ones which provide limited access to certain users like use of the tools or access to view rankings in a campaign without being able to edit anything??
Moz Pro | | Qology1 -
Problem in doing seo for a German medical website?
Hi We have been doing seo for http://www.internetapotheke-holland.com/ since last 4 month including onpage and offpage process like link building etc. but not getting a proper output keywords are as follow: xenical holland rezeptfrei xenical Xenical Ohne Rezept Xenical Bestellen Xenical Kaufen Rezeptfrei bestellen Rezeptfrei kaufen Tramadol TropfenRezeptfrei Please could you add some more cool tips so that we can help our client. Thanks & Regards
Moz Pro | | expertvillagemedia0 -
Do you know if there is an problem with the rank Tracker on SEOmoz
My rank tracker is stating that I am not in the top 50 for my keyword which is auto insurance fremont ca. I check on bing and yahoo and i am listed as #2. Also on Google it states that my rank has gone down by 2 place but I acutally increase one from spot 12 to spot 11. Is anyone else having the same issue??
Moz Pro | | Greenpeak0 -
SEOMOZ Crawling Our Site
Hi there, We get a report from SEOMOZ every week which shows our performance within search. I noticed for our website www.unifor.com.au that it looks through over 10,000 pages, however our website sells less than 500 products so not sure why or how so many pages are trawled? If someone could let me know that would be great. It uses up a lot of bandwidth doing each of these searches so if the amount of pages being trawled reduced it would definitely assist. Thanks, Geoff
Moz Pro | | BeerCartel750 -
It is possible to view the older Domain Authority for a domain?
It is possible to view the older Domain Authority for a domain with open site explorer? Thanks.
Moz Pro | | caroy0