Crawling Website
-
How often does Moz crawl the website? I have a lot of high priority crawl issues but my web development team says that it fixed them. However, Moz still shows me that none of the errors are fixed.
-
The crawl test will show the results in a CSV file, but not update on your dashboard. It is a way to spot-check and see if the changes have been made.
When you go to your crawl diagnostics page, it will tell you the date of the last crawl. We crawl on the same day each week.
-
So will it update automatically in my overview as well? When will it be crawled on a weekly basis, in the beginning of the week or end of the week? How do I know if my website has been crawled yet?
-
We crawl on a weekly basis. You can force a crawl for up to 3000 URLs at the crawl test at https://moz.com/researchtools/crawl-test.
-
Please see this discussion to help you out.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Moz unable to crawl my Zenfolio website
Hey guys, I am attempting to optimize a website for my wife's business but Moz is unable to crawl it. Zenfolio is the web hosting service (she is a photographer). The error message is: **Moz was unable to crawl your site on Apr 1, 2019. **Our crawler was not able to access the robots.txt file on your site. This often occurs because of a server error from the robots.txt. Although this may have been caused by a temporary outage, we recommend making sure your robots.txt file is accessible and that your network and server are working correctly. Typically errors like this should be investigated and fixed by the site webmaster. Read our troubleshooting guide. I did read the troubleshooting guide but nothing worked. My robots.txt file disallows a few bots, but not roger bot. Anyone have any idea what is going on? Or do I need to request server logs from Zenfolio? Thanks
Getting Started | | bpenn111 -
How to have MOZ site crawl pre-launch
Hi, Our new website is about to launch. We would love to have moz.com SITE CRAWL our site before launch. For issues like "missing meta description" and everything else that moz.com checks. We would love to do it before we launch. The new site is currently on a different domain than our live site. example.com <-- this is our live site. new-site.com <-- this is our "staging" server with the new site. We have a long running campaign for example.com Do we need to create a new campain for new-site.com ? Or is there some other simpler way? When we launch we will switch the site from new-site.com to example.com .. example.com will be the address for the new site.. Any ideas or suggestions? best practices? edit Forgot to say thank you for your help and input 🙂
Getting Started | | tandvarden0 -
Scheduled update - Re-Crawl - recrawl
Can I not perform a manual update? I setup a campaign without GA as I did not have access, I got access, added the GA account to the campaign but no data is showing as I think I require an update, but have to wait 7 days? Is that right? Thanks
Getting Started | | SJMDT0 -
My website does not allow all crawler to crawl, Now my question is that whether i need to give permission to moz crawler if yes then whaat is moz bot name?
My website does not permit all crawler to crawl website. Whether ii need to give permission to moz bot to crawl website or not? If yes what is the moz bot name?
Getting Started | | irteam0 -
Tools or guide to upgrade the Ranking of my website
Hi everybody, I'm new on SEO Moz and we don't realy know where I need to start to optimized the SEO of my client When i go in the On-Page Optimization section of my SEO Moz account in my campaign, i have a lot of page with a Rank of #1 but with a Grade of F. Why it's our page who are first and it's not another page? Why I have a F grade but i'm first? We don't know where to start when we want to increase the SEO of a page from a client. Where I can find a tools, a guide or a checklist that explain what i need to do to increase the ranking? Can someone help me please? Thank you
Getting Started | | cFocus0 -
My site is not being fully crawled
Our site has been crawled several times by RogerBot but each time only 6 pages are crawled even though we have more than 100 pages. Do I need to submit my sitemap.xml to Moz?
Getting Started | | Scurri0 -
How do I update the crawl issues & Notifications?
I have a list of errors, most relating to missing meta descriptions. I have added a meta description to a page, visited the site and viewed the source, and the meta description is now there. When I go to analyze issues, the report it gives back for the link contains the same missing meta description as previously. How do I get it to update and realize the issue has been fixed?
Getting Started | | ETGg0 -
How do I interpret Duplicate Content in a Crawl Report, when it only gives me a URL? How do I know what is duplicated on that page somewhere else?
I need help interpreting the Crawl Report for Duplicate Content. It gives me the URLs of pages that have duplicate content, but how do I know what content exactly is duplicated elsewhere? And how do I figure out where it is duplicated? Also, are there Moz Analytics articles or videos teaching you how to use each component of the analytics programs? Thanks!
Getting Started | | NancyBryan0