Crawling Website
-
How often does Moz crawl the website? I have a lot of high priority crawl issues but my web development team says that it fixed them. However, Moz still shows me that none of the errors are fixed.
-
The crawl test will show the results in a CSV file, but not update on your dashboard. It is a way to spot-check and see if the changes have been made.
When you go to your crawl diagnostics page, it will tell you the date of the last crawl. We crawl on the same day each week.
-
So will it update automatically in my overview as well? When will it be crawled on a weekly basis, in the beginning of the week or end of the week? How do I know if my website has been crawled yet?
-
We crawl on a weekly basis. You can force a crawl for up to 3000 URLs at the crawl test at https://moz.com/researchtools/crawl-test.
-
Please see this discussion to help you out.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
I want to audit my website
Hi professionals, I am newbie in SEO field, I want to audit my company blog "https://iowatreeplanting.com/" How to check domain authority, page authority and other matrics of this blog. I want to rank this blog in google and other search engines but no one can help me to do this! Please guide me in depth i am very thankful to you Thanks!
Getting Started | | Smartlanjabdul1 -
Moz not able to crawl our site - any advice?
When I try and crawl our site through Moz it gives this message: Moz was unable to crawl your site on Aug 7, 2019. Our crawler was banned by a page on your site, either through your robots.txt, the X-Robots-Tag HTTP header, or the meta robots tag. Update these tags to allow your page and the rest of your site to be crawled. If this error is found on any page on your site, it prevents our crawler (and some search engines) from crawling the rest of your site. Typically errors like this should be investigated and fixed by the site webmaster. I have been through all the help and doesn't seem to be any issues. You can check the site and robots.txt here: https://myfamilyclub.co.uk/robots.txt. Anyone got any advice on where I could go to get this sorted?
Getting Started | | MyFamilClubLtd1 -
Multiple campaigns for one website with multiple services?
Hi. Brand new to Moz Pro today. Question about whether to set up multiple campaigns for one website that promotes multiple services. I own a law firm and we have one main location, one website, 5 attorneys, and about 8-10 practice areas we promote. Each practice area has its own page and related content in the form of guides and blogs. Website is southernmarylandlaw.com. I only have this one website to track and promote, so I initially thought I only have one campaign. But I do want to track these separate practice areas separately. Each will have separate competitors, for instance. Each will need a separate set of keywords. People tend to search for a particular type of attorney, not just for the generic "lawyers." Should I use my 10 campaigns to set up a different campaign for each practice area we promote? Am I doing something wrong by tracking my one website 10 different times in my account? If I do not do this, then how do I possibly track the various practice areas and their separate keywords and competitors? Thanks for all your help and suggestions!
Getting Started | | tuckervc0 -
Standard Syntax in robots.txt doesn't prevent Moz bot from crawling
A client is getting many false positive site crawl errors for things like duplicate titles and duplicate content on pages that include /tag/ in the URL. An example is https://needquest.com/place_tag/autism-spectrum-disorder/page/4/ To resolve this we have set up a disallow statement in the robots.txt file that says
Getting Started | | btreloar
Disallow: /page/ For some reason this appears not to work, as the site crawl errors continue to list pages like this. Does anyone understand why that would be and what we need to do to properly disallow crawling these pages?0 -
Why has my website stopped working?
Signed up to Moz and my site is now down as it's "gathering data". Is this normal?
Getting Started | | BurrowsPT0 -
Creating a daily to-do list for SEO my website
Hi all, I'm very new here, and I'm quite slow in terms of learning things. Now I've had MOZ for about one and half month, but still only just understand to use on-page grade to SEO my site. I've been spending quite a bit money (for a small business) for online marketing: facebook adverts $50 a day; google adwords $10 a day; MOZ $99 a month... However, the conversion rate isn't ideal... only less than 2% Now what I'm trying to do is, while keep doing the cash burning online adverts, do more SEO on a daily basis. Can I please ask: other than optimise 5 pages' on-page grade, what else should I include into my daily SEO to-do list? Many thanks!
Getting Started | | alexlovesmoz0 -
Crawl Diagnostics
Hello Experts, today i was analyse one of my website with moz and get issue overview and get total 212 issue 37 high all derive to this same url http://blogname.blogspot.com/search?updated-max=2013-10-30T17:59:00%2B05:30&max-results=4&reverse-paginate=true so can anyone help me how to find this url and remove all high priority error. and even on page website get A grade then why not performing well in SE ?
Getting Started | | JulieWhite0 -
Cant download my crawl csv
When I click on the [download csv] in my crawl campaign section nothing happens.
Getting Started | | digitalmedialounge0