How Do I Scan My New Site & Grade My Work With The Robots Turned Off? For Pre-Inspection before I launch my Site?
-
I have a new site that has all the bots turned off so google can't index my site until I'm finished it. I've been working on this site for a couple months now optimizing and I was wondering if there was anyway I can run a preliminary scan on the site for my titles, URLs, Headers, Alt Tags and pretty much anything else that will grade my work and tell me if i did anything wrong? Can MOZ do this with the Bots turned off?
Thanks
-
You could block the site for all bots except rogerbot (
Mozilla/5.0 (compatible; rogerBot/1.0; http://www.seomoz.org/dp/rogerbot).
Alternative would be to use a local crawler like Screaming Frog (and modify the spider settings so that it ignores the robots.txt)
rgds,
Dirk
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Unsolved What would the exact text be for robots.txt to stop Moz crawling a subdomain?
I need Moz to stop crawling a subdomain of my site, and am just checking what the exact text should be in the file to do this. I assume it would be: User-agent: Moz
Getting Started | | Simon-Plan
Disallow: / But just checking so I can tell the agency who will apply it, to avoid paying for their time with the incorrect text! Many thanks.0 -
Why does a new page in Page Optimization say [No Title]?
I've just added a keyword and URL in Page Optimization. It gave me a 98 score - all good so far. But then when it added this page to the list of earlier pages it shows the keyword and URL but the title is [No Title] - and without a title I can't select it to edit it. Is this familiar to anyone?
Getting Started | | swbaxter0 -
How to authenticate Moz crawler so that others don't use Rogerbot useragent to scrape data from our site?
Is there any way to authenticate genuine Moz crawler. Because, our website keeps getting scrapping attacks and if there is no way to authenticate Moz crawler, then, any scraper can just set user agent as Rogerbot and scrape all our pages. Is there a fixed IP that can be used or any other customization that will help us authenticate and allow only Moz crawler to crawl our site. Looking forward to a solution to this problem. We haven't been able to use Moz crawler due to this issue.
Getting Started | | longclimber0 -
Google+ Not Working in my Profile
I would like to add my Google+ and Facebook addresses in my profile but when I do it says my vanity names are not correct. I am copying and pasting directly from the Google+ page and Facebook page so I do not know what to do. When I do try to just add the numbers for my Google+ page that is not working either. Thanks - Travis
Getting Started | | TravG1230 -
Why Moz New Users Are Left Alone?
New users when sign up to an application like Moz need concrete support to make them familiarized the basics at least, but Moz seems to be not realizing this hardship of the new users or intentionally ignoring it. There is no chat service or phone numbers to quickly contact the designated agent for some quick help. Mail responses that are received after much time have their limitations -- not much interactive. What the new users should do, because the Moz is not that simple for new users. [edited question for formatting]
Getting Started | | Sequelmed1 -
Site traffic vs other sites
I have a quick question. Does Moz have a tool I can use to see how much traffic my site gets a month versus some competition? I know Alexa has this but you have to create an account. I am just wondering if Moz has one and I am missing it.
Getting Started | | trumpfinc0 -
Can't setup new campaign
Hi everyone, I'm trying to set up a new campaign for a website which has Cloudflare installed. After I enter the campaign name and URL the loading circle comes up and spins for a while, but then it just stays on the same page. No error message is given. I can't get to the next page of the setup campaign form sequence so that I can set up this campaign. Has anyone else had this problem and is there any fix? Thanks in advance
Getting Started | | _jrmo
James0 -
How to get moz to crawl a staging domain that is blocked by robots.txt
Is it possible to get Moz to do a crawl report on a domain blocked by robots.txt and actually display all errors instead of only one saying the domain was blocket in robots.txt? Anything i can add to robots.txt to make moz able to do the crawl report but still hinder google from crawling a staging domain?
Getting Started | | classifiedtech0