My page cant be crawled by MOZ
-
Hi everyone,
I have a page named zita.vn. Im using Moz to do SEO for the page, but when I use On-page grader in MOZ, I got an error that the page cant be accessed. Additionally, I got an issue with page crawling.
Google search still crawl my page normally.
Please help.
Thanks a lot.
-
Hi there,
I have been checking my site for a week, but I havent seen any new updates about my site in backlink. Im sure that since last week there are a few sites pointing a link to my website but I havent found it on Moz.
Please explain it for me.
Tks a lot,
-
The useragent is: rogerbot
-
Hi Kristina,
Thank you for your answer.
Currently to protect our site from being hacked we need your user agent to add it into our whitelisted, so you will be able to crawl our website.
Regards,
Phuong.
-
Hi there! Kristina from Moz's Help Team here.
When our crawler attempts to access your site, we are running into this error: http://screencast.com/t/CmprKKb7a "403 Forbidden".
This normally means that you're blocking us on a server level. When I ran your site in a header checker I was able to see a 200, but when I curled your site from our servers it returned a 403.
The bummer here is that I can't do anything on the support side to fix this.
This 403 error is coming directly from your server, so at this point the best thing to do will be to send this information to your web dev. They'll need to make sure our crawler (rogerbot) is whitelisted, so our tools can start working hard for you.
As always, you can contact our team on issues like this in the future by emailing help@moz.com, but hopefully this helps point you in the right direction towards a solution here.
Thank you,
-Kristina -
I'm seeing a 403 response for your website:
wget zita.vn
--2016-08-02 11:46:09-- http://zita.vn/
Resolving zita.vn... 107.167.188.173
Connecting to zita.vn|107.167.188.173|:80... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: https://www.zita.vn/ [following]
--2016-08-02 11:46:09-- https://www.zita.vn/
Resolving www.zita.vn... 107.167.188.173
Connecting to www.zita.vn|107.167.188.173|:443... connected.
HTTP request sent, awaiting response... 403 Forbidden
2016-08-02 11:46:10 ERROR 403: Forbidden.If Moz is seeing a 403 they're not going to be able to crawl.
-
there is a possiblity of this issue is due to TLD, as i believe Moz crawler do not support to all TLDs as of yet. this could be the only reason comes to my mind. However, i reckon to raise a suppert ticket for more certian answer for this.
-
Maybe you've set a number of threads too high and pause between requests too low? Some hosts will block IPs that send too many requests. How much is too many depends on your host.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Crawl Test is now On-Demand Crawl!
If you've been with Moz a while, you may have used our old Crawl Test tool. A year ago we launched an all new, campaign-based Site Crawl (with an entirely rebuild crawl engine), but Crawl Test fell into disrepair and we haven't had a solid tool for crawling non-campaign domains. I'm happy to announce that we've just launched an all new On-Demand Crawl, built on the new Site Crawl engine, with a UI that's focused on quick insights. Moz Pro Standard tier customers can run up to 5 crawls per month at 3,000 page per crawl (crawls are saved for 90 days), with per-month limits increasing at higher levels. Most On-Demand Crawls should run in a few minutes, making the tool perfect to get quick insights for sales meetings, vetting prospects, or analyzing competitors. We've written up a sample case study or logged-in customers can go directly to On-Demand Crawl. Try it out -- we'd love to hear your use cases (either here or in the blog post comments).
Moz Bar | | Dr-Pete6 -
Has anyone had to deal with Moz crawl issues on their Zendesk support site?
If so - how did you end up resolving them? For instance we have 85 "temporary redirect" errors from our Zendesk support site in our crawl error report and we don't have access to the robots.txt file through Zendesk.
Moz Bar | | zspace0 -
Crawl Test : Error attempting to request HTTPS page
Hallo When I launch the crawl report I get csv file with this error : 804 : HTTPS (SSL) error encountered when requesting page.
Moz Bar | | micvitale
Error attempting to request page; see title for details. Website is https://bastabollette.it0 -
How Do I Troubleshoot 804 HTTPS Crawl Error?
In my Moz crawl report I get: Crawl Error
Moz Bar | | digium
Moz encountered an error on one or more pages on your site
Error Code 804: HTTPS (SSL) Error Encountered The Moz Help Section only says: 804 HTTPS (SSL) error 804 errors result from a site with misconfigured SSL software. If Moz's crawlers cannot correctly interpret an SSL response for a home page, the crawl ends immediately. My site is publicly accessible on https - https://www.respoke.io/ And I'm not seeing any issues with my certificate. Can anyone help me out? What steps can I take to troubleshoot this error? If SSL is misconfigured, how do I configure it properly?0 -
Checking Do-follow using Moz Bar
Does anyone know how to check if a link is a do-follow or no-follow using the Moz bar? I believe there's a function that highlights and color codes the link so as to tell? Can't seem to turn this function on in my Moz Bar settings. Am I missing something completely obvious?
Moz Bar | | Gavo0 -
Moz / more changes on the way?
I love Moz and the community and all the tools here. I admit I haven't rolled around in all the new things rolled out a few months ago. I thought there were more changes on the way but I wasn't sure if those already happened and I missed them or if I need to be patient? Affiliate program, client reporting? Thanks for any response. Have a great weekend! Matthew
Moz Bar | | Mrupp441 -
Is Manual Crawl Test option available now to Pro Users?
Hi all, I have worked on my Crawl Issues and want to see how many still exist. Earlier I was using Manual Crawl Test. However, now I don't see this tool in Moz Account. Please suggest. Thanks
Moz Bar | | chandman0 -
How do I get Moz to crawl my site?
If I'm not mistaken, Moz will crawl my site once a week and let me know of any errors, notices etc.. But can I ask them to crawl once I have updated all the errors? or do I have to wait for them to do this automatically?
Moz Bar | | david.smith.segarra0