Welcome to the Q&A Forum

Browse the forum for helpful insights and fresh discussions about all things SEO.

Your browser does not seem to support JavaScript. As a result, your viewing experience will be diminished, and you have been placed in read-only mode.

Please download a browser that supports JavaScript, or enable it if it's disabled (i.e. NoScript).

Moz can't crawl our site

Link Explorer

669

SigneerHFS last edited by

Moz can't crawl our site because of an error in the robots.txt, we've tried everything in the troubleshooting guide but nothing works - I believe its a server error but have no idea how to fix it pls help
1 Reply Last reply
Reply Quote 0
Natalie-Alexis Staff last edited by

Hi there!

We would be happy to look into this for you! Can you please send us an email over at help@moz.com so we can pinpoint what is going on here?

We would also need the URL of the site you are trying to crawl.

We look forward to hearing back from you soon!
1 Reply Last reply
Reply Quote 0

Got a burning SEO question?

Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.

Start my free trial

Browse Questions

View

From

Sorted by

With category

Explore more categories

Related Questions

Can MOZ keyword explorer output sentences instead keywords?

Can MOZ keyword explorer output sentences instead keywords? We need to optimize our website for the Knowledge Graph (OG). There was an old tool that does it, but I can't remember its name.
Link Explorer | | Elchanan

1
Doubt Regarding Moz Spam Score

My website is https://www.whatiscbd.com/ . When I check spam score of whatiscbd.com, moz show that its spam score is 9. And when i search spam score of www.whatiscbd.com, spam score is 0. Although both are same website. Why i am getting such issue. Is there any issue in my website's URL redirection. If yes then what's the issue?
Link Explorer | | HuptechWebseo

0
Angular SPA & MOZ Crawl Issues

Website: https://www.exambazaar.com/ Issue: Domain Authority & Page Authority 1/100 I am using Prerender to cache/render static pages to crawl agents but MOZ is not able to crawl through my website (https://www.exambazaar.com/). Hence I think it has a domain authority of 1/100. I have been in touch with Prerender support to find a fix for the same and have also added dotbot to the list of crawler agents in addition to Prerender default list which includes rogerbot. Do you have any suggestions to fix this? List: https://github.com/prerender/prerender-node/commit/5e9044e3f5c7a3bad536d86d26666c0d868bdfff Adding dotbot to Express Server:
prerender.crawlerUserAgents.push('dotbot');
Link Explorer | | gparashar

0
Can I check outbound links from a website?

Hi! Is there an easy way to check if a certain domain has a link to my domain. Is there something similar to "checking outbound links" on a website?
Link Explorer | | AnnabelH

0
Will Moz move DA from an old domain to the new?

Hi folks, I'm moving a digital magazine of mine, which has a great deal of authority (DA53!) to a new domain, which is much more brandable. I'm curious if Moz ever passes the actual DA/PA numbers to the new domain. Does the tool pick this up, eventually? I know once the domain is switched, DA will show a "1" -- as the domain is brand new, technically. This is NOT a question about authority/link juice being passed, but rather the numbers being displayed by Open Site Explorer. Some authors do look at Moz DA as an indicator of a sites "power" before applying for authorship, so I'm trying to find a way to move to a more brandable domain, while keeping the OSE stats in place. If it's not possible, I'll have to live with it 🙂 Any input? Thanks everyone!
Link Explorer | | avilawebfirm

0
Crawl Errors on a Wordpress Website

I am getting a 902 error, "Network Errors Prevented Crawler from Contacting Server" when requesting a site crawl on my wordpress website, https://www.systemoneservices.com. I think the error may be related to site speed and caching, but request a second opinion and potential solutions. Thanks, Rich
Link Explorer | | rweede

0
Moz Toolbar "Get Keyword Difficulty"

New Moz Pro user here, and I've a question about the green "Get Keyword Difficulty" button that shows up to the right of the Google Search Keyword Term Input field after a successful search. Clicking on this seems to have no effect for me what-so-ever. Doesn't take me to the keyword tool, download a report, or add anything to the one screen link analyses in the search results. In short, i'm not sure how this tool is supposed to function. I'm using the latest version of Google Chrome (Version 52.0.2743.60 beta-m (64-bit)) on a Windows 10 machine.
Link Explorer | | bvkinsight

1
Open Site Explorer Only showing 2 Internal Links

Hi, for some reason Open Site Explorer only shows 2 internal links linking to our homepage (and other pages). When looking at a competitors site, they have thousands of internal links to their homepage! Are there any reasons why OSE is only showing 2 for our site, when in fact there are hundreds and thousands? Thanks.
Link Explorer | | Jaybeamer

0