Welcome to the Q&A Forum

Browse the forum for helpful insights and fresh discussions about all things SEO.

Your browser does not seem to support JavaScript. As a result, your viewing experience will be diminished, and you have been placed in read-only mode.

Please download a browser that supports JavaScript, or enable it if it's disabled (i.e. NoScript).

Site Crawl Error

Moz Pro

1035

nopsts last edited by

In moz crawling error this message is appears:

MOST COMMON ISSUES

1Search Engine Blocked by robots.txt

Error Code 612: Error response for robots.txt

i asked help staff but they crawled again and nothing changed.

there's only robots.XML (not TXT) in root of my webpage it contains:

User-agent: *
Allow: /
Allow: /sitemap.htm

anyone please help me? thank you
1 Reply Last reply
Reply Quote 0
nopsts @Abe_Schmidt last edited by

Thank you,

per your instructions i have changed robots.txt exactly like:

User-agent: *
Disallow:

As you can see there's no Allow: / and Disallow line is blank

is that correct?
1 Reply Last reply
Reply Quote 0
Abe_Schmidt last edited by

Hello, this is Abe from the Moz Help team. After our reported fix in your latest support ticket, you will see correct crawl crawl results in your next weekly update
1 Reply Last reply
Reply Quote 0
justin-brock last edited by

Would you mind sharing a link to your site?
1 Reply Last reply
Reply Quote 0

Got a burning SEO question?

Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.

Start my free trial

Browse Questions

View

From

Sorted by

With category

Explore more categories

Related Questions

Major site overhaul

Hi guys and girls After reading all the content It became apparent that our website is a bit of a dog. I'm about to update our site with a new theme and content on moz.com I've found myself becoming so much more confident at all this. I've read all the material I can on Moz re: Site migration but still have one question. How do I find out all the links that I need to 301 before I make the new site live? Please don't worry about patronizing me! I'm really new at this! Ben
Moz Pro | | SussexChef83

0
Inherited a site - no progress being made, no authority?

Hi all, would really appreciate some input on this one. I've been given the task of optimising a website which was built by another company. It's a Wordpress site using Jigoshop. It wasn't in the best of states, and I don't claim that it is now, though we've done quite a bit of work on it. However, after months of work on it, I feel it's not going anywhere. Naturally my customer wants to see results, and they're not getting much. I've submitted the website to a couple of highly regarded directories, and I believe it's listed on some speciality directories (Asian clothing directories). I've also started them on the task of blogging, and I'm trying to get them involved with guest blogging. There's a DA of 1, OSE reports No Data Available for this URL. The Competitive Domain Analysis shows no links at all. I could understand if I started the campaign last week, but we got going in around January this year. One thing that's crossed my mind is that it's quite a slow-loading website. It's with GoDaddy on some cheap hosting plan (not my doing!!). I understand loading times can affect results - could this be the case? It's a bit disheartening really, because we've had really good success with a lot of local businesses' websites but with this one we just can't seem to get it anywhere! 😞 If anyone could shine a bit of light I'd really appreciate it. The URL is bangle-box.co.uk. Thanks in advance!
Moz Pro | | s7media

0
Moz & Xenu Link Sleuth unable to crawl a website (403 error)

It could be that I am missing something really obvious however we are getting the following error when we try to use the Moz tool on a client website. (I have read through a few posts on 403 errors but none that appear to be the same problem as this) Moz Result Title 403 : Error Meta Description 403 Forbidden Meta Robots_Not present/empty_ Meta Refresh_Not present/empty_ Xenu Link Sleuth Result Broken links, ordered by link: error code: 403 (forbidden request), linked from page(s): Thanks in advance!
Moz Pro | | ZaddleMarketing

0
Find out what keywords a site is ranking for

I want to find out what keywords a site is ranking for. In other words, I want to find out how Google is currently seeing a site. I've used Google KW tool - not very accurate when you simply type in the URL. Is there anything in SEOMoz I can use? New to this. thanks
Moz Pro | | goodge

0
Has any on else experienced a spike in crawl errors?

Hi, Since the last time our sites were crawled in SEOmoz they are all showing a spike in Errors. (Mainly duplicate page titles and duplicate content). We haven't changed anything to the structure of the sites but they are all using the same content management system. The image is an example of what we are witnessing for all our sites based on the same system. Is anyone else experiencing anything similar? or does anyone know of any changes that SEOmoz has implemented which may be affecting this? Thanks in advance, Anthony. WzdQV WzdQV WzdQV.jpg WzdQV.jpg
Moz Pro | | BallyhooLtd

1
Crawl Diagnostics - unexpected results

I received my first Crawl Diagnostics report last night on my dynamic ecommerce site. It showed errors on generated URLs which simply are not produced anywhere when running on my live site. Only when running on my local development server. It appears that the Crawler doesn't think that it's running on the live site. For example http://www.nordichouse.co.uk/candlestick-centrepiece-p-1140.html will go to a Product Not Found page, and therefore Duplicate Content errors are produced. Running http://www.nhlocal.co.uk/candlestick-centrepiece-p-1140.html produces the correct product page and not a Product Not Found page Any thoughts?
Moz Pro | | nordichouse

0
Only 1 page has been crawled. Why?

I set a new profile up a fortnight ago. Last week seomoz crawled the entire site (10k pages), and this week has only crawled 1 page. Nothing's changed on the site that I'm aware of, so what's happened?
Moz Pro | | tompollard

0
Potential site architecture problem.

We are experiencing a problem in which category pages do not appear to be passing Page Authority to subcategory pages. See the example below: Home page has PA of 47 http://www.minespress.com/ The next level category page down has a PA of 34 http://www.minespress.com/category/apparel The next sub category down has PA of 1, and shows no links at all in spite of the fact that we got to it from a category link http://www.minespress.com/category/golf-shirts The page in question is crawlable, is indexed and has nothing that we can see that would be prevent it from showing Page Authority and a link profile using your tools. Do we have a site architecture problem?
Moz Pro | | smines

0