Robots.txt being blocked

[SOM24]

I think there is an issue with this website I'm working on here is the URL: http://brownieairservice.com/

In Google Webmaster tools I am seeing this in the Robots.txt tester:

User-agent: *
Crawl-delay: 1
Disallow: /wp-content/plugins/
Disallow: /wp-admin/

Also when I look at "blocked resources" in the webmaster tools this is showing to be blocked:

http://brownieairservice.com/wp-content/plugins/contact-form-7/includes/js/jquery.form.min.js?ver=3.51.0-2014.06.20It looks like the form plug in is giving the issues but I don't understand this.

There are no site errors or URL errors so I don't understand what this crawl delay means or how to fix it.  Any input would be greatly appreciated.  Thank you

[SOM24]

Hi Matt,

Thank you for checking back.  I did change the robot.txt in the dashboard as people suggested but when I go here: http://brownieairservice.com/robots.txt

It is still showing the disallow.  I need to load this:

User-agent: *
Disallow:

to the root folder and I'm not sure how to do that if I need to FTP it or how I do that so that's where I'm at now.

Anybody have any thoughts?  I have googled this question on how to do it and I keep getting put into this loop of information that does not address this questions directly.

Thank you

[MattRoney]

Hi Wendy! Did this get worked out?

[SOM24]

Thanks Dirk for your input I will look at this too and respond back.

[SOM24]

Thank you for your answer.  I went in and uploaded this plug in: WP Robots Txt  Now I can see the robots.txt content.  This is what I see:

User-agent: *
Disallow: /wp-admin/
Disallow: /wp-includes/

I don't see this as I see in Webmaster tools:

User-agent: *
Crawl-delay: 1
Disallow: /wp-content/plugins/
Disallow: /wp-admin/

My question now is this:  is the Disallow:/wp-includes/ the same as Disallow:/wp-content/plugins/

so if I do this: allow:/wp-includes/ then that should solve my issue?

I'm still going through your other suggestions so will type back later on that.  Thank you for your help.

Wendy

[DirkC]

To add to the previous comment - crawl delay is ignored by Googlebot. Check http://tools.seobook.com/robots-txt/

It can be used to limit the speed for the bots - it is however not part of the original robots.txt specification. Since this value is not part of the standard, its interpretation is dependent on the crawler reading it

Yandex: https://yandex.com/support/webmaster/controlling-robot/robots-txt.xml#crawl-delay

Didn't find more info for Bing (they mention it here but do not provide additional info: https://www.bing.com/webmaster/help/how-to-create-a-robots-txt-file-cb7c31ec

If you want to limit the speed for Google bot you have to do it in Webmastertools.

Dirk

[GreenStone]

Wendy,

Google likes to have access to all your css and js. Plugins can contain these files, as seen with your blocked resources message.

The way to fix this would be by removing the Disallow: /wp-content/plugins/ line from your robots.txt file, and thus allowing google full access.

Another solution as provided by a useful article on moz: https://moz.com/blog/why-all-seos-should-unblock-js-css 

"How to unblock your JavaScript and CSS

For most users, it's just a case of checking the robots.txt and ensuring you're allowing all JavaScript and CSS files to be crawled. For Yoast SEO users, you can edit your robots.txt file directly in the admin area of Wordpress.

Gary Illyes from Google also shared some detailed robots.txt changes on Stack Overflow. You can add these directives to your robots.txt file in order to allow Googlebot to crawl all Javascript and CSS.

To be doubly sure you're unblocking all JavaScript and CSS, you can add the following to your robots.txt file, provided you don't have any directories being blocked in it already:

User-Agent: Googlebot

Allow: .js

Allow: .css

If you have a more specialized robots.txt file, where you're blocking entire directories, it can be a bit more complicated.

In these cases, you also need to allow the .js and.css for each of the directories you have blocked.

For example:

User-Agent: Googlebot

Disallow: /deep/

Allow: /deep/*.js

Allow: /deep/*.css

Repeat this for each directory you are blocking in robots.txt."

Hope this helps.

[wearehappymedia]

What problem is it causing Wendy?