Hi all,

We've recently migrated a Wordpress website from staging to live, but the robots.txt was deleted.  I've created the following new one:

User-agent: *
Allow: /
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /wp-content/plugins/
Disallow: /wp-content/cache/
Disallow: /wp-content/themes/
Allow: /wp-admin/admin-ajax.php

However, in the site audit on SemRush,  I now get the mention that a lot of pages have issues with blocked internal resources in robots.txt file. These blocked internal resources are all cached and minified css elements: links, images and scripts.

Does this mean that Google won't crawl some parts of these pages with blocked resources correctly and thus won't be able to follow these links and index the images? In other words, is this any cause for concern regarding SEO?

Of course I can change the robots.txt again, but will urls like https://example.com/wp-content/cache/minify/df983.js end up in the index?

Thanks for your thoughts!

I mostly agree with Robin here.

Also, be sure NOT to mix 'noindex' and canonical tags. Google will (in most cases) end up picking rel=canonical over noindex when you use both of these. So it is very possible that even when using 'noindex', your pages will appear in search results.

The approach of canonicalising all your paginated pages to the first one, is not good practice. We all just found out that Google hasn't been using rel=next/prev for a couple of years now, but most of the pagination was indexed in a correct way.

So doing nothing is maybe not that bad of an option. If you see things going wrong, you can further evaluate and test other possibilities.

Hi,

I would like to block all URLs with the parameter '?filter=' from being crawled by including them in the robots.txt.

Which directive should I use:

User-agent: *
Disallow: ?filter=

or

User-agent: *
Disallow: /?filter=

In other words, is the forward slash in the beginning of the disallow directive necessary?

Thanks!