Robots.txt Allowed

[ThomasHarvey]

Hello all,

We want to block something that has the following at the end:

http://www.domain.com/category/product/some+demo+-text-+example--writing+here

So I was wondering if doing:

/*example--writing+here

would work?

[GlobeRunner]

Yes, that should work just fine. As Logan mentioned, I recommend you test it in the robots.txt testing tool in Google Search Console.

[CPR_PTANTONO]

Yes, that would work. I'm sure everyone already knows that if in case you have a product that has the word example at the end of URL, it would block that too. A little off tangent here but blocking in robots.txt does not mean that every single spiders out there is going to honor this rule. The major ones like Google Spiders does honor this. Also, it doesn't mean that the URL won't be indexed. Sorry for the long winded answer but just make sure that if this is truly an example or demo page that you don't want search engines to index to make sure that you include "noindex, nofollow" in the metainfo.

I agree with Logan Ray. In case you want the "Robots TXT" Tester, you can google it "Robots Txt Tester" and the first one should be from support.google.com

[LoganRay]

Hi Thomas,

That should work. You can confirm this by modifying your robots.txt file in Search Console and testing a handful of URLs to ensure they're blocked the way you want.