Welcome to the Q&A Forum

Hey thank you for your answer, really appreciate it.

Hello everyone !

I need help setting up a robot.txt.

I'm trying to block all pdf files in particular directories so I'm using this command. In the example below the line is blocking all .gif in the entire site.

Block files of a specific file type (for example, .gif) | Disallow: /*.gif$

2 questions :

• Can I use this command to specify one particular directory in which I want to block pdf files ? Will this line be recognized by googlebots ?

Disallow: /fileadmin/xxxxxxx/xxx/xxxxxxx/*.pdf$

• Then I realized that I would have to write as many lines as many directories there are in which I want to block pdf files.

Let's say I want to block pdf files in all these 3 directories

/fileadmin/directory1

/fileadmin/directory1/sub1

/fileadmin/directory1/sub1/pdf

Is there a pattern-matching rule I could use to blocks access to pdf files in all subdirectories instead of writing 3x the above line for each subdirectory ? For exemple :

Disallow: /fileadmin/directory1*/

Many thanks in advance for any insight you may have.

Hi Rob !

Thank you very much for taking the time to write a really detailed and clear answer, I really appreciate it.

This was insightful and answered my question.

Cheers.

Hi everyone

Just out of curiosity, what would happen if in my disavow report I have this line :

domain:example.com**/** instead of domain:example.com as recommended by google.

I was just wondering if adding a / at the end of a domain would automatically render the line invalid and ignored by Google's disavow backlinks tool.

Many thanks for your thoughts

Thanks for the response!

These are the pages it crawled.

<colgroup><col width="420"></colgroup>
| http://redken.com |
| http://www.redken.com/ |
| http://www.redken.com/international/ |
| http://www.redken.com/USA |
| http://www.redken.com/?country=USA |

Robots.txt looks clean, nothing that should have stopped it from crawling more.

I'm trying to crawl Redken.com in Moz Analytics and the Search Diagnostics is only crawling 4 pages. The domain uses a "select your country" the first time you visit, and it seems as though the bot is not getting beyond that (aka, not clicking on "USA") and is therefore not crawling the rest of the domain. There is no country specific URL other than redken.com.

I've tried entering both "redken.com" and "www.redken.com" as the URL, but no luck.

Any tips?

G'd everyone,

I need help with understanding how special characters impact SEO.  Eg. é , ë ô in words

Does anyone have good insights or reference material regarding the treatment of Special Characters by Google Search Engine?

• how Page Title / Meta Desc with Special Chars are being index  & Crawl

• Best Practices when it comes to URLs -

• uses of Unicode, HTML entity references - when are where?

• any disadvantage using special characters

• Does special characters in URL have any impact on SEO performance & User search, experience.

Thanks heaps,

Amy

HI All,

I was wondering if Moz has something that allows me to "Bulk download of crawl diagnostics csv files for multiple campaigns"?  Or are there APIs out there?

I am trying to download all the campaigns diagnostics (High & Medium crawl-issues) on a regular basis, then with these cvs files then run some bulk extracts etc etc ... Appreciate any hint?

Thank heaps

Amy