Are robots.txt wildcards still valid? If so, what is the proper syntax for setting this up?
-
I've got several URL's that I need to disallow in my robots.txt file. For example, I've got several documents that I don't want indexed and filters that are getting flagged as duplicate content. Rather than typing in thousands of URL's I was hoping that wildcards were still valid.
-
Great job. I just wanted to add this from Google Webmasters
http://googlewebmastercentral.blogspot.com/2008/06/improving-on-robots-exclusion-protocol.html
and this from Google Developers
https://developers.google.com/webmasters/control-crawl-index/docs/robots_txt
-
Yup wildcard syntax is indeed still valid. However I can only confirm that the big 3 (Google, Yahoo and Bing) actively observe it. Other secondary search engines may not.
In your case you are probably looking for a syntax along the lines of:
User-agent: *
Disallow: /*.pdf$ This would set that any user agent should be blocked from any file name that ends in .pdf (a $ ties it to the end so pdf.txt would not be blocked in this case)Keep an eye on how you block them. Missing a trailing slash could block a directory rather than a file, or not appending a strict symbol ($) could mean that phrases throughout a directory could be blocked rather than just a filename.
Also keep in mind if you are using URL re-writing this may play into how you need to block things; and you may also want to remember that disallowing access in a robot.txt does NOT prevent search engines from indexing the data, it is up to them if they honor the request. So if it is very important to block the file access from search engines then robots.txt may not be the way to do it.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Unsolved Google Search Console Still Reporting Errors After Fixes
Hello, I'm working on a website that was too bloated with content. We deleted many pages and set up redirects to newer pages. We also resolved an unreasonable amount of 400 errors on the site. I also removed several ancient sitemaps that listed content deleted years ago that Google was crawling. According to Moz and Screaming Frog, these errors have been resolved. We've submitted the fixes for validation in GSC, but the validation repeatedly fails. What could be going on here? How can we resolve these error in GSC.
Technical SEO | | tif-swedensky0 -
Blocking subdomains with Robots.txt file
We noticed that Google is indexing our pre-production site ibweb.prod.interstatebatteries.com in addition to indexing our main site interstatebatteries.com. Can you all help shed some light on the proper way to no-index our pre-prod site without impacting our live site?
Technical SEO | | paulwatley0 -
Robots.txt issue - site resubmission needed?
We recently had an issue when a load of new files were transferred from our dev server to the live site, which unfortunately included the dev site's robots.txt file which had a disallow:/ instruction. Bad! Luckily I spotted it quickly and the file has been replaced. The extent of the damage seems to be that some descriptions aren't displaying and we're getting a message about robots.txt in the SERPs for a few keywords. I've done a site: search and generally it seems to be OK for 99% of our pages. Our positions don't seem to be affected right now but obviously it's not great for the CTRs on those keywords affected. My question is whether there is anything I can do to bring the updated robots.txt file to Google's attention? Or should we just wait and sit it out? Thanks in advance for your answers!
Technical SEO | | GBC0 -
Setting preferred domain as www or none www
Way back before panda I used to rank for certain keywords pretty well. Of course like many others after panda I lost some of those rankings. I have been getting better since then so its not that bad. I was poking around in Google Webmaster Tools and I noticed something which I need some clarification in. History my site freescrabbledictionary.com used to be indexed as a none www. Then some time ago I can't remember when I set it to www. Tonight I was looking through my webmaster tools and I noticed something that did not make sense to me. In my content keywords section for the none www my list is as follows Content Keywords <form action="https://www.google.com/webmasters/tools/keywords-list?hl=en&siteUrl=http://freescrabbledictionary.com/" method="GET"> Keyword Significance 1. scrabble 2. words (2 variants) 3. dictionary 4. cheat 5. finder 6. friends 7. maker (2 variants) 8. noun 9. letter (2 variants) 10. hasbro 11. mattel 12. spear 13. found (2 variants) 14. sowpods 15. freescrabbledictionary 16. builder 17. affiliated 18. search 19. solver 20. lists </form> Then I looked at my www lists and its Content Keywords <form action="https://www.google.com/webmasters/tools/keywords-list?hl=en&siteUrl=http://www.freescrabbledictionary.com/" method="GET"> Keyword Significance 1. words (3 variants) 2. scrabble (2 variants) 3. letter (4 variants) 4. points 5. cheat (3 variants) 6. friends (2 variants) 7. finder (2 variants) 8. anagram (2 variants) 9. dictionary 10. tool (2 variants) 11. hasbro 12. mattel 13. spear 14. game (4 variants) 15. mobile 16. affiliated (3 variants) 17. berkshire 18. canada 19. calculations (5 variants) 20. coming (4 variants) </form> My none www version has the order (especially the first 5 keywords) that I want, my www version is no were near it. If I change back to the none www version could I possible see an change in rank? or can it effect it if I change it? I am starting to think I shot myself in the foot when I switched...
Technical SEO | | cbielich0 -
Blocked by meta-robots but there is no robots file
OK, I'm a little frustred here. I've waited a week for the next weekly index to take place after changing the privacy setting in a wordpress website so Google can index, but I still got the same problem. Blocked by meta-robots, no index, no follow. But I do not see a robot file anywhere and the privacy setting in this Wordpress site is set to allow search engines to index this site. Website is www.marketalert.ca What am I missing here? Why can't I index the rest of the website and is there a faster way to test this rather than wait another week just to find out it didn't work again?
Technical SEO | | Twinbytes0 -
301 Redirect Properly To Keep the Juice
I have a bunch of WP Blogs and was thinking of taking all linkjuice from these to my main money site. The most of the other WP Blogs is hosted at godaddy.com (domain and site) and I know they have a URL Redirects page in site manager but I`m not sure this is the right way to go. Also I wonder some of these sites have hundreds of blogposts there is no way I can "re-create" those on the money site but I am sure that is not a must-thing to do in order to keep the "juice" right or wrong? Last but not least, I was wondering if you think it would be best to redirect the sites to relevant pages on money sites. For instance if i had a domain called cheap-ties.com with 100 blogposts about this and on money site a webshop with a category called ties, should redirect to this or to main domain or doesnt it matter?
Technical SEO | | fAgBxa8b0 -
Robots.txt file getting a 500 error - is this a problem?
Hello all! While doing some routine health checks on a few of our client sites, I spotted that a new client of ours - who's website was not designed built by us - is returning a 500 internal server error when I try to look at the robots.txt file. As we don't host / maintain their site, I would have to go through their head office to get this changed, which isn't a problem but I just wanted to check whether this error will actually be having a negative effect on their site / whether there's a benefit to getting this changed? Thanks in advance!
Technical SEO | | themegroup0 -
Why are old versions of images still showing for my site in Google Image Search?
I have a number of images on my website with a watermark. We changed the watermark (on all of our images) in May, but when I search for my site getmecooking in Google Image Search, it still shows the old watermark (the old one is grey, the new one is orange). Is Google not updating the images its search results because they are cached in Google? Or because it is ignoring my images, having downloaded them once? Should we be giving our images a version number (at the end of the file name)? Our website cache is set to 7 days, so that's not the issue. Thanks.
Technical SEO | | Techboy0