Blocking out specific URLs with robots.txt
-
I've been trying to block out a few URLs using robots.txt, but I can't seem to get the specific one I'm trying to block. Here is an example.
I'm trying to block
but not block
It seems if it setup my robots.txt as so..
Disallow: /cats
It's blocking both urls. When I crawl the site with screaming flog, that Disallow is causing both urls to be blocked. How can I set up my robots.txt to specifically block /cats? I thought it was by doing it the way I was, but that doesn't seem to solve it.
Any help is much appreciated, thanks in advance.
-
Do not play with Robots as it may block out series of pages and folders out of index
Correct command as stated by Lesley is /cats/ . Refer official documentation
https://developers.google.com/webmasters/control-crawl-index/docs/robots_txt
-
You can either use /cats/ or /cats/* that should just block the cats folder and not the other folder. Note the first use is the preferred one.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
URL Structure for geo location for specific page
On hackerearth.com/challenges page, there is an option to select languages. This option is in the footer. Once you select the language the url changes. Ex - if we select French, the URL changes to hackereath.com/fr/challenges. In case we decide to change the URL of this page with Geo, what should be the URL structure which accommodates languages as well. My research says that it would good to keep the url like domainname.com/page/language.
Intermediate & Advanced SEO | | Rajnish_HE0 -
Changing URLs
URLs of my web pages are based on the titles of pages. For sampel, if a title page is called "product ABC", then the URL for this page is /product-abc. Google and all other search engines have indexed all pages. Now I want to change the titles of some sites. Should I change the URLs accordingly, or should I rather leave URLs as they are. SEO Best Practice says that keywords must be placed both in the title, and in the URL. I think that Google will think that pages have douplicate content with diffrent titles, and it comes to many 404 error, if I change the URLs. What do you recommend in this case?
Intermediate & Advanced SEO | | kian_moz0 -
Robots.txt issue for international websites
In Google.co.uk, our US based (abcd.com) is showing: A description for this result is not available because of this site's robots.txt – learn more But UK website (uk.abcd.com) is working properly. We would like to disappear .com result totally, if possible. How to fix it? Thanks in advance.
Intermediate & Advanced SEO | | JinnatUlHasan0 -
Does having a ? on the end of your URL affect your SEO?
I have some redirects that were done with at "?" at the end of the URL to include google coding (i.e. you click on an adwords link and the google coding follows the redirected link). When there is not coding to follow the link just appears as "filename.html?". Will that affect us negatively SEO-wise? Thank you.
Intermediate & Advanced SEO | | RoxBrock1 -
Hash URLs
Hi Mozzers, Happy Friday! I have a client that has created some really nice pages from their old content and we want to redirect the old ones to the new pages. The way the web developers have built these new pages is to use hashbang url's for example www.website.co.uk/product#newpage My question is can I redirect urls to these kind of pages? Would it be using the .htaccess file to do it? Thanks in advance, Karl
Intermediate & Advanced SEO | | KarlBantleman0 -
What is the best URL structure for categories?
A client's site currently uses the URL structure: www.website.com/�tegory%/%postname% Which I think is optimised fairly well, as the categories are keywords being targeted. However, as they are using a category hierarchy, often times the URL looks like this: www.website.com/parent-category/child-category/some-post-titles-are-quite-long-as-they-are-long-tail-terms Best practise often dictates (such as point 3 in this Moz article) that shorter URLs are better for several reasons. So I'm left with a few options: Remove the category from the URL Flatten the category hierarchy Shorten post titles two a word or two - which would hurt my long tail search term traffic. Leave it as it is What do we think is the best route to take? Thanks in advance!
Intermediate & Advanced SEO | | underscorelive0 -
URL for New Product
Hi, We run an established website (mindflash.com) selling online training software. We are getting ready to launch a new section of the site where our users can sell their own online training programs. This will be branded as the 'marketplace'. This section will have a main page, category pages, tag pages, search and individual course pages. In our URL structure, I'd love to target the word 'training courses' but I don't want to neglect the product brand either. Is it better to use /training-courses in the marketplace urls or to use /marketplace? Or is it better to use both like /marketplace-training-courses or /marketplace/training-courses? Option 1: Example main section page: mindflash.com/training-courses Example category page: mindflash.com/training-courses/software-training Option 2: Example main section page: mindflash.com/marketplace Example category page: mindflash.com/marketplace/software-training Option 3: Example main section page: mindflash.com/marketplace-training-courses Example category page: mindflash.com/marketplace-training-courses/software-training Option 4: Example main section page: mindflash.com/marketplace/training-courses Example category page: mindflash.com/marketplace/training-courses/software-training Which option is better and why?
Intermediate & Advanced SEO | | mindflash0 -
Should I shorten my urls?
For my informational site I have a lot of urls that are way too long. When I first created the site, I wrote a script that takes out the common words of a post and fashions a url. So, for example, if the first few words of a question were: Hi there, I have a question about back pain. I'm wondering what drugs would be good for relief and how I can get some help? then my url may be: www.mydomain.com/question?id=123-question-back-pain-wondering-drugs-good-relief-how-get-some-help Once I got learning about seo I realized that these urls were too long but I never did anything about them. Should I be shortening these, or is my time best spent doing something else?
Intermediate & Advanced SEO | | MarieHaynes2