How to Block Urls with specific components from Googlebot
-
Hello,
I have around 100,000 Error pages showing in Google Webmaster Tools. I want to block specific components like com_fireboard, com_seyret,com_profiler etc.
Few examples:
I tried blocking using robots.txt. Just used this
Disallow: /com_fireboard/
Disallow: /com_seyret/But its not working. Can anyone suggest me to solve this problem.
Many Thanks
Shradda
-
I agree with Sha that your 404 page has a nice appearance. My main concern is it lacks functionality.
If I click on a link to your site and end up on that page, what is my next action? Likely I would hit the <back>button on my browser and leave your site. It is either that or typing a URL.</back>
I recommend you offer users the option to stay on your site. Your site navigation, a search box, some links, anything would be helpful.
-
Hi Shradda,
I agree with Ryan that the use of a meta noindex tag is the preferable way to block the pages, but obviously there may be difficulties with applying the tag, depending upon how your pages are generated and whether you are able to alter the code or not.
You can also use ?option=com_fireboard etc to create 301 redirects back to a higher order category page or search.
You should be able to use a single line of code to 301 all pages within each directory.
Using 301 redirects will also send a signal to search engines to de-index those pages.
Very clever 404 page too! Had to watch him go all the way across the page and back just so I knew I wasn't missing anything!
Sha
-
You can log into Google Webmaster Tools and adjust your parameter settings. It was designed for this exact purpose. Site Parameters > URL Parameters. If you use this solution, be sure to do the same in Bing WMT as well.
A better solution would be to noindex the pages. Using robots.txt should be avoided when possible.
If you do need to use robots.txt, your current disallow statement is set up to not crawl the folder named "com_fireboard". You intention is to not crawl the parameter ?option=com_fireboard. I know wildcards work for the trailing portion of a path but I have not tried them for the beginning part of the path.
I suggest you try the following:
Disallow: ?option=com_fireboard
For more on the robots.txt file, please view the following site: http://www.robotstxt.org/
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Removed URLs
recently my site has got some problem some of my URLs are repeating in the SERP ! I removed them by search console and also site : but they show up again Does anyone know what is wrong?
Technical SEO | | talaabshode20200 -
Folders in url structure?
Hello, Revamping an out-of-date website and am wondering if I need to include the folders (categories) in the url structure? The proposed structure has 8 main folders. I've been reading that Google is ok if the folder is not included in the url, but is it really? The hesitation I have is that the urls are getting long and the main folder only has only a sub folder beneath it. So, /folder-name/facility-name/treatment-overview. This looks too long, doesn't it? Thanks!
Technical SEO | | lfrazer1230 -
WP URL issue - Concatenated URLs (LOTS of them)
WP is doing this somehow, and creating URLs for hundreds of pages that don't exist. HOW is this happening, and how do I stop It? I have many, many URLS like this: https://www.atouchofrust.com/terms-of-use/atouchofrust.com/vendor-news. Of note, atouchofrust.com/terms-of-use, and atouchofrust.com/vendor-news are both legit pages on the site. Why they are being concatenated is beyond my limited understanding of WP. Please, somebody, help. Cori
Technical SEO | | FlyingC0 -
Special characters in URL
Will registered trademark symbol within a URL be bad? I know some special characters are unsafe (#, >, etc.) but can not find anything that mentions registered trademark. Thanks!
Technical SEO | | bonnierSEO0 -
Case sensitive url's
Hi, Really appreciate advice on this one in advance! We had a problem with case sensitive urls (eg: /web-jobs or /Web-jobs) We added a code to convert all urls into lowercase letters and added 301 redirection. We are now experiencing problems with duplicate page content. Each time a url contains caps letter it is converted and redirected to small letter url. I can convert all urls into lowercase letters (all places) but the problem now is google have already indexed urls so they may cause duplicate content issue. The solution: Remove 301 redirection added to convert url into small letter. Add canonical url which converts url into complete small letter, so google index content only from canonical url. But I am little confused about what will happen to already indexed pages with caps in url. Appreciate any advice you can give? Simon
Technical SEO | | simmo2350 -
What is your opinion on ideal url structure?
Which url structure do you think is better... website.com/CO/denver/555-your-address-way-denver-co-55678/98347578 or website.com/classifying-keyword/555_your_address_way-denver_colorado-55678-98347578 orrr website.com/classifying-keyword/555-your-address-way-denver-colorado-55678-98347578
Technical SEO | | jessefriedman0 -
Block url with dynamic text in
I've just ran a report and I have a lot of duplicate page titles, most of which seem to be the review page, I use Magento and my normal url would be something like blah-blahtext.html but the review url is something like blah-blahtext/reviews/category/categoryname So I want to block the /reviews url bit as no one ever leaves reviews and it's not something I will be using in the future. Also I have a dynamic navigation which creates urls that look like product-name.html?size=2&colour=14 these are also creating duplicate urls, anyway to fix this? While I'm asking, anyone any tips for Magento?
Technical SEO | | Beermonster0 -
How to change URL of RSS Feed?
Hi, There are some websites that keeps on scraping my content. I have blocked them already from accessing my website using .htaccess but they still get my content via RSS feed. I have tried delaying the RSS feed but I think this affected google rankings. My question is, is there a way to change the URL of my RSS Feed? From: http://www.mysite.com/feed to http://www.mysite.com/feed2
Technical SEO | | Trigun0