Disallow robots on a url effect?
-
Hello, I am using wordpress and on some of my top tag urls e.g : http://www.designzzz.com/tag/brushes/ there is an avg page rank of 4.
I was reading somewhere that my huge jump in not selected count in index status in GWT is becuase of tags/categories urls.
So i added disallow in my robots.txt for the url /tag
What sort of effect will it have on my tag urls rankings or PR ?
-
If your site is ranking "well", well enough for you I mean don't change anything if users are finding there way to what they want. Now for the noindex-follow tag that will not rank the certain pages in the serps but the links on the page will pass pagerank . For the nofollow-noindex or nofollow-index will just hurt the site rankings as they are average pagerank 4 cutting of the power of the posts in the certain category-tag pages not passing any link juice. The best thing to do is leave the pages as is. If the category-tag pages rank better than the posts then i would just noindex them leaving follow on the pages to pass juice towards the posts.
-
but the problem in my case is that for example tags like font, brushes. doesn't have there seperate categories. like font related posting goes under freebies categ and brushes related posts goes under photoshop category . Photoshop categories lots of types of postings like tutorials, downloads, brushes, actions etc.
What i am concerned right now is that if i noindex tag pages. how will they affect my pr and rnakings of these pages?
Or if i nofollow them in meta tag, how will that effect ?
-
The fix i did was to 301 redirect the tag pages to the similar category pages or content pages. This might not be the right thing for your site but I saw new rankings within a day or two. I think this might have lifted some kind of penalty and also siloed the pagerank of the tag pages towards the categories and articles giving them a boost in serps.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Dark Traffic & Long URLs - Clarification
Hi everyone, I've been reading the 2017 report by Groupon and the 2017 article by Neil Patel r.e. dark traffic. Both of these articles work on the assumption that most long URLs are not direct traffic because people wouldn't type a long URL into their browser. However, what happens to me personally all the time is that I start typing a URL into the browser, and the browser brings up a list of pages I've visited recently, and I click on some super long URL that I didn't bookmark but have visited in the past. That is legitimate direct traffic, but it's a long URL. I'm just wondering if there's something flawed in my reasoning or in the reasoning of Patel and Groupon. Maybe most people aren't relying on browsers like I am, or maybe things have changed a lot in the past 3 years. What do you think? And are there any more recent resources/articles that you would recommend r.e. trying to parse out dark traffic? https://neilpatel.com/blog/dark-traffic-stealing-data/ Thanks!
Search Behavior | | LivDetrick0 -
Is there a tool that will find a string from the source code from a list of URLs?
I need a tool that will run through a long list of URLs (from multiple domains), look for a specific string in the source code and then produce a CSV showing in which URLs the the string was found. This is for legitimate research, not for any nefarious purposes Any ideas of such a tool already exists?
Search Behavior | | richdan0 -
Help with robots.txt on Magento
Hi everybody, I need your help in order to fix some problems with HTML errors and Crawling errors generated by Magento on my client's website www.casabiancheria.it I have some problems with duplicate meta informations due to the fact that there are a lot of links such as /stampe-romagnole/tovaglie-con-tovaglioli**/colore/**beige,marrone,giallo,lilla/show/all.html /stampe-romagnole/tovaglie-con-tovaglioli**/colore/**beige,marrone,lilla/show/all.html that are generated by the filter /colore/ and so they have duplicate content and meta information on them. I activated the canonicals on Magento but this hasn't fixed the problem yet. On the sitemap there are only 1 link for each product, so it seems that the canonicals are working, but bot Google Webmaster Tools and SEO Moz are giving me errors on duplicate content and meta informations. I would like to solve these problems by excluding from robots.txt all the urls that contain the filter parameters, such as /colore/, /price/, /dimensions/, etc. (take a look to the attachment). I tried different solutions in order to exclude these links from robots, but I wasn't able to succeed. Below you can find my current robots.txt... can someone help me in order to write the correct form of this file and finally exclude all these urls generated by filters on Magento? Finally, is it worth it to exclude also the images from Magento? (take a look to the final lines of the robots below). Thank you very much for your help! Alberto User-agent: *
Search Behavior | | OptimizedGroup
Disallow: /CVS
Disallow: /.svn$
Disallow: /.idea$
Disallow: /.sql$
Disallow: /.tgz$
Disallow: /w1nL1f3L0g1c/
Disallow: /app/
Disallow: /downloader/
Disallow: /errors/
Disallow: /includes/
Disallow: /lib/
Disallow: /pkginfo/
Disallow: /shell/
Disallow: /var/
Disallow: /404/
Disallow: /cgi-bin/
Disallow: /magento/
Disallow: /report/
Disallow: /scripts/
Disallow: /shell/
Disallow: /skin/
Disallow: /stats/
Disallow: /api.php
Disallow: /cron.php
Disallow: /cron.sh
Disallow: /error_log
Disallow: /get.php
Disallow: /install.php
Disallow: /LICENSE.html
Disallow: /LICENSE.txt
Disallow: /LICENSE_AFL.txt
Disallow: /README.txt
Disallow: /RELEASE_NOTES.txt
Disallow: /?dir
Disallow: /?dir=desc
Disallow: /?dir=asc
Disallow: /?limit=all
Disallow: /?mode*
Disallow: /index.php/
Disallow: /?SID=
Disallow: /checkout/
Disallow: /onestepcheckout/
Disallow: /customer/
Disallow: /customer/account/
Disallow: /customer/account/login/
Disallow: /catalogsearch/
Disallow: /catalog/product_compare/
Disallow: /catalog/category/view/
Disallow: /catalog/product/view/
Disallow: /cgi-bin/
Disallow: /cleanup.php
Disallow: /apc.php
Disallow: /memcache.php
Disallow: /phpinfo.php
Disallow: /control/
Disallow: /customize/
Disallow: /newsletter/
Disallow: /poll/
Disallow: /review/
Disallow: /sendfriend/
Disallow: /tag/
Disallow: /wishlist/
Disallow: /catalog/product/gallery/
Disallow: /?*
Disallow: //colore/
Disallow: //price/
Disallow: //misura/
Disallow: //marca/
Disallow: //sort-by/
Disallow: //combinazione/
Disallow: /*/seleziona-colore/
Disallow: /colore/
Disallow: /price/
Disallow: /misura/
Disallow: /marca/
Disallow: /sort-by/
Disallow: /combinazione/
Disallow: /seleziona-colore/
Disallow: /*colore/
Disallow: /*price/
Disallow: /*misura/
Disallow: /*marca/
Disallow: /*sort-by/
Disallow: /*combinazione/
Disallow: /*seleziona-colore/ UmuEX4z0 -
GWMT - "Tag Site For Child Directed Treatment" Effect On Search / Rankings?
Hi All, We have a client who has been directed to tag their site for "Child Directed Treatment" in Webmaster tools to comply with AdExchange policies. The site is, generally speaking, directed at those between the ages of 13 and 16 along with their parents, but does NOT collect any data (No sign in, login, signups etc). You can find out more about the specific tag here (unfortunately not much more about it) https://support.google.com/webmasters/answer/3221080?hl=en Our concern is that we have never heard of this specific tag, and our client is asking us to let them know if that this will have no effect on search traffic or ranking. I can't find much in the way of anyone who HAS implemented this tag and the effects it has had on their site. They are ad supported, receive millions of unique hits a month, and the majority of their traffic is from branded keywords. Would love to hear from anyone with ANY experience or thoughts on this process and what to be aware of. Your assistance is muchly appreciated.
Search Behavior | | SearchMarketers0 -
Two sites with different URLs
I help organize an ecommerce site for a company that is named after the state that the company was founded in (ex: Florida Pipes). Our programmer is thinking about creating a duplicate site that would have a name that was more location agnostic so people shopping on the site would not think that they may be ordering from far away hence incurring extra shipping charges. He said that the site would only have a different URLs, name and homepage but would link to all of the same stuff that is on the site that is up now. He said that there would be no way to tell (possibly for the layperson) that the two sites were related. Is this a good practice? Would we be penalized in search results for having two URLs linking to the same content? (We are a 30 year old company that ranks very high for our main keyword) Thank you for your input!
Search Behavior | | Winoman0 -
How long til meta robots noindex takes effect?
I have a wordpress site with about 3,000 posts and over 1,000 tags. All of the tag archives are currently indexed in Google and I don't want them to be. I just set the meta robots to no-index all the tag archives and was wondering how long it will take til they're out of the search engines? Since there are close to 1,500 of these and they are duplicate content it would be nice to have them gone asap. I noticed Webmaster Tools allows me to resubmit my site to index if my site has changed significantly... should I try that?? Any other advice would be greatly appreciated!
Search Behavior | | gfreeman230 -
Trailing slash at end of URLs?
Hi, I'm just about to put up a new site and I need to decide between having no trailing slash at the end of the all the URLs, or putting one in there. I think Matt Cutts has a slight preference for them, as stated here: http://www.seomoz.org/blog/whiteboard-interview-googles-matt-cutts-on-redirects-trust-more "Matt says he would slightly advocate for using a trailing slash simply because it clearly indicates that a URL is a folder and not a document." However, I'm really wondering. I mean, if people link to my site, they'll tend not to insert a trailing slash, I'm thinking... Your thoughts would be welcome on this one! Cheers, Luke
Search Behavior | | McTaggart0 -
Where can I find research into consumer behavior in URL input?
I have a hypothesis that the % of users that type URLs is decreasing. Instead I think users will be arriving at sites by clicking on links in emails, search and social. Where can I find data / research to back me up?
Search Behavior | | TomCritchlow0