Disallow robots on a url effect?
-
Hello, I am using wordpress and on some of my top tag urls e.g : http://www.designzzz.com/tag/brushes/ there is an avg page rank of 4.
I was reading somewhere that my huge jump in not selected count in index status in GWT is becuase of tags/categories urls.
So i added disallow in my robots.txt for the url /tag
What sort of effect will it have on my tag urls rankings or PR ?
-
If your site is ranking "well", well enough for you I mean don't change anything if users are finding there way to what they want. Now for the noindex-follow tag that will not rank the certain pages in the serps but the links on the page will pass pagerank . For the nofollow-noindex or nofollow-index will just hurt the site rankings as they are average pagerank 4 cutting of the power of the posts in the certain category-tag pages not passing any link juice. The best thing to do is leave the pages as is. If the category-tag pages rank better than the posts then i would just noindex them leaving follow on the pages to pass juice towards the posts.
-
but the problem in my case is that for example tags like font, brushes. doesn't have there seperate categories. like font related posting goes under freebies categ and brushes related posts goes under photoshop category . Photoshop categories lots of types of postings like tutorials, downloads, brushes, actions etc.
What i am concerned right now is that if i noindex tag pages. how will they affect my pr and rnakings of these pages?
Or if i nofollow them in meta tag, how will that effect ?
-
The fix i did was to 301 redirect the tag pages to the similar category pages or content pages. This might not be the right thing for your site but I saw new rankings within a day or two. I think this might have lifted some kind of penalty and also siloed the pagerank of the tag pages towards the categories and articles giving them a boost in serps.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Dark Traffic & Long URLs - Clarification
Hi everyone, I've been reading the 2017 report by Groupon and the 2017 article by Neil Patel r.e. dark traffic. Both of these articles work on the assumption that most long URLs are not direct traffic because people wouldn't type a long URL into their browser. However, what happens to me personally all the time is that I start typing a URL into the browser, and the browser brings up a list of pages I've visited recently, and I click on some super long URL that I didn't bookmark but have visited in the past. That is legitimate direct traffic, but it's a long URL. I'm just wondering if there's something flawed in my reasoning or in the reasoning of Patel and Groupon. Maybe most people aren't relying on browsers like I am, or maybe things have changed a lot in the past 3 years. What do you think? And are there any more recent resources/articles that you would recommend r.e. trying to parse out dark traffic? https://neilpatel.com/blog/dark-traffic-stealing-data/ Thanks!
Search Behavior | | LivDetrick0 -
Is there a tool that will find a string from the source code from a list of URLs?
I need a tool that will run through a long list of URLs (from multiple domains), look for a specific string in the source code and then produce a CSV showing in which URLs the the string was found. This is for legitimate research, not for any nefarious purposes Any ideas of such a tool already exists?
Search Behavior | | richdan0 -
Help with robots.txt on Magento
Hi everybody, I need your help in order to fix some problems with HTML errors and Crawling errors generated by Magento on my client's website www.casabiancheria.it I have some problems with duplicate meta informations due to the fact that there are a lot of links such as /stampe-romagnole/tovaglie-con-tovaglioli**/colore/**beige,marrone,giallo,lilla/show/all.html /stampe-romagnole/tovaglie-con-tovaglioli**/colore/**beige,marrone,lilla/show/all.html that are generated by the filter /colore/ and so they have duplicate content and meta information on them. I activated the canonicals on Magento but this hasn't fixed the problem yet. On the sitemap there are only 1 link for each product, so it seems that the canonicals are working, but bot Google Webmaster Tools and SEO Moz are giving me errors on duplicate content and meta informations. I would like to solve these problems by excluding from robots.txt all the urls that contain the filter parameters, such as /colore/, /price/, /dimensions/, etc. (take a look to the attachment). I tried different solutions in order to exclude these links from robots, but I wasn't able to succeed. Below you can find my current robots.txt... can someone help me in order to write the correct form of this file and finally exclude all these urls generated by filters on Magento? Finally, is it worth it to exclude also the images from Magento? (take a look to the final lines of the robots below). Thank you very much for your help! Alberto User-agent: *
Search Behavior | | OptimizedGroup
Disallow: /CVS
Disallow: /.svn$
Disallow: /.idea$
Disallow: /.sql$
Disallow: /.tgz$
Disallow: /w1nL1f3L0g1c/
Disallow: /app/
Disallow: /downloader/
Disallow: /errors/
Disallow: /includes/
Disallow: /lib/
Disallow: /pkginfo/
Disallow: /shell/
Disallow: /var/
Disallow: /404/
Disallow: /cgi-bin/
Disallow: /magento/
Disallow: /report/
Disallow: /scripts/
Disallow: /shell/
Disallow: /skin/
Disallow: /stats/
Disallow: /api.php
Disallow: /cron.php
Disallow: /cron.sh
Disallow: /error_log
Disallow: /get.php
Disallow: /install.php
Disallow: /LICENSE.html
Disallow: /LICENSE.txt
Disallow: /LICENSE_AFL.txt
Disallow: /README.txt
Disallow: /RELEASE_NOTES.txt
Disallow: /?dir
Disallow: /?dir=desc
Disallow: /?dir=asc
Disallow: /?limit=all
Disallow: /?mode*
Disallow: /index.php/
Disallow: /?SID=
Disallow: /checkout/
Disallow: /onestepcheckout/
Disallow: /customer/
Disallow: /customer/account/
Disallow: /customer/account/login/
Disallow: /catalogsearch/
Disallow: /catalog/product_compare/
Disallow: /catalog/category/view/
Disallow: /catalog/product/view/
Disallow: /cgi-bin/
Disallow: /cleanup.php
Disallow: /apc.php
Disallow: /memcache.php
Disallow: /phpinfo.php
Disallow: /control/
Disallow: /customize/
Disallow: /newsletter/
Disallow: /poll/
Disallow: /review/
Disallow: /sendfriend/
Disallow: /tag/
Disallow: /wishlist/
Disallow: /catalog/product/gallery/
Disallow: /?*
Disallow: //colore/
Disallow: //price/
Disallow: //misura/
Disallow: //marca/
Disallow: //sort-by/
Disallow: //combinazione/
Disallow: /*/seleziona-colore/
Disallow: /colore/
Disallow: /price/
Disallow: /misura/
Disallow: /marca/
Disallow: /sort-by/
Disallow: /combinazione/
Disallow: /seleziona-colore/
Disallow: /*colore/
Disallow: /*price/
Disallow: /*misura/
Disallow: /*marca/
Disallow: /*sort-by/
Disallow: /*combinazione/
Disallow: /*seleziona-colore/ UmuEX4z0 -
Canonical url for eshop, to do or not to do?
Hello everyone, I'm having a headache right now in designing a url base for a eshop. I would be thankful for any suggestions. Just to be clear about my problem, I'll be using this product example:
Search Behavior | | donatasBart
Product name: Product
Categories: Sale, Women
Brand: Superbrand Basicly every name of product, category or brand will resolve as url, if I enter url: /product it will open Product page /superbrand it will open brand page /superbrand/product will open product page from superbrand page /superbrand/sale/product would open product page selected from superbrand, then category sales /superbrand/woman/product would open product page selected from superbrand, then category woman So I will have many variations how to reach the product page, and of course i should set canonical url for it. I could set it just to /product , but then I don't know if I won't loose in terms of seo for other more detailed urls. Will google still index my other urls for product if i will state the canonical url in each product page? I know, maybe it would be the best solution to create a link like this: /superbrand_sale_product , which I can, but however it will be a duplicate content issue, since there will me many url pointing to the same product, and the only difference between them will be category, brand or whatever else... Thanks for all your ideas!0 -
Tracking forwarding / vanity URLS
Buongiorno from 18 Degrees sunny wetherby UK 🙂 Am i right in thinking you can track a forwarding urls via Google analytics. Put another way if a site points www.kissmymelons.co.uk to www.lickmybanana.co.uk can you measure how many referrals where generated from kissmymelons.co.uk I'm 69% sure the referral data will appear in the referral source section in Google analytics but I need to be 100% sure 😉 Grazie tanto,
Search Behavior | | Nightwing
David0 -
Would it be worthwhile to update URLs to include "for sale"
My team is new to the world of SEO. We were both in other departments when our last "expert" left and we became the entire department. Needless to say, we have many questions. I have a site for an existing customer - she wants to know if it is a good idea to change her URLs. She went to a webinar that suggested she include "for sale" in URL, since that is the site's primary purpose These are the current inventory URLs: http://www.dealerskinssite.com/used-inventory http://www.dealerskinssite.com/new-inventory The change would give something like: http://www.dealerskinssite.com/used-cars-for-sale
Search Behavior | | DealerskinsSearch
http://www.dealerskinssite.com/new-cars-for-sale I did some searching and could not find a solid recommendation one way or another. If it is true that people typically search for "used cars for sale" rather than a "used inventory", then there might be benefit to including that as part of the URL... but that is as far as my knowledge got me. Please let me know your thoughts Steve0 -
Trailing slash at end of URLs?
Hi, I'm just about to put up a new site and I need to decide between having no trailing slash at the end of the all the URLs, or putting one in there. I think Matt Cutts has a slight preference for them, as stated here: http://www.seomoz.org/blog/whiteboard-interview-googles-matt-cutts-on-redirects-trust-more "Matt says he would slightly advocate for using a trailing slash simply because it clearly indicates that a URL is a folder and not a document." However, I'm really wondering. I mean, if people link to my site, they'll tend not to insert a trailing slash, I'm thinking... Your thoughts would be welcome on this one! Cheers, Luke
Search Behavior | | McTaggart0 -
Where can I find research into consumer behavior in URL input?
I have a hypothesis that the % of users that type URLs is decreasing. Instead I think users will be arriving at sites by clicking on links in emails, search and social. Where can I find data / research to back me up?
Search Behavior | | TomCritchlow0