Dates in URL's
-
I have an issue of duplicate content errors and duplicate page titles which is penalising my site. This has arisen because a number of URLs are suffixed by date(s) and have been spidered . In principle I do not want any url with a suffixed date to be spidered.
Eg:-
www.carbisbayholidays.co.uk/carbis-bay/houses-in-carbis-bay/seaspray.htm/06_07_13/13_07_13
http://www.carbisbayholidays.co.uk/carbis-bay/houses-in-carbis-bay/seaspray.htm/20_07_13/27_07_13
Only this URL should be spidered:-
http://www.carbisbayholidays.co.uk/carbis-bay/houses-in-carbis-bay/seaspray.htm
I have over 10,000 of these duplicates and firstly wish to remove them on block from Google ( not one by one ) and secondly wish to amend my robots.txt file so the URL's are not spidered. I do not know the format for either.
Can anyone help please.
-
Thanks Kyle.
Particularly grateful for the Disallow format, they are the only URL's using an underscore so will work for me. WIll be checking why these are being created.
Do I need to remove them using the Removal Tool in Google, is there a format for doing this on block ?
Thanks again,
Alan
-
Hi Alan,
I would probably start by adding a disallow rule to robots.txt.
**Disallow: /*_** _may work and block all your dated URLs from being indexed but may also have adverse affects if you have any URLs containing underscores. To test whether this solution would work I would firstly implement a disallow directly on a chosen dated URL, _**Disallow: /20_07_13 **_for example, and then test whether Google has noindexed the page. GWT should tell you whether you have inadvertently blocked any other pages by doing so.
You should also be thinking about how these URLs are being created and taking actions to prevent it. Consider implementing canonical tags if you haven't already to clean up any potential duplication issues.
Cheers,
K
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Newbie SEO ?: Does my About page URL have to contain the word About?
New to WordPress and SEO. Built and launched my website last week. The URL was originally domain/about. However, I installed Yoast plugin and it told me "about" was a stop word. So, without too much thought (my first problem), I changed the url (before Google crawled me) to clearwingcommunications.com/storytelling. Since then, I've noticed that sites I know are optimized have their URL with the word "about." So, is this considered a bad practice? My site HAS been crawled at this point. If I change it back to About and do a 301 redirect, does that hurt reporting? Thanks for your help! Christy
On-Page Optimization | | christyr0 -
301 vs. keeping identical URL
Hey everybody! I have a question pertaining to our redesign. The situation is as follows: /drug-rehab/alcohol-withdrawal-los-angeles gets a decent amount of views on out website, and we would like it to be on our redesigned site. I was curious what impact, if any, I would see given the two scenarios below. 301 to /alcohol-withdrawal make the new page /drug-rehab/alcohol-withdrawal-los-angeles as well The second situation is that there are a serious of other pages which don't seem to be of drastic benefit, which I don't feel NEED to be on the website. For example: /post-acute-alcohol-withdrawal-treatment/drug-los-angeles /rehabs-resources/drug-abuse/sub-acute-alcohol-withdrawal etc It appears to me that the content on these pages is rather similar, and I feel like they don't really say anything special. Can I 301 them to the new page? Should I let them die in the black hat inferno they were made in? Any thoughts are greatly appreciated! Thanks guys!
On-Page Optimization | | HashtagHustler0 -
Wordpress category url problem.
I have set up wordpress categories but the permalinks are showing as www.mydomain.com/?cat=12 as opposed to the category name. The child categories though work fine and show as www.mydomain.com/category/chidcatgegory I've obviously got my permalink settings wrong somewhere. How do I fix this?
On-Page Optimization | | SamCUK0 -
Replacing website, same URL, lose ranking
We had to do a rush job for a client and get a website up very quickly. We had to just replace the existing files on the server, but URL stayed the same. Now "teeth whitening glasgow" which they were number #1 for isn't coming up. Know there is still some on page work to be done - but the link profile should still remain the same? Just looking for some advice to get back to where they were - should we do a website change of address or just verify the site in Webmaster and get all the on page done? Thanks, Laura
On-Page Optimization | | lauratagdigital0 -
URL Structure for Home Decorating site
Hi, I am about to launch a home decorating ideas website. I am stuck to determining the ideal URL structure, and the options are: Category: myhomedecoratingsite.com/room/master-suite/
On-Page Optimization | | adepalma
myhomedecoratingsite.com/kitchen/
myhomedecoratingsite.com/room/living
myhomedecoratingsite.com/master-suite/
myhomedecoratingsite.com/living-room/ Post: myhomedecoratingsite.com/interior/beautiful-master-suite/
myhomedecoratingsite.com/interior/cozy-living-room/
myhomedecoratingsite.com/room/master-suite/beautiful-modern-suite/
myhomedecoratingsite.com/room/living/cozy-contemporary-living Are these any good? What your best suggestion? Thanks in advance for your inputs. Al0 -
My Images Aren't Indexed By Goolge
My site is indexed by google, and there hasn't really been much of a problem with my content, but now I am noticing that none of our images are coming up in the images searches of google. I mean none. I have even typed in the alt text of the image verbatim and nothing shows up. I use wordpress if this helps anyone. Any advice would be awesome, Thanks a lot.
On-Page Optimization | | Caseman0 -
Creating Authority and choosing URL's
Creating Domain Authority and choosing URL's: A: What is better if you want to get higher Domain Authority? Choose keyword.domain.com or www.domain.com/keyword when other sites link to it? B: And for Page Authority? Choose keyword.domain.com or www.domain.com/keyword? Thanks!
On-Page Optimization | | HMK-NL0 -
Has anyone had experience with the Wix platform and it's SEO qualities?
Wix offers an inexpensive, user friendly platform for building websites. Most of the site is flash, but Wix claims to be SEO friendly. I'm all ears for your feedback and experience with Wix.
On-Page Optimization | | ksracer0