Welcome to the Q&A Forum

Hi Deb,

Thank you for your reply.

I have never thought, that Google would crawl the robots.txt this rarely. I actually read it somewhere, which makes complete sense, that before they start crawling, they validate the process against robots.txt. This is one page only, but basically one of the most important ones.

This is now a shocking experience for me, thank you for drawing my attention to it. Anyway, I have submitted the page through 'Fetch as Google' now.

Regarding your url suggestion, I do not want them to be 404-d, at least not all of them, as for examply the login pages I still want to use, and why we have individual urls, is that because we would like our visitors to return back the page they left, before we asked them to log in. So status 200 is ok, because these pages we have for customers, but the very same pages are totally useless for Google to crawl or to index.

I hope this clarifies.

Really good article, indeed!

I have been thinking about the whole concept during the weekend, and now I have a further concept, definetely worth considering.

Thank you again, Ryan.

Thank you for the further info, Ryan.

Although I see your point and can accept lots of truth in it, checking all the competitors and even the largest sites all around the web, they still keep using robots.txt (even Google does so).

I however accept noindex to be a superior solution to robots.txt and will use it for all the contents I do not want to be indexed.

I will then see, if I need and how I might need to use robots.txt. I hope, it does not hurt having a noindexed page included in robots.txt (at a later time, when it is already out of the index).

Hi Ryan,

Thank you for your reply.

The only worry I have regarding the crawl budget, that I currently have three times more indexed pages than useful pages, due to the issues I have mentioned earlier.

It is true, that I do not have daily content updates on all of my useful pages, however I have thought that Google allocates individual crawling budget to all sites, based on the value he assigns to them.

I just want this budget to be spent wisely, and not causing my useful pages to be crawled less frequently, due to crawling no-value (but noindexed) content instead.

Hi,

I have been checking every possible resources for content removal, but I am still unsure on how to remove already indexed contents.

When I use robots.txt alone, the urls will remain in the index, however no crawling budget is wasted on them, But still, e.g having 100,000+ completely identical login pages within the omitted results, might not mean anything good.

When I use meta noindex alone, I keep my index clean, but also keep Googlebot busy with indexing these no-value pages.

When I use robots.txt and meta noindex together for existing content, then I suggest Google, that please ignore my content, but at the same time, I restrict him from crawling the noindex tag.

Robots.txt and url removal together still not a good solution, as I have failed to remove directories this way. It seems, that only exact urls could be removed like this.

I need a clear solution, which solves both issues (index and crawling).

What I try to do now, is the following:

I remove these directories (one at a time to test the theory) from the robots.txt file, and at the same time, I add the meta noindex tag to all these pages within the directory. The indexed pages should start decreasing (while useless page crawling increasing), and once the number of these indexed pages are low or none, then I would put the directory back to robots.txt and keep the noindex on all of the pages within this directory.

Can this work the way I imagine, or do you have a better way of doing so?

Thank you in advance for all your help.

Thank you again, John. I will fix this, based on our discussion.

Thank you for your reply, Josh.

I will then use the 301, but should I also use the noindex tag for these pages to be removed from the index?

Does it make an emphasis on my intention, or it adds no extra to the process? Perhaps, they should not be used together at all, as basically they are meant for different tasks.

(Unfortunatyly, robots.txt is not really a solution, as we have the following url structure:

www.example.com/city/tag

Since all the cities have at least a couple of valid tags, I can't specify the path to be excluded from indexing. I would also try not to add 2,000+ cities individually.

As for GWT, url removal for this number of pages might also not be an option, as I have minimum 100,000+ no-value pages to be removed (the limit is 500 per month).)

Hi,

We have accidentally had Google indexed lots os our pages with no useful content at all on them.

The site in question is a directory site, where we have tags and we have cities. Some cities have suppliers for almost all the tags, but there are lots of cities, where we have suppliers for only a handful of tags.

The problem occured, when we created a page for each cities, where we list the tags as links.

Unfortunately, our programmer listed all the tags, so not only the ones, where we have businesses, offering their services, but all of them!

We have 3,142 cities and 542 tags. I guess, that you can imagine the problem this caused!

Now I know, that Google might simply ignore these empty pages and not crawl them again, but when I check a city (city site:domain) with only 40 providers, I still have 1,050 pages indexed. (Yes, we have some issues between the 550 and the 1050 as well, but first things first:))

These pages might not be crawled again, but will be clicked, and bounces and the whole user experience in itself will be terrible.

My idea is, that I might use meta noindex for all of these empty pages and perhaps also have a 301 redirect from all the empty category pages, directly to the main page of the given city.

Can this work the way I imagine? Any better solution to cut this really bad nightmare short?

Thank you in advance.

Andras

Hi Mat,

You have provided some great clarification, thank you.

Now, I only have one thing to be sorted out:

Should I add the DMCA protection badges to all pages with unique content, as soon as they are created?

My dilemma is, that let's say I find my content somewhere else. I will submit a take down request through DMCA. How it is going to be proven, whose site owns the original content?

I am definetely not an expert in this, but it can easily happen, that my page is not the older one, and the page where my content is placed to, simply just changed its previous content, but the page itself is older than my relevant page.

Thank you

Hello,

Can anyone tell, if it is possible to protect text –type content without the rel=author tag?

We host a business listing site, where, apart from the general contact information, we have also started to write original 800+ character-long unique and original contents for the suppliers, where we expect visits, so rankings should be increased.

My issue is that this is a very competitive business, and content crawling is really an everyday practice. Of course, I would like to keep my original content or at least mark it as mine for Google.

The easiest way would be the author tag, but the problem is, that I do not want our names and our photos to be assigned to these contents, because from one hand, we are not acknowledged content providers on our own (no bio and whatsoever), and on the other hand, we provide contents for every sort of businesses, so just having additional links to our other contents, might not help readers to get what they want. I also really do not think that a photo of me could help increase the CTR from the SERP:)

What we currently do, is that we submit every major fresh content through url submission in WMT, hoping that first indexing might help. We have only a handful of them within a day, so not more than 10.

Yes, I could perhaps use absolute links, but this one is not a feasible scenario in all cases, and about DMCA, as our programmer says, what you can see on the internet, that you can basically own.

So finally, I do not mind our contents being stolen, as I can’t possibly prevent this. I want however our original content to be recognized as ours by Google, even after the stealing is done.

(Best would be an ’author tag for business’, so connected to our business Google+ page, but I am not aware, this function can be used this way.)

Thank you in advance for all of you, sharing your thoughts with me on the topic.

Ok, this is something I can take as homework.

Thank you for having checked my issue in details.

Your assumption is correct about the snippets.

We either use part of the business description or some of the actual review wording on the category pages.

While I understand the importance of your suggestion, that is all I have about these businesses, and I add these snippets to the lists, so that I could increase CTR through previews.

Yes, there is no original content, basically just 'copied contents' from several pages, so the category page is total duplicate to nothing, but not genuine either.

Thank you again for your help.

Thank you for your reply.

Your suggestion matches the actual way of how we handle the issue right now.

I have thought, that we might be able to improve the process a bit, but I am happy, that there is no need to change the way of how it works now.

Hello,

We run a site, which lists local businesses and tag them by their nature of business (similar to Yelp).

Our problem is, that our category and sub-category(i.e.: www.example.com/budapest/restaurant or www.example.com/budapest/cars/spare-parts) pages are extremely weak, and get almost no traffic, but most of the traffic (95+ percent) goes for the actual business pages.

While this might be a completely normal thing, I still would like to strengthen our category (listing) pages as well, as these should be the ones targeted by some of general keywords, like ‘restaurant’ or ‘restaurant+budapest’.

One of the issues I have identified as a possible problem, that we do not have a clear hierarchy within the site, so while the main category pages are linked from the homepage (and the sub-categories from here), there is no bottom-up linking from the business pages back to the category pages, as the business page URLs look like this: www.example.com/business/onyx-restaurant-budapest.

I think, that the good site- and url structure for the above would be like this: www.example.com/budapest/restaurant/hungarian/onyx-restaurant.

My only issue is, perhaps not with the restaurants but with others, that some of the businesses have multiple tags, so they can be tagged i.e. as car saloon, auto repair and spare parts at the same time. Sometimes, they even have 5+ tags on them.

My idea is, that I will try to identify a primary tag for all the businesses (we maintain 99 percent of them right now), and the rest of their tags would be secondary ones. I would then use canonicalization and mark the page with the primary tag in the url as the preferred one for that specific content.

With this scenario, I might have several URLs with the same content (complete duplicates), but they would point to one page only as the preferred one, while our visitors could still reach the businesses in any preferred ways, so either by looking for car saloons, auto-repair or spare parts.

This way, we could also have breadcrumbs on all the pages, which now we miss completely.

Can this be a feasible scenario? Might it have a side-effect? Any hints on how to do it a better way?

Many thanks,

Andras

Thank you again for your comments, Istvan.

Although, I can't take everything you suggested, I understand your point, and will initiate some changes based on that.

András

We run a site, which is a directory type one, where visitors can look for local businesses per city as well (at some point similar to the 'Yelp concept').

Now, we use www.example.com as the homepage, and the www.example.com/city1, where city1 is the capital of our country, is 301 redirected to the homepage, so this is your default setting.

When you choose e.g city2, your url changes to www.example.com/city2, and the city value is stored in a cookie. Then, when you leave the session, and return to the site later on, you will see the homepage url, but with your previous choice of city (in case you let cookies be stored).

My concern is, that the meta title always contains the chosen city name, so when you return to the website, and you previously used city2, you will now see the homepage url (which normally belongs to city1), but with the meta title of city2 or with any other previously chosen city.

Does this mean a problem, and should I always use the correct url, which would be www.example.com/cityX, or this could not cause a problem for me? If it does, would you mind sharing me the exact problems as well?

Thanks,