Welcome to the Q&A Forum

Just to follow up - I have now actually 410'd the pages and the 410's are still being re-indexed.

I'll check this one out as well, thanks! I used a header response extension which reveals the presence of x-botots headers called web developer.

Thank you for the quick response,

The pages are truly removed, however, because there were so many of these types of pages that leaked into the index, I added a redirect to keep users on our site - no intentions of being "shady", I just didn't want hundreds of 404's getting clicked and causing a very high bounce rate.

For the x-robots header, could you offer some insight into why my directive isn't working? I believe it's a regex issue on the wp-content. I have tried to troubleshoot to no avail.

<filesmatch <strong="">"(wp-content)">
Header set X-Robots-Tag: "noindex, nofollow"</filesmatch>

I appreciate the help!

Thank you! I am in the process of doing so, however with a 410 I can not leave my JS redirect after the page loads, this creates some UX issues. Do you have any suggestions to remedy this?

Additionally, after the 410 the non x-robots noindex is now being stripped so it only resolves to a 410 with no noindex or redirect. I am still working on a noindex header, as the 410 is server-side, I assume this would be the only way, correct?

Yes, all pages have a noindex. I have also tried to noindex them using htaccess, to add an extra layer of security, but it seems to be incorrect. I believe it is an issue with the regex. Attempting to match anything with wp-content.

<filesmatch "(wp-content)"="">Header set X-Robots-Tag: "noindex, nofollow"</filesmatch>

Hi All,

We are experiencing issues with pages that have been 404'd being indexed. Originally, these were /wp-content/ index pages, that were included in Google's index. Once I realized this, I added in a directive into our htaccess to 404 all of these pages - as there were hundreds. I tried to let Google crawl and remove these pages naturally but after a few months I used the URL removal tool to remove them manually.

However, Google seems to be continually re/indexing these pages, even after they have been manually requested for removal in search console. Do you have suggestions? They all respond to 404's.

Thanks

Hello,

Currently, we have a .com English website serving an international clientele. As is the case we do not currently target any countries in Google Search Console. However, the UK is an important market for us and we are seeing very low traffic (almost entirely US). We would like to increase visibility in the UK, but currently for English speakers only. My question is this - would geo-targeting a subfolder have a positive impact on visibility/rankings or would it create a duplicate content issue if both pieces of content are in English? My plan was:

1. Create a geo-targeted subfolder (website.com/uk/) that copies our website (we currently cannot create new unique content)

2. Go into GSC and geo-target the folder to the UK

3. Add the following  to the /uk/ page to try to negate duplicate issues. Additionally, I can add a rel=canonical tag if suggested, I just worry as an already international site this will create competition between pages

However, as we are currently only targeting a location and not the language at this very specific point, would adding a ccTLD be advised instead? The threat of duplicate content worries me less here as this is a topic Matt Cutts has addressed and said is not an issue.

I prefer the subfolder method as to ccTLD's, because it allows for more scalability, as in the future I would like to target other countries and languages.

Ultimately right now, the goal is to increase UK traffic. Outside of UK backlinks, would any of the above URL geo-targeting help drive traffic?

Thanks

Hello,

Our company is international and we are looking to gain more traffic specifically from Europe. While I am aware that translating content into local languages, targeting local keywords, and gaining more European links will improve rankings, I am curious if it is worthwhile to have a company.eu domain in addition to our company.com domain.

Assuming the website's content and domain will be exactly the same, with the TLD (.eu vs .com) being the only change - will this add us benefit or will it hurt us by creating duplicate content - even if we create a separate GSC property for it with localized targeting and hreflang tags? Also - if we have multiple languages on our .eu website, can different paths have differing hreflangs?

IE: company.eu/blog/german-content German hreflang and company.eu/blog/Italian-content Italian hreflang.

I should note - we do not currently have an hreflang attribute set on our website as content has always been correctly served to US-based English speaking users - we do have the United States targeted in Google Search Console though.

It would be ideal to target countries by subfolder rather if it is just as useful. Otherwise, we would essentially be maintaining two sites.

Thanks!

It seemed to work. Hopefully the noindex is respected, thank you!

It looks like it is active. Thanks, John! Can you no-index an entire directory in GSC? I thought it was only per URL.

Hello,

We are ranking for an acronym, which I understand can lead to fickle rankings. However, we have two pages ranking page one - two for the same keyword, but they do so in spite of each other.

By this I mean, one page will rank, while the other is nowhere to be found. It seems that the one page (a blog post) is more likely to rank on the weekends while the product page is more likely to rank on the weekdays.

I would like the product page to rank all the time, and to target another keyword with the blog post. Would removing the keyword from the blog post allow the product page to rank all the time - or would it lead to no pages ranking during times when the blog post would otherwise be ranking?

I should note the blog post has more external links and is not exactly optimized for the keyword, while the product page has more internal links and is optimized for the keyword.

Awesome, thank you for answering everything!

Thanks for the response, I do have the keyword as the anchor text linking from my blog post to my product page. I don't know why it is when one ranks, the other does not, rather than alongside each other.

Would de-optimizing my blog post allow for my product page to rank all the time - or will it cause a lack of coverage when the blog post would otherwise rank?

Thank you for your response! I will certainly use the regex in my robots.txt and try to change my Htaccess directive to 410 the pages.

However, the issue is that a defunct plugin is randomly creating hundreds of these URL's without my knowledge, which I can not seem to access. As this is the case, I can't add a no-index tag to them.

This is why I manually de-indexed each page using the GSC removal tool and then blocked them in my robots.txt. My hope was that after doing so, Google would no longer be able to find the bad URL's.

Despite this, Google is still actively crawling & indexing new URL's following this path, even though they are blocked by my robots.txt (validated). I am unsure how these URL's even continue to be created as I deleted the plugin.

I had the idea to try to write a program with javascript that would take the status code and insert a no-index tag if the header returned a 404, but I don't believe this would even be recognized by Google, as it would be inserted dynamically. Ultimately, I would like to find a way to get the plugin to stop creating these URL's, this way I can simply manually de-index them again.

Thanks,

Thanks for the input, Nicholas. This is what I was thinking, however, it seems that the blog post is now ranking for the last four days, and my solutions page isn't ranking at all for the keyword. Usually, the blog post would rank 1-2 days a week while the product page would rank the rest.

Would you still suggest de-optimizing the blog? Ranking for the keyword has been a months-long initiative, and I don't want to ruin my efforts.

Or should I wait and see if the product page begins ranking instead of the post again before de-optimizing the post?

I do agree, I may have to pass this off to someone with more backend experience than myself. In terms of plugins, are you aware of any that will allow you to add noindex tags to an entire folder?

Thanks!

Hi All,

We are experiencing issues with pages that have been 404'd being indexed. Originally, these were /wp-content/ index pages, that were included in Google's index. Once I realized this, I added in a directive into our htaccess to 404 all of these pages - as there were hundreds. I tried to let Google crawl and remove these pages naturally but after a few months I used the URL removal tool to remove them manually.

However, Google seems to be continually re/indexing these pages, even after they have been manually requested for removal in search console. Do you have suggestions? They all respond to 404's.

Thanks

Thank you! I am in the process of doing so, however with a 410 I can not leave my JS redirect after the page loads, this creates some UX issues. Do you have any suggestions to remedy this?

Additionally, after the 410 the non x-robots noindex is now being stripped so it only resolves to a 410 with no noindex or redirect. I am still working on a noindex header, as the 410 is server-side, I assume this would be the only way, correct?