Welcome to the Q&A Forum

Hi! - I wrote this guide a few years ago on penalty recovery which may help you as it contains a lot of methods around auditing the links - https://moz.com/blog/ultimate-guide-to-google-penalty-removal

If we were to approach a product with 70k URLs. We'd do the following steps:

1. Pull all the URLs into a Spreadsheet
2. Split the URLs into domains
3. Filter the URLs are search for common spammy words. e.g 'Link', 'Best', 'Free', 'Cheap', 'Dir', 'SEO' etc (mark as spam accordingly)
4. Run contact finding across all URLs using a tool such as URL Profiler with Whois Lookups
5. Filter by contact name and find duplicates (mark as spam accordingly)
6. Filter by website type and mark as spam accordingly
7. Manually check remaining links

By working through by domain, you'll rule out thousands of spammy links very quickly. Though 70k will ultimately take a few solid days of work.

Hope this helps,

Lewis

As Logan pointed out, the most likely cause of this is because of the canonical tags pointing to the HTTP variation. I'd probably suggest carrying out the following:

Download a tool like Screaming Frog SEO Spider and crawl your full website - https://www.screamingfrog.co.uk/seo-spider - Once the crawl has run, export the data and check both the URLs and the Canonicals across all pages to ensure only HTTPS variants appear.

If you find http variants, then it's likely that the redirect rules you have in place aren't working. This is a global rewrite rule that can be applied to force the https:// version to show at all times, so you shouldn't need to make the changes once by one. I'd also make sure that if you're linking to any pages internally, that you're linking to them using a relative path, or straight to the https:// version. If your canonicals are causing issues, just ensure they're always choosing the HTTPS version and you shouldn't have any issues!

Finally, in GSC, just make sure you've got the HTTPS version of the site setup and that your sitemap includes only HTTPS URLs.

Hope this helps!

I'm not sure the answers previously presented are related to the issues you're having. Having worked with Magento for a long time, this can be an issue that occurs over and over again.

To answer your initial question, truncating your core_url_rewrite table will remove all of these URLs, but it'll only delay the problem until it reoccurs again in the future (unless you've had a problem in the past which has been rectified). You're also correct in that any rewrites in the system previously there will disappear, so you'll probably end up with a lot of crawl issues appearing in Search Console.

Your best move would be to find out why you have so many URLs in there in the first place. Do you have a huge product catalog with multiple stores? Or is this something to do with an issue in your Magento version or some setup issues. The most common time this usually occurs is if two products get added to your site with the same URL Key. Every time the reindex process runs, your core_url_rewrite table will grow. You could check this by looking at the number of rows in the table, reindexing the site and if it grows further, then it's likely to be the problem. The quickest way to fix this is to ensure all URL key are unique.

There's also an article here about duplicate keys - https://firebearstudio.com/blog/magento-url-reindex-core_url_rewrite-duplicates-patch.html - this should hopefully clear the issue.

I hope this helps! If it doesn't solve the problem, then sending over a little more information around the number of stores, catalog site and the split between system generated URL rewrites and custom URL rewrites would be great so we can try to help further!

Thanks,
Lewis

Hi!

We do a lot of consultancy for Magento projects and this is a question that comes up quite regularly as it can't really be handled perfectly straight out of the box with Magento.

Every implementation is a little bit different, but I'll put together some recommendations below based on the information available at the moment.

For your faceted navigation, you ideally don't want to index any of these pages, unless you believe that you'll rank in your own right for specific filters (e.g. Colour, like you pointed out in your last message).

That then comes with some additional complications. In Magento, if you have 3 colours available in the faceted nav, you'll have all the different variations indexed in each combination.

For example:

Blue
Black
Red

Blue + Black
Blue + Red
Black + Red
Black + Blue
Red + Blue
Red + Black

Magento as standard doesn't always keep the filters in the same order, so you can end up with literally thousands of pages ending up in the index for a relatively small number of attributes being shown on your pages.

There are a few recommendations here:

1. Go and look at the MageWorx Ultimate SEO Suite Plugin - http://www.mageworx.com/seo-suite-ultimate-magento-extension.html - For $249, it solves a lot of issues Magneto has straight out of the box and gives you ultimate control over your meta titles.

What you want to do is set all of your facets to 'NOINDEX,FOLLOW' where possible. This will reduce the number of URLs in the index gradually. An example of this would be adding ?min=* and mode=* etc (grid/list variants).

2. For your canonicals, you're probably best setting the canonical to the current filtered page (for example, if you're on a category page with colour = blue selected in your faceted nav, you'd have this URL as your canonical). Some sites we work on have it setup so the canonical points to the category URL (like you currently have).

Finally, you probably want to build an extension to allow you to inject content into the filtered content pages. If you're using an extension like ManaDev for your facet navigation, this can be achieved fairly easily and allows you to add a block of text to each filter applied on a page.

You should also look to request each of the incorrectly indexed URLs is removed from the index (although this does take a long time if you have a lot!).

We wrote a really long guide around launching a Magento website last month which may be of interest - https://www.pinpointdesigns.co.uk/the-definitive-guide-to-launching-a-magento-website/. We've also done a guide on Common Magento SEO Issues here - https://www.pinpointdesigns.co.uk/common-magento-seo-issues/ and I previously wrote a guide on setting Magento up for Search Engines on Moz - https://moz.com/ugc/setting-up-magento-for-the-search-engines (Although this is likely to be a little outdated now)

I hope this helps!
Lewis

Hi Luke,

What data are you trying to get from Google Webmaster Tools?

They have an export for features like backlinks that you can put into something like Google Docs, but it depends on what data you're wanting to collect. I would guess that depending on your requirements, someone will have built something to pull the information out into an external system.

They've also got an API available which would allow you to develop your own application and pull out a selection of data fairly easily - https://developers.google.com/webmaster-tools/

Many thanks,
Lewis

You're right in with the **Disallow: /fish **in the robots file blocking all those initial links, but if you wanted to block everything inside the /en/ folder, you would need to do disallow: /en/fish

You could use a wildcard in the robots.txt file to do something along the lines of Disallow: /BTS-*

This _'should' _work, but it's always worth checking using a tool to make sure it's all implemented correctly. Distilled did a post a while back about a JS tool which allows you to test if robots.txt files work correctly which can be found here - http://www.distilled.net/blog/seo/js-bookmarklet-for-checking-if-a-page-is-blocked-by-robots-txt/

In addition to this, you could also use the 'blocked URLs' tool in GWT to see if the pages are successfully blocked once you've implemented the code.

Hope this helps!

Hi all,

I've just tried adding a new campaign to our Moz analytics account but it's saying that it already exists. I actually removed all of the campaigns we had in Moz a while ago, so it doesn't already exist.

Any one else had the same issues and if so, how did you fix it?

Thanks,
Lewis

Hi Manan,

If you go to http://pro.moz.com/campaigns - then go to Global Settings, you can upload your logo for the branded reports.

Someone posted a similar message last week and I think they were referring to Moz Analytics. I can't find the Q&A post now but I'm sure a member of staff responded saying that Branded reports weren't available as of yet on Moz analytics so that might be where your issue is.

Hope this helps,
Lewis

Hi Zach,

Yes, you should be indexing these pages and ensuring you add them to your sitemap. On the basis the content is unique and provides a benefit to the people reading your blog, then there would be no reason to not index the pages.

Hope this helps.
Lewis

Have you received any notifications in your Google Webmaster Tools account with regards to a penalty? You can usually find this by going to https://www.google.com/webmasters/tools/ and clicking on 'Search Traffic > Manual Actions'. Alternatively, you might have a message in there.

This would only show for a manual penalty, so if you've received a penalty due to an algorithm, this is a bit harder to spot. The easiest way to do this is look at the date your traffic dropped and comparing this with the updates that Google have released - http://moz.com/google-algorithm-change - The moz link there has a list of all known updates.

Alternatively, you could look at the following plugin which will overlay over a chart over the top of your analytics results and give you an indication of when penalties have come in along with your traffic drops.
https://chrome.google.com/webstore/detail/chartelligence/njhdcfdiifemfnfddhfjmfbkajajceag?hl=en

I wouldn't have thought that moving hosting provide / IP would cause you any huge drops in the search engines. Maybe a little bit of fluctuation, but nothing significant. Where is your new server located in relation to your store location?

EDIT: I started writing my response before any other comments were posted, so looks like Peter beat me to it on most of the points

No problem at all, glad I could help!

Good luck with the site!

Hi Iris,

Yes, you want the URLs to be included in the sitemap too as this is what the search engines will use to build a map of your website and list them in their search engines.

With regards to your duplicate page issue, this is to do with your pagination of the pages. There's a guide here which tells you the SEO best practices when it comes to handling pagination, so hopefully this will help - http://searchengineland.com/the-latest-greatest-on-seo-pagination-114284

Really, you don't want those additional pages to be picked up in the search engines, so a fix will need to be sorted for that. You'll probably need to find a developer / a plugin to allow you to not index the /2 /3 pages etc.

I don't do much work with Wordpress, so maybe someone else could advise you with regards to the archive section. If your categories and posts are indexed, then I would have thought that you've done the correct thing with the archives and set them to noindex. That being said, I have limited knowledge of carrying out SEO for wordpress so I would need to take a closer look into that to give you a better response..

Hope this helps at least a little!

On a separate note, you could try accessing the Yoast website through a proxy such as http://hidemyass.com/ - Might be worth a go?

Looking at your domain (I've only checked out the homepage at the moment), the Title tag is implemented correctly in the code. That being said, your title tag is very long:

Gary has pointed out some reasons why Google might be ignoring your title tag. I would suspect that because you've exceeded the recommended length (generally 65 characters but up to 69 characters), they may be trying their own titles to see if it works better.

Looking at the search term you've sent over, the title for this page is as follows:

Again, this is too long for Google to show in the search results, so they're probably testing their own version instead of truncating it with ... at the end to see if they can improve your CTR.

[identifying information removed by staff at request of author]

I'm not sure if I fully understand your question, but I'll give it my best shot to answer it.

If I was in your position, I would set the Yoast plugin to index both the categories and the posts, but make sure that the posts only show a short summary / the first 'x' number of words than the post. The reason behind doing this is similar to how an ecommerce store works.

On eCommerce stores, you have your categories which are used as funnels to find inner product pages. Categories are generally set to be indexed as they combine lots of relevant data for a user which helps create a better user experience. In your case, your category may be 'Villas' and inside that, you may list 20 individual Villas. If you didn't have your category pages indexed, you'd be relying on the search engine to filter a user down to either your homepage or one specific villa which you hope is right for the users choice (which most likely will result in a higher bounce rate).

Again, I'm not sure if I've understood exactly what you're looking for, but that's what I would recommend if you're asking which sections of your site should be indexed. Hope this helps!

No problem - Glad I was able to help.

Best of luck in the future!
Lewis

Sorry, I copied the wrong file path. I meant this one:

public_html/villasdiani.com

Glad it worked!

Yeah, you just need to make the same change to the following index.php file and that will solve that problem too:

public_html/villasdiani.com/wp-content/themes/decorum/index.php

That should get you all up and running.

It should just be the one inside the standard public_html folder as this is what is referenced in the PHP error (/home2/damian/public_html/index.php on line 17).

Hope this helps! Let us know how you get on.

Hi Iris,

It's hard to debug the problem without being able to see the issue, but it sounds like it's an incorrect file path. Open up your index.php file and find the following:

require('./wp-blog-header.php'); change it to this:

require('wp-blog-header.php');

Hopefully that will help fix your issue!

Lewis

Like Alejandro has explained, the header.phpxxx isn't actually being used. .phpxxx is not a valid php extension, so they've added the xxx in so it isn't used in the wordpress theme.

Looking at the code in header.php, your title tag is here:

<title><?php if( is_home() || is_front_page() ) bloginfo('name'); else wp_title(); ?></title>

This basically means:

If you're on the homepage or on the front page of the website, show the blog name (which is pulled from your wordpress settings). If not, show the wordpress title (this will be pulled from the name of the page which the user is on).

If you wanted to make it so the order of the title page is different, for example, if you wanted your title tag to show 'Page Title Here | Site Name', you could go into your wordpress settings and should be able to change the order in which this is done... Alternatively, download a plugin such as Yoast's SEO plugin and you'll be able to customise everything there!

I think that sounds absolutely fine on the basis the backlinks pointing to the URLs are ok and like Gary said, check you haven't received any penalties on the URLs. If you've done any blackhat link building on the URLs and you're planning on 301 redirecting them, you're potentially opening yourself up for trouble.

That being said, why do you want to 301 redirect them? Do the sites have regular traffic coming to them? Or have you done some particularly good form of link building on them?

If there's no real value in redirecting them, you may as well just leave them and cut your loses early...

Personally, I would agree with you an opt for the following option:
clone.com/Apartments/New-York?price=30&size=100

I don't think it matters whether that section of the URL is readable to everyone. I would actually say that anyone who has a technical background would find the URL above easier to change than the other one, as having /'s in the URL almost symbolised different directories rather than a parameter (that's how I would generally interpret it anyway).

I think in the grand scheme of things, It's going to make little different as you don't want the additional sections to actually be indexed in the search engines. Like Gary correctly pointed out, you can setup 'URL Parameters' in GWT and I think that's your best option. There's more information about that here - http://googlewebmastercentral.blogspot.co.uk/2011/07/improved-handling-of-urls-with.html

You could also use robots.txt to block the parameters in the URL but this depends on whether the search engine crawling your website chooses to use it.

Hope this helps!
Lewis

Hi Daniel,

Yeah - I was kind of expecting that kind of a response.

It's pretty frustrating really, more because I know I haven't tried to bribe anyone. That being said, if they won't shift, then I'll have to move on. Like you say, I just need to avoid letting it get under my skin.

Plenty of other good backlinks out there! It'd just be nice to think I could clear the air with the DMOZ crew!

Many thanks,
Lewis

Hi everyone,

Before I start, I want to point out that I realise DMOZ is not everything. If I can't get a link, then I'll accept it, but I'd like to try damn hard to get a link in there if possible as I honestly believe we are a reputable company that deserves to be in there with other companies in our region.

So, lets go back to the start:

A few years ago, when I was young and foolish, I posted a request on rentacoder (now vworker) asking for someone to submit us to DMOZ for $xxx. If I remember correctly, we accepted someone for the job and they submitted our site. Since then, we've never been accepted.

I tried submitting the site a couple of years ago and realised it still hadn't been submitted around a year ago. I thought this was a bit strange, so I did a bit of digging. I ended up talking to someone who was a DMOZ editor. They looked at my listing on DMOZ and basically said that I'd be rejected for 'bribing an editor' - something that I still maintain I didn't do.

Never at any point have I said to someone that I would pay them to guarantee us a listing. At the time, I thought there was a 'certain way' to write a submission, but now that I've grown up a bit, I've realised it's fairly easy to get a listing if you write a well written description and are honest - I've done this for many clients sites which are now accepted.

Unfortunately, this does mean that It looks like I won't be able to get our agency on DMOZ, but I just wanted to give one last push and see if anyone on SEOMoz knows of any one I could speak to in order to lift this ban?

I realise now that it was probably a stupid thing to do, and in hindsight, I should have just waited like everyone else. I hold my hands up to that and I've worked over the last few years to grow our agency into something special for our clients.

As I say, I realise dmoz isn't everything, but it is a directory that still holds value and I would like to be linked off there if possible.

Any tips on how I may be able to beg for my ban to be lifted would be much appreciated.

Kind regards,

Lewis

PS: I applied to become an editor at one point, this was when I realised something wasn't right - http://www.resource-zone.com/forum/index.php?showtopic=52658

Hi Daniel,

Thanks for the reply.

I've contacted LinguaLinx and received a response to say that they'll be in touch as they're on holiday at the moment. We'll see what the price comes back with. I've also posted out a job offer for people who can speak multiple languages to do content writing. I'll have to weigh up which seems more cost effective but Ideally I want native speakers to work with.

If anyone else has any suggestions on companies who can help me achieve this, that would be great!

Thanks,

Lewis

Thank you for the recommendation guys.

SEO5 - I was considering putting this on oDesk, but I didn't want to run the risk of people taking on the job and then just putting it into Google Translate and sending it back. I'll need someone to proof-read the work for me and make sure it's all ok, and need to ensure this really is top quality content.

Daniel - I'll get in touch with LinguaLinx shortly and see if it's something they can help me out with. I've been in touch with another couple of companies already and had some quotes through, but I would prefer a recommendation as I can then have someone vouch to say they're doing good work and it's not just scraped off the internet and then translated using an auto translator.

Anymore suggestions anyone has would be great!

Many thanks,
Lewis.

Hi All,

I'm currently doing a lot of work for a UK client who has multiple sites outside the UK (all part of the same business). We're currently discussing the option of us handling all of his SEO for his German, French, Spanish and Italian sites too, but we only have access to one person in the office who can speak French and Spanish.

They're currently booked up on other jobs that we can't really move them off, so I'm looking for options of outsourcing some of the content writing.

My question is, does anyone know of any high quality content writing services that have writers available to write for the countries languages above?

We're going to focus initially on their on-site strategy and building up their high quality content. At the moment, they don't have much relevant content on their website, so we're going to initially look at this. Moving forward, we'll be looking at their off-site strategy and trying to find areas to submit high quality articles, look at guest blogging and PR opportunities.

Any tips anyone has on this side (in terms of outsourcing to native speakers) would be quite useful too!

Many thanks,
Lewis