Welcome to the Q&A Forum
Browse the forum for helpful insights and fresh discussions about all things SEO.
After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.
Are those main category pages (like /collections/living-room-furniture) or are they different?
Hi Dana,
Expires headers and other caching headers can help improve site performance (as you said), and that will be a good thing for SEO. There is no reason to be concerned - they are common headers and there isn't much they could do to have any negative impact on SEO.
Good luck!
Tom
Hi there!
Ok, it is difficult to know all the ins and outs without looking at the site, but the immediate issue is that your robots.txt setup is incorrect. robots.txt files should be one per subdomain, and cannot exist inside sub-folders:
A **robots.txt **file is a file at the root of your site that indicates those parts of your site you don’t want accessed by search engine crawlers
From Google's page here: https://support.google.com/webmasters/answer/6062608?hl=en
You shouldn't be blocking Google from either site, and attempting to do so may be the problem with why your hreflang directives are not being detected. You should move to having a single robots.txt file located at https://www.clientname.com/robots.txt, with a link to a single sitemap index file. That sitemap index file should then link to each of your two UK & US sitemap files.
You should ensure you have hreflang directives for every page. Hopefully after these changes you will see things start to get better. Good luck!
Hey NeatIT!
I see you have a working solution there. Did you have a specific question about the setup?
I did notice that your setup cane sometimes result in chaining 301 redirects, which is one area for possible improvement.
Let me know how we can help! 
If you have many URLs from the old site in the index that are all in the same directory (or a handful of directories) you can quickly and easily remove whole directories of URLs from the index via Google Search Console. We have found it to work very quickly.
Go into Search Console, selected ‘Remove URLs’ under ‘Google Index’ in the left hand menu.
Add the page or folder you want to remove, and click next. If you add the homepage, that's the same as all pages on the site. If you add a folder you'll get three options under the ‘Reason’ drop down.
One of those options is ‘Remove directory’. Select that.
Ok, cool. To reiterate - with escaped_fragment you are just serving the same content in a tweaked format and Google recommend it rather than frown upon it. Good to be sure though.
See you at SearchLove!
Hi,
I am not sure I follow your concerns around serving an alternative version of the page to search engines - is that concern based on concerns it will be frowned upon or technical concerns?
Using the escaped_fragment methodology would work for your purposes, and would be the best approach. If you have technical concerns around creating the HTML snapshots you could look at a service such as https://prerender.io/ which helps manage this process.
If that doesn't answer your question, please give more information so we can understand more specifically where you concerns are.
It seems like the issue is a bug in the way Google handle data from your site ('null' being computer speak for 'empty', and often appearing after buggy handling of data). However, it seems that the indication from Umar is correct, and that this buggy data handling is likely prompted by crawling issue so that is the best place to start.
Is it possible for you to give a clearer description of the categories? You say they are different products but that one is a second category of the other
Does the page you want to rank show up for any other searches? In your analytics are you getting any traffic from Google to that page?
So, there are lots of 'ifs' here, but the primary problem I see with your plan is that the CDN will return the content to Googlebot without the request hitting your server so you won't have the option to serve different headers to Googlebot.
Remember that every page is the main HTML content (which may be static or dynamically generated for every request), and then a whole bunch of other resources (Javascript and CSS files, images, font files etc.). These other resources are typically static and lend themselves far better to being cached.
Are your pages static or dynamic? If they are dynamic then you are possibly not benefitting from them being cached anyway, so you could use the 'vary' header on just these pages, and not on any static resources. This would ensure your static resources are cached by your CDN and give you a lot of the benefit of the CDN, and only the dynamic HTML content is served directly from the server.
If most of your pages are static you could still use this approach, but just without the full benefit of the CDN, which sucks.
Some of the CDNs are already working on this (see http://www.computerworld.com/s/article/9225343/Akamai_eyes_acceleration_boost_for_mobile_content and http://orcaman.blogspot.co.uk/2013/08/cdn-caching-problems-vary-user-agent.html) to try and find better solutions.
I hope some of this helps!
I would say a bit of both. It is fine to repeat your primary keyword phrase several times on the page; the number of times depends upon the amount of content. SEOmoz's On Page tool recommends 4 repetitions. However, you should also try to use some synonyms and secondary target keyword phrases also.
A good resource I saw posted today which might be of interest:
Hi there!
Ok, it is difficult to know all the ins and outs without looking at the site, but the immediate issue is that your robots.txt setup is incorrect. robots.txt files should be one per subdomain, and cannot exist inside sub-folders:
A **robots.txt **file is a file at the root of your site that indicates those parts of your site you don’t want accessed by search engine crawlers
From Google's page here: https://support.google.com/webmasters/answer/6062608?hl=en
You shouldn't be blocking Google from either site, and attempting to do so may be the problem with why your hreflang directives are not being detected. You should move to having a single robots.txt file located at https://www.clientname.com/robots.txt, with a link to a single sitemap index file. That sitemap index file should then link to each of your two UK & US sitemap files.
You should ensure you have hreflang directives for every page. Hopefully after these changes you will see things start to get better. Good luck!
Ok, cool. To reiterate - with escaped_fragment you are just serving the same content in a tweaked format and Google recommend it rather than frown upon it. Good to be sure though.
See you at SearchLove!
Putting aside server load / config issues, and from the pure SEO point of view.
No, you shouldn't have any major issues with that many 301s. However, what you might find is that depending on the size of your site and the frequency of Googlebots visits that some of these pages take a long time (months) to drop out of the index and be replaced by their newer alternatives. This normally isn't cause for alarm.
In some instances you might end up with pages that now have now links to them (as their parent categories were all redirected also) and so seem to get stuck and never get recrawled by Google to update. In a couple of instances I have had success using XML sitemap files that just include these 'blocked' pages (the old URLs still in the index) to prompt Google to recrawl them.
Also there is Google Webmaster Tools feature to 'crawl as Googlebot' which then prompts you to 'submit to index' which you can use to prompt recrawls on a per-page basis (but you have credits here, so should only be for the more important pages).
Best of luck!
Just to chime in on this, albeit maybe a little late now... I had the same thought as I was reading through this with using rel=canonical to point the old pages to the new for now, so the search engines don't have any duplicate content issues until a 301 redirect can take over when the new site is fully launched.
However, depending on your rollout schedule, this would mean that the SERPs would soon be indexing only the new pages. You'd need to ensure that the traffic diverter you are using would handle this. Otherwise you could put the rel=canonical on the new pages for now, which would avoid the duplicate content until you are fully launched. Then you'd remove it and 301 redirect the old pages to the new.
Just something you maybe want to think about! Hopefully your traffic diverter can handle this though.
I will just add that you need to be very careful. Maybe try the ones out that everyone has kindly suggested and then use SEOmoz Campaigns or other tools to check the quality of it.
This Whiteboard Friday post from last week has one of Bing's team pointing out how it is very important that your sitemap is of a high quality (no 404s, 302s or 301s for example), or it could be ignored completely:
If you have many URLs from the old site in the index that are all in the same directory (or a handful of directories) you can quickly and easily remove whole directories of URLs from the index via Google Search Console. We have found it to work very quickly.
Go into Search Console, selected ‘Remove URLs’ under ‘Google Index’ in the left hand menu.
Add the page or folder you want to remove, and click next. If you add the homepage, that's the same as all pages on the site. If you add a folder you'll get three options under the ‘Reason’ drop down.
One of those options is ‘Remove directory’. Select that.
Hi David,
Google/Bing etc. have very few problems recognising such characters in the Latin alphabet. It looks like you are mainly concerned with umlauts, which Google handles intelligently. For example...
Google will identify the difference between a search for "Küchen" (kitchens in German) and "Kuchen" (cake in German) and offer up relevant results. This is true in Google US and Google UK, not just localised Googles.
Search suggestions work just fine with these characters, and even with the standardised way of rewriting them when there is no accessible way to type them (for umlauts this is with an e following the letter). For example, in Google.de, type "Kue" and you will be given the suggestion "Küchen".
You are mainly concerned with brands, which muddies the waters a little because many people in English speaking markets won't bother/know how to type the umlauts. However, Google normally handles this well and recognises the intent.
I would recommend you ensure you consistent use the brand name with the foreign characters, as intended. Google/Bing and co. shouldn't have any problems. Which HTML encoding you use is by the by, in my opinion, as long as the characters are rendering correctly.
Hi Panos,
I don't necessarily disagree with Eric's answer, but I wanted to answer from a different point of view. I'm going to assume you really want or need some refresh mechanism built into the page.
In which case I'd agree that a Javascript approach using AJAX is probably a better solution. It will mean that users only need to load the new article headlines, and not the whole page, so the strain on your servers should be reduced. Furthermore, I find it a neater solution all around anyway - you could provide a notice 'new headlines available' that people click to refresh the articles list. This might be the best of both worlds?
Either way, meta refresh isn't as flexible, isn't as clean, and will put more strain on your servers.
Good luck!
-Tom
So, there are lots of 'ifs' here, but the primary problem I see with your plan is that the CDN will return the content to Googlebot without the request hitting your server so you won't have the option to serve different headers to Googlebot.
Remember that every page is the main HTML content (which may be static or dynamically generated for every request), and then a whole bunch of other resources (Javascript and CSS files, images, font files etc.). These other resources are typically static and lend themselves far better to being cached.
Are your pages static or dynamic? If they are dynamic then you are possibly not benefitting from them being cached anyway, so you could use the 'vary' header on just these pages, and not on any static resources. This would ensure your static resources are cached by your CDN and give you a lot of the benefit of the CDN, and only the dynamic HTML content is served directly from the server.
If most of your pages are static you could still use this approach, but just without the full benefit of the CDN, which sucks.
Some of the CDNs are already working on this (see http://www.computerworld.com/s/article/9225343/Akamai_eyes_acceleration_boost_for_mobile_content and http://orcaman.blogspot.co.uk/2013/08/cdn-caching-problems-vary-user-agent.html) to try and find better solutions.
I hope some of this helps!
11/3/2023
Explore the imminent paradigm shift in search behavior as explicit context explodes in SEO queries.
1/27/2023
In today's Whiteboard Friday, Tom digs into his research on Apple’s moves in search, specifically their recent launch of what he believes is a search engine, how it works, and how they could possibly hope to compete with Google in the future.
12/30/2019 Site speed is incredibly important, but we often rely on the same old methods for improvement. Tom Anthony covers some unusual and surprising tricks to improving site speed that you may not have considered.
4/9/2018 How does Google measure a page's loading speed? New information confirms that Google is using Chrome usage data to evaluate site speed, rather than Googlebot. Learn just what that means, why it matters, and what it could portend for users.
5/30/2016 In the way websites once replaced phones, will intelligent personal assistants like Siri and Cortana soon replace websites? Tom Anthony examines 4 trends that are contributing to the IPA takeover and has some advice on how you can keep up.
5/20/2016 App search is growing and changing, and there's more opportunity than ever to both draw customers in at the top of the funnel and retain them at the bottom. In today's special British Whiteboard Friday, Tom Anthony and Will Critchlow of Distilled dig into everything app search and highlight a future where Google may have some competition as the search engine giant.
1/18/2016 The idea of Do-Know-Go queries have long been useful for SEOs in understanding and categorizing user search intent for web search. But with the increasing usage of Intelligent Personal Assistant apps, this framework requires an update to remain relevant. In today's post, Tom Anthony discusses a new layer to the Do-Know-Go model and speculates about the future of search.
7/13/2015 With the increasing number of SEO directives and annotations available, and the ever-changing guidelines around how to deploy them, it is important to automate whatever areas possible. In this blog post I'm going to share an open source Python library which makes it easy to read the hreflang entries from a page and identify errors with them.
4/28/2015 We increasingly use apps, wearables, smart assistants, smart watches and smart TVs for searches, and none of these are returning the 10 blue links of days gone by. These technologies and trends are intersecting and converging, giving rise to a new paradigm of search: Data-Driven Search.
11/3/2014 Google's Physical Web project is a new initiative to establish a standard for making 'smart' objects which people can interact with via their smart phones or similar devices. This post introduces the Physical Web concept, how it works, and will make a couple of first predictions about how this might affect SEO in the future, including how interactions will be ranked.
6/16/2014 You have probably heard about IPv6, but you might remain a bit confused about the details of what it is, how it works, and what it means for the future of the Internet. This post gives a quick introduction to IPv6, and discusses possible SEO implications that could follow from IPv6 being rolled out (touching specifically on the concept of C-Blocks).
12/23/2013 Since the Panda and Penguin updates, the SEO community has been talking more and more about machine learning, and yet often the term still isn't well understood. We know that it is the "magic" behind Panda and Penguin, but how does it work, and why didn't they use it earlier?
8/26/2013 Gone are the days when a search query simply involved a string of words. Today's queries include all sorts of implicit data — location, device, etc. — that add to the search engines' understanding of what searchers really wanted to find. That fact has some very real implications for the way we think about keywords.
12/11/2012
[htaccess SEO] In the Moz Q&A, there are often questions that are directly asked about, or answered with, a reference to the all-powerful .htaccess file. I've put together a few useful .htaccess snippets which are often helpful, but are generally underused by SEOs.
9/17/2012 People talk a lot about APIs in the SEO industry - the tools you can build with them, the competitive analysis data you can access, thereports you can automate. However, we tend not to discuss the wider picture, the thousands of APIs out there for other things, and most importantly the profound effect that APIs are going to have on the web, and thus the SEO industry, in the coming decade.
2/20/2012 Recently Mat Clayton from Mixcloud provided a great snippet of Javascript that could be used to record whether visitors to your site were logged into Facebook or not. I extend that idea to present similar code for Twitter and Google+ and then wrap it all up in Google Analytics goodness. Using this code you can monitor which social networks your website visitors are logged into.
12/12/2011 This weeks free tool is one that allows you to automatically download a list of backlinks to any URL, broken down not only by TLD/subdomain but also by the language the page is written in (using code from Google's Chrome browser). Great for analysing your own international sites, or as part of your competitor research.
11/21/2011 Measuring CTR data in search engine results is notoriously difficult, and with Google's recent move to HTTPS for logged in users it is probably going to keep on getting harder. What I wanted was a simple way to measure the change in CTR for a given search query's results when I adjusted entries, but nothing existed.... so I built the SERP Turkey tool.
9/19/2011 What I'm going to share today is a tool I've built that automates a process to evaluate a domains link profile looked and whether it stands out against other domains in its niche. I'm going to show you the real data I saw for a client who I knew had bought links and how you can use the same tool to identify domains that have maybe bought links or other anomalous link profiles. It all works via Google Docs and the Linkscape API and only takes a couple of minutes to run the full report.
5/15/2011 Faced with a new client, and having established a list of keywords they need to target, you want to evaluate the competition to find out what sites are dominating the SERPs for these keywords. However... being an SEO you're a busy guy (or gal), and you need it done right now. I've built a Google Docs tool to automagically do exactly that and this post will walk you through it. The ...