Hi there,

While doing some research on the indexation status of a client I ran into something unexpected. I have my hypothesis on what might be happing, but would like a second opinion on this.

The query 'site:example.org inurl:index.php' returns about 18.000 results. However, when I hover my mouse of these results, no index.php shows up in the URL. So, Google seems to think these (then duplicate content) URLs still exist, but a 301 has changed the actual goal URL? A similar things happens for inurl:page. In fact, all the 'index.php' and 'page' parameters were removed over a year back, so there in fact shouldn't be any of those left in the index by now. The dates next to the search results are 2005, 2008, etc. (i.e. far before 2013). These dates accurately reflect the times these forums topic were created.

Long story short: are these ~30.000 'phantom URLs' in the index out of total of ~100.000 indexed pages hurting the search rankings in some way? What do you suggest to get them out? Submitting a 100% coverage sitemap (just a few days back) doesn't seem to have any effect on these phantom results (yet).

The website of a partner of mine was recently migrated to a new platform. Even though the content on the pages mostly stayed the same, both the HTML source (divs, meta data, headers, etc.) and URLs (removed index.php, removed capitalization, etc) changed heavily. Unfortunately, the URLs of ALL forum posts (150K+) were redirected using a 302 redirect, which was only recently discovered and swiftly changed to a 301 after the discovery. Several other important content pages (150+) weren't redirected at all at first, but most now have a 301 redirect as well. The 302 redirects and 404 content pages had been live for over 2 weeks at that point, and judging by the consistent day/day drop in organic traffic, I'm guessing Google didn't like the way this migration went.

My best guess would be that Google is currently treating all these content pages as 'new' (after all, the source code changed 50%+, most of the meta data changed, the URL changed, and a 302 redirect was used). On top of that, the large number of 404's they've encountered (40K+) probably also fueled their belief of a now non-worthy-of-traffic website.

Given that some of these pages had been online for almost a decade, I would love Google to see that these pages are actually new versions of the old page, and therefore pass on any link juice & authority. I had the idea of submitting a sitemap containing the most important URLs of the old website (as harvested from the Top Visited Pages from Google Analytics, because no old sitemap was ever generated...), thereby re-pointing Google to all these old pages, but presenting them with a nice 301 redirect this time instead, hopefully causing them to regain their rankings. To your best knowledge, would that help the problems I've outlined above? Could it hurt?

Any other tips are welcome as well.

I don't think there is a better way to name a dated document than by using a form of a date.

However, you might want to consider using a different format (because you've mentioned these posts were weekly), for example "Beauty Industry News - Week 13 of 2013". Just a personal preference though, no gains or losses in the search engines there that I'm aware of.

I personally don't think that changing your URL as you described will result in big increases in rankings on the organic search. Especially considering the work required (and the potential loss of incoming links to URLs you forget to redirect), I wouldn't recommend the change you've described.

If however, you really want to change the URLs, this is the structure I'd advice:

www.example.org/category-name/123-product-name

This allows people to cut a piece of the URL and land of your category overview page, shows them to what category a product belongs and keeps the amount of 'sub levels' to a minimum by including the id in the second level.

No. Actual text is slightly more effective than an image alt text. You might want to have a look at CSS image replacement techniques?

This one might be a bit outdated, but it contains some great directories: http://www.seomoz.org/directories

What exactly is an 'external' or 'internal' menu? Every menu that is on your website is internal by definition that it is 'on your website'?

Menus should be fully accessible with Javascript turned off. With that in mind you could add Javascript to make the menu function better for users that have Javascript enabled.

I was talking about the same concept you're describing when I mentioned category listings. The next / previous and related items sound exactly like the things that I would recommend to get links to the page > 1 items! Lastly, yes the canonical URL should be the page we're actually viewing and not always page 1.

You're welcome! It is a link from one page of your website to another, thus an internal link. I don't see how noindex,follow would change that. Yes, they will receive link juice. Because of the follow in the robots tag the pages (even though they aren't indexed) still pass link juice. Like I said in my original post, it is best to have other pages (such as category listings for example) link to these items as welll though.

From what I've read on the internet, it is best to "noindex,follow" all pages >1. This issue had bugged me for quite some time as well, and I've struggled to find good resources explaining why their solution was the best. Now that I've actually given the subject some thought, and finally managed to read some quality material on the matter, it all makes sense.

It's basically a checklist. Do you want search engines to

• index your paginated result pages: yes / no

• reach the items that are listed in your paginated result pages: yes / no

In most cases you don't want your paginated result pages to be indexed. With our without Panda, visitors get little value from actually viewing 'page 7' in your result pages. That actual page provides little or no value to those visitors. However, you DO want those items listed on these paginated pages to be crawled, especially when you don't have any other pages linking to them (which you should by the way). This boils down to:

• Don't nofollow your paginated links (because you want search engine spiders to reach them)

• Put "noindex,follow" in the meta robots tag for all pages >1 (thus page 2 and greater) so the engines will no index these paginated results, but will crawl on to the pages that are behind the listings

Good luck!

I'd say you can safely put this issue aside. As you've mentioned yourself, xml files don't need title tags, and I presume you aren't trying to have this page rank in the SERPS in the first place anyway. Perhaps one of the SEOmoz fellows can look into this?

From what I've read on the internet, it is best to "noindex,follow" all pages >1. This issue had bugged me for quite some time as well, and I've struggled to find good resources explaining why their solution was the best. Now that I've actually given the subject some thought, and finally managed to read some quality material on the matter, it all makes sense.

It's basically a checklist. Do you want search engines to

• index your paginated result pages: yes / no

• reach the items that are listed in your paginated result pages: yes / no

In most cases you don't want your paginated result pages to be indexed. With our without Panda, visitors get little value from actually viewing 'page 7' in your result pages. That actual page provides little or no value to those visitors. However, you DO want those items listed on these paginated pages to be crawled, especially when you don't have any other pages linking to them (which you should by the way). This boils down to:

• Don't nofollow your paginated links (because you want search engine spiders to reach them)

• Put "noindex,follow" in the meta robots tag for all pages >1 (thus page 2 and greater) so the engines will no index these paginated results, but will crawl on to the pages that are behind the listings

Good luck!

You'll have to wait a LOT more. Results such as these coming from a couple backlinks might take weeks if not months to actually process into higher ranks for specific keywords. Hang in there, keep getting great links, and eventually your rank will start increasing!

Not necessarily entirely 'their issue', because if Jeremy is linking with a 302 redirect without wanting to do so, this issue might occur on more places across his website.

To expand on the question by Alan: is this (1) an internal link between two pages on your website, (2) an external link from your website to another website or (3) a link from another website to your website?

Is this happening every day at 10AM? What happens between midnight and 10AM? Before midnight? Is this static IP on a blacklist? Does this static IP change over night? Where are you getting these 'impossible CAPTCHAs'? Where is 'Google blocking you'?

I would personally place the keywords at the end for clarity. It indeed seems unnatural to have the id as the final part of the URL. Even if that does indeed cost you a tiny bit of 'keyword power', I would glady sacrifice that in exchange for a more user-friendly URL.

Limiting the amount of words in the URL does indeed make it look slightly less spammy, but slightly less user friendly as well. I guess this is just one of those 'weigh the pros/cons and decide for yourself'. Just make sure the URLs don't get rediculously long.

Noindexed pages are pages that you want your link juices flowing through, but not have them rank as individual entries in the search engines.

• I think your legal pages should rank as individual pages. If I wanted to find your privacy policy and searched for 'privacy policy company name', I'd expect to find an entry where I can click and find your privacy policy

• Your search results page (the internal ones) are great candidates for a noindex attribute. If a search engine robot happens to stumble upon one (via a link from somebody else for example), you'd want the spider to start crawling pages from there and spreading link juice over your site. However, under most circumstances you don't want this result page to rank on itself in the search engines, as it usually offers thin value to your visitors

• Blog archive and category pages are useful pages to visitors and I personally wouldn't noindex these

Bonus: your paginated results ('page 2+ in a result set that has multiple pages') are great candidates for noindex. It'll keep the juices running, without having all these pretty much meaningless (and highly dynamic) pages in the search index.

This one might be a bit outdated, but it contains some great directories: http://www.seomoz.org/directories

I got the same question during a presentation I was giving recently and have to admit I didn't knew the answer on the spot either. Some thinking and discussing with others has given me a pretty clear picture on this though, which I will now try to pass on to you.

I don't think code order matters that much anymore. A couple years ago, when Google was crawling only a portion of a large webpage (mostly due to hardware restrictions), you'd better make sure your valuable text or links were placed in the first part of your webpage, otherwise it wouldn't get crawled at all! With Google crawling large webpage in total (if I recall some quote from Matt Cutts correctly he stated that they now index webpages over several MBs in size, as long as they contain enough valuable information).

With Google advancing their detection of the 'visual location' where text and links are placed on a webpage (see #5 on http://www.seomoz.org/blog/10-illustrations-on-search-engines-valuation-of-links), source code ordering will most likely have dropped in value as well. Using CSS styling, we can now order our source code pretty much at will anyway, which has changed it from a valid signal to a 'SEO trick' (just like adding a suffix to the URL has, see http://www.finishjoomla.com/blog/5/does-adding-a-suffix-to-my-urls-affect-my-seo/).

By 'viewing' (and perhaps manually categorizing or using machine learning http://en.wikipedia.org/wiki/Machine_learning) webpages Google will notice patterns in webpage source code. For example: 'that div containing a large amount of links, usually placed in an ul-li, often containing links to 'home' and 'contact' will most likely be your menu. Just like 'that div containing more text than any other div, often starting with a H1 or H2 tag, containing the most images and ending with a call-to-action' link will most likely be your page content area. Thus, Google doesn't 'know' whether a certain part of your source code is your menu, your sidebar or your page content, it deducts it by looking at common patterns.

(lol, my answer is more than three times as long as your short question!)

Perhaps HTML5 tags are used as one of the signals in determining which piece of source code is what. Seeing how easily one could manipulate these tags, I don't think it'll be a strong signal though. Of course it can be a good guidance for future web developers to identify pieces of source code!

Google is able to read CSS files (for example to determine if a link is hidden), but I don't think it will parse these files and apply them to the webpage to determine the visual layout of it. I think it would require a great amount of processing power (and time) to actually render a webpage, rather than sort out the pieces based on the source code like I described in my answer above.

Glad I could help!