Hi Marie!

Sadly, I don't use Xenu anymore either. Most of the solutions to find orphaned pages are either hit-and-miss manual methods (search OSE, search your server files). Or you could use a method like Agents of Value describes here.

Couple of posts that may help:

1. Find Orphaned Pages From Your Sitemap.xml File with Excel and IIS Toolkit

Requires IIS toolkit, which unless your installing on an external machine, isn't mac friendly

2. 4 Tips for Technical SEO

Ian has some great tips here, including:

• Search the server log files for every unique URL loaded over a 6-month period. Compare that to all unique URLs found in a site crawl. People have a funny way of stumbling into pages you’ve accidentally blocked or orphaned. Chances are, blocked pages will show up in your log file, even if they’re blocked.
• Do a database export. If you’re using WordPress or another content management system, you can export a full list of every page/post on the site, as well as the URL generated. Then compare that to a site crawl.
• Run two crawls of your site using your favorite crawler. Do the first one with the default settings. Then do a second with the crawler set to ignore robots.txt and nofollow. If the second crawl has more URLs than the first, and you want 100% of your site indexed, then check your robots.txt and look for meta ROBOTS issues.

3. Supposedly, Webseo has an automated option to find orphaned files, but I haven't used it nor can I vouch for it:http://www.webseo.com/

Hope this helps! Let us know what works.

Howdy richardo24hr,

One of articles that changed my SEO life was written by Rand back in 2010:

All Links are not Created Equal

The article is still valid today, and update link Panda and Penguin have further changed the landscape of links. Here's a few important points to keep in mind:

1. Multiple links from the same domain don't always help. It's better to have 50 links from 50 domains than 10,000 links from one domain. After the first link, other links from the same domain may pass value, but that value tends to diminish.

2. Since the Penguin update, sitewide, over-optimized anchor text can lead to penalties and/or filters targeting your keywords.

For example, a sitewide footer link (or sidebar link) that pointed to your site with optimized anchor text is often seen as non-editorial (as it's placed automatically by your CMS) and this could actually hurt you.

3. Google is getting better at sniffing out site with "administrative" relationships, and tend to devalue these links. So a site that links to you from 100,000 pages is likely to broadcast a relationship between the 2 sites to Google, so Google may possibly devalue these links.

Links from this site can help! But the danger is to overdo it in a way that can actually act counter to what you're trying to achieve.

The best links from this site, from an SEO point of view, would be:

• Editorial. Meaning they are linked to in the body of a text article, and not auto-generated by a CMS
• This means the links are varied in thier anchor text, and not over-optimized
• I would avoid sitewide links
• RSS links are tricky, but in general I don't see them adding much value. Although these links are often scraped and posted on 3rd party sites. In this case it's best to use generic or branded anchor text like the full url of your site: example.com

If I had to choose, one link on the homepage may well give you more value than 100,000 RSS links. There's probably an opportunity to do more than this, but I'd do a thorough link audit to look for as many over-optimized links from this domain as possible.

Hope this helps! Best of luck with your SEO.

Hi Evgeny,

Good question. For starters, one thing to keep in mind from the beginning is that link juice flowed to subdomians don't pass the same link juice as would links to the same domain. So while these links and/or redirects may help the individual storefronts to rank - assuming there is sufficient link juice behind them - it doesn't necessarily help your root domain.

Of course, the way around this is to link these individual storefront subdomains to your main domain, making sure to vary the anchor text and do it in a non-spammy, Penguin friendly way.

Okay, onto the main question. In my experience working with 100's of clients, the best way to get them to redirect to your site is anyway they can. Seriously, it's almost impossible to choose a single method that works for all vendors, so I think it's probably best to offer a variety of solutions, such as changes in DNS, server-side redirects, .htaccess , etc.

You may even need to offer tech support to manually make these changes for your client. Although this is a sticky area fraught with headaches. (I know from experience)

In some cases, it may pass better link juice if you merely have the vendors link to you, instead of going through the trouble of a redirect. Links can carry relevancy signals that 301's can't, and redirects can often loose much of their relevancy if the target page(s) differ too much from the original.

Regardless, if you choose to go the redirection route, you'll want all of your redirects to be 301's, no matter what method you choose. The URL in the browser will be your subdomain. (There are ways to do URL masking, but you don't want to go there)  A common practice is to have the name of the vendor in the subdomain, such as vendor.yoursite.com.

Hope this helps!  Best of luck with your SEO!

Hard to say what is meant by that video. Often, Google is purposely vague.

If the content is truly no longer relevant, I would 301 it to more relevant URLs on a page by page basis.  This will remove low performing pages from Google's index, and potentially improve your rankings.

On the other hand, if the content still has value but doesn't need to be front and center, a clearly organized archive based on date or some other organizational method should work fine.

Great answers here.

Keep in mind the idea of "keyword stuffing" is a fluid concept. A well-written Wikipedia article may use the same keyword 60 times without being flagged for stuffing.

Context is key. If it makes sense to use your keywords multiple times than it's usually fine. The biggest violators that we usually see are in the title tag "Buy Cheap Florida Homes - Florida Home Finder - Your Best Florida Home Broker" - ouch!

As for plurals, Google will rank different pages differently based on what they perceive as searcher intent. I don't know of any hard and fast rules here - some plurals are considered synonymous with the singular version, while other times it can make a difference in ranking.

Nice Article, Eliathah

Hi Daniel,

To add to what David said....

Look at the URL of this page that we are on now: http://www.seomoz.org/q/adding-keyword-to-your-url

Our CMS automatically generates the URL from the title of the question. Additionally, this also becomes part of the title tag. Many blogging platforms like Wordpress allow you to customize the URLs of your posts.

Hi Mikkel,

Like Chris, I spidered your site and couldn't find any links to /index.php files, which probably indicates one of two things:

1. You've fixed the problem - Yay!
2. Or Google is finding those links from external sources
3. Google found those links at one time in the past, and is still trying to crawl them.

In the Crawl Errors report in Google Webmaster Tools, if you click on the link of each 404, there's often a "linked from" source where you can see where Google discovered the broken link. This is really helpful in rooting out the cause.

Regardless, I'm going to go with #1 and optimistically believe that you were able to fix the problem.

Sorry for the late reply. Feel free to send me a PM. (not sure I can help, but more than happy to take a look)

It can be tricky blocking parameters with robots.txt. The first thing you want to do is make sure your are actually blocking the URLs. There are a few good robots.txt checkers out there that can help:

• http://tool.motoricerca.info/robots-checker.phtml
• http://www.frobee.com/robots-txt-check

You're file is probably going to look something like:

User-agent: *
Disallow: /*?nocrawl=1

... but this could vary depending on exactly you don't want crawled

+1 to Igal's suggestion of handling these via parameter settings in Google Webmaster Tools: http://support.google.com/webmasters/bin/answer.py?hl=en&answer=1235687

Finally, if your goal is to keep search results out of the index (it probably should be) then you should also highly consider using a meta robots NOINDEX tag on all search results pages. You can also slap a nofollow on links pointing to search results as this might also help Google steer clear of those pages.

Best of luck!

Edit: Here's what John Wu of Google Webmaster has to say...

"We show this warning when we find a high number of URLs on a site -- even before we attempt to crawl them. If you are blocking them with a robots.txt file, that's generally fine. If you really do have a high number of URLs on your site, you can generally ignore this message. If your site is otherwise small and we find a high number of URLs, then this kind of message can help you to fix any issues (or disallow access) before we start to access your server to check gazillions of URLs :-)."

Hi there,

You've got the right idea, but let me suggest another tactic.

It's true that search functions can generate 1000's of urls that all tend to look like one another. Google suggests that you keep search result pages non-indexed, as these pages offer very little value and create tons of duplicate content.

http://www.seomoz.org/learn-seo/duplicate-content

Here's one way to handle your situation:

1. Put a meta "noindex,follow" tag in your search pages header, like this:

This tells search engines not to index the page, but allows them to follow the links on the page and flow link juice.

2. Hopefully you have a good site architecture and ways for search engines to discover your content. After step one, you can put a directive in your robots.txt file to block that directory from being crawled.

Something like:

User-agent: *
Disallow: /search/

Which blocks anything in the search directory.

3. Find out if search engines have already indexed a lot of your search pages by performing a site: search in Google, like so:

site:yourdomain.com/search

If you find pages in Google's index that shouldn't be there, you can use Google Webmasters URL removal tool to take these out of the index. You can remove the entire search directory with a single request.

http://support.google.com/webmasters/bin/answer.py?hl=en&answer=1663427

This is a powerful and sometimes dangerous tool, so be careful!

4. Finally, if you'd like to add "nofollow" to your search results pages, this should be fine, but only after you've completed the steps above.

Keep in mind, this is only one possible solution. If you have significant link juice flowing through your search results, this strategy may not be the best. But in general, you want to keep search results out of Google's index, so I'm comfortable recommending this strategy for 90% of all cases.

Hope this helps! Best of luck with your SEO.