Welcome to the Q&A Forum

I've always stuck to Disallow and followed -

"This is currently a bit awkward, as there is no "Allow" field. The easy way is to put all files to be disallowed into a separate directory, say "stuff", and leave the one file in the level above this directory:"

http://www.robotstxt.org/robotstxt.html

From https://developers.google.com/webmasters/control-crawl-index/docs/robots_txt this seems contradictory

| /* | equivalent to / | equivalent to / | Equivalent to "/" -- the trailing wildcard is ignored. |

I think this post will be very useful  for you - http://moz.com/community/q/allow-or-disallow-first-in-robots-txt

As long as you dont have directories somewhere in /* that you want indexed then I think that will work.  There is no allow so you don't need the first line just

disallow: /directory/*

You can test out here- https://support.google.com/webmasters/answer/156449?rd=1

If you have mod_rewrite enabled on your server you could try something like this in your .htaccess file:

RewriteCond %{REQUEST_URI} ^(.).htm$
RewriteRule ^(.) /$1.php [R=301,L]

I can answer that question from a slightly different perspective.  I worked with software that parsed varying file types and csv files were so slow to process because of their size.  It was the same scenario writing to them.  So i'm not at all surprised at the slowness of exporting your data to them - not really Moz's fault.

This should go a long way to answering your question - http://moz.com/community/q/what-are-the-good-strategies-using-satellite-sites-in-seo

I'd disavow the domain.

It might tie in with either Penguin or Panda updates, although content is Panda and Penguin is links (URLs).

• Did you get a manual penalty notice in Webmaster Tools?
• Have you examined your link profile closely for obvious spammy, dubious, unrelated links?
• Are your inbound links heavy with keyword anchor text?
• Is your textual content unique and give value to your visitor?
• have you changed your meta content?
• is the drop relating to certain keywords
• how have rankings stacked up over the past 2yrs - have they fallen?

You need to examine closely so that all your work doesn't fall flat again.

Initially it depends who the visitor is.  This data doesn't tell you that it's a worker or visitor just browsing from that IP and your site is pretty random.

If you analysed the data and found some strong trends it might be worth investigating further, but it might be a little bit 'big brother'.  On the other hand you might want to contact those companies on a different footing, not mentioning the stats but knowing there is some kind of interest there - unless a competitor!

Ok, try secondary dimension - Visitors >> Network Domain

For example, you may draw some data from Service Provider 'single static ip addresses'.

Unless I am missing something this data is the ISP that the user uses to connect to the internet. So the actual visitor could be anyone.

I would monitor the links in OSE.  Any whiff of dubious links disavow the domain, any more disavow the new domain etc, etc.  They will soon get tired of wasting their resources, if indeed they carry out their threat in the first place.

• Has the level of (not provided) increased in rough proportion to the drop in branded traffic?
• Have you double checked you are ranking #1 (or same position) for all your branded keywords?
• Are alternative Adwords ads sitting above your results (apparently 40% of people don't know they are ads - which should now change with the distinct yellow 'Ad' icon)?
• Are you still strongly competing in your market?
• Is it just brand traffic, have you analysed other keywords?
• is your branding looking like 'keywords' used in anchor text - small chance Google might see it that way?

I would suggest blog commenting has no real benefit for a link profile as it has been hugely abused.

However, there are very good reasons to do this with good authority, somehow related, spam free sites using quality comments.

• If no-follow they still give your link profile a natural mix
• Opportunity to develop relationship with the site
• Opportunity to write informative,authoritative comments to be seen as an authority
• Slight possibility for referral traffic
• still a small signal if part of a much bigger link strategy

Took another look, also looked at documentation/online and don't see any way to exclude URLs from crawl based on response codes.  As I see it you would only want to exclude on name or directory as response code is likely to be random throughout a site and impede a thorough crawl.

Seems drastic.  What caused the drop in rankings - penalty, algorithmic penalty, competition, content, poor link profile...?

Yes a toxic domain will infect a new domain if 301 is implemented.

I would lean towards cleaning up the existing domain.  Even if you end up disavowing every linking domain the existing domain is likely to have created more trust than starting from scratch. If it is a manual penalty ensure you document all steps to try and clean up so you can detail in the reconsideration request.

"Websites that provide content for different regions and in different languages sometimes create content that is the same or similar but available on different URLs. This is generally not a problem as long as the content is for different users in different countries. While we strongly recommend that you provide unique content for each different group of users, we understand that this may not always be possible. There is generally no need to "hide" the duplicates by disallowing crawling in a robots.txt file or by using a "noindex" robots meta tag. However, if you're providing the same content to the same users on different URLs (for instance, if both example.de/ and example.com/de/ show German language content for users in Germany), you should pick a preferred version and redirect (or use therel=canonical link element) appropriately. In addition, you should follow the guidelines onrel-alternate-hreflang to make sure that the correct language or regional URL is served to searchers."

My scenario has completely independent domains served from a central CMS/database, so all that differs is the language content:

• example.co.uk
• example.de
• example.it
• etc...

I'm just not so sure I should be interlinking with every homepage on ever page of each domain with a do follow link, so hesitating on reverting.

thanks David.  No there is no group homepage as such - each domain is only different in language so each homepage is the same and important to rank in the relevant Google engine.

The only 'central point' as such is a .com which is a page linking out to each domain.

A site i'm working on has about 12 language domains - .es, it, .de etc.

On each page of every domain the header has links to every homepage.  At the moment these are all set to no-follow as an initial step to stop potential link profile issues spreading around.

Moving forward i'm not totally sure how to handle these links.  On one side I see and agree that no-follow is not necessary, but do-follow is just filtering out and weakening link juice.  What is the best way to handle this scenario?

Doesn't Apache mod_rewrite do the trick?

pagerank has no direct impact on ranking. It's just a tiny signal - a site with little pagerank can outrank a site with high pagerank.

In any case the rank you see is the public level which is never current and Google seldom does an update.  I ignore pagerank.

If you 301 a toxic domain then the toxicity will pass on to the new domain.

The first thing to do is determine if there is an algorithmic penalty applied to the domain, which could relate to either content or the link profile.

I don't think you can filter out on response codes.

However, first I would ensure you are running the right version of Java if you are on a 64bit machine.  The 32bit version functions but you cannot increase the memory allocation which is why you could be running into problems.  Take a look at http://www.screamingfrog.co.uk/seo-spider/user-guide/general/ under Memory.

Forum links have little value, so I would not participate just for the links.  But if you've something to say, can build your authority as a source of good knowledge. you'd get a weak sniff of link juice and possible referral traffic - and possible linking opportunities elsewhere on the back of that work through relationship building.

There is some excellent stuff in here - http://moz.com/community/q/best-to-spend-marketing-budget-on-high-quality-articles-or-link-building-services that might help sculpt your attitude/approach to a strategy.

A new site can dance all over the place and stable good rankings are dependent on so many factors including

• competition
• architecture
• content
• optimisation
• link profile
• authority
• and on...

A completely natural link profile will have a range of authority sites with the strongest group probably being around 35 to 50 as they are probably more common, with a few either end.  In that link profile will be a healthy smattering of no-follow links because otherwise it would be engineered.  You will not be penalised/disadvantaged for having 100 no-follow links and 2 do-follow, as with 2 no-follow links and 2 do-follow.

I would keep focusing on your social media channels along with sourcing opportunities on good authority sites you can get a do-follow link on like interviews, articles, reviews etc.  Also bear in mind that good content on a popular site with a no-follow link can still draw referral traffic.

There is nothing wrong with comments as long as the only reason for them is to participate and add value.  If there are any hints of spam, off-topic or piles of vague 'great post' replies on the post or elsewhere on the site then leave alone.

I'm going through the same issue with a client with 2 offices - both are shared.

I'm taking heed of the advice given at http://moz.com/community/q/local-seo-how-to-handle-multiple-business-at-same-address and http://moz.com/community/q/multiple-businesses-at-the-same-address-avoiding-google-places-trouble

Go into the Adwords account >> hit Help on the right >> you should have a phone link within Contact Us

Are the addresses unique and not shared with any other business? https://maps.google.co.uk/?q=Tall+Trees+Cottage%2C+Winkfield+Road%2C+ASCOT%2C+Berkshire%2C+SL5+7EX

I've heard of some real strange behaviour if the business does not have a fully unique address in some cases getting merged with another business, although I cannot see the connection here.

I would give them a call.  A ticket is then raised and they are usually very helpful.

Absolutely agree with Frederico.  I've had first hand experience of infected domains poisoning when coming in as 301's.  Always make sure the link profile is clean before redirecting.

I'm working on a site that has 10 languages served from centrally located core files in Magento.  So each language has its own TLD with localised content served from SQL.  GWT has also had the preferred country set for each domain.

The problem is that each and every domain is indexed in each of the local Google indexes.  In DE Google the FR homepage is ranking higher for the brand keyword.  I kind of think I am wasting my time with hreflang but would like some advice whether this is an option or clues how I can handle this situation best.

I am working on a revamped multi-language site that has moved to Magento.  Each language runs off the core coding so there are no sub-directories per language.

The developer has created sitemaps which have been uploaded to their respective GWT accounts.  They have placed the sitemaps in new directories such as:

• /sitemap/uk/sitemap.xml
• /sitemap/de/sitemap.xml

I want to add the sitemaps to the robots.txt but can't figure out how to do it.  Also should they have placed the sitemaps in a single location with the file identifying each language:

• /sitemap/uk-sitemap.xml
• /sitemap/de-sitemap.xml

What is the cleanest way of handling these sitemaps and can/should I get them on robots.txt?

The penalty is for links rather than thin or duplicate content, so at this stage I would primarily focus on your link profile.  If your content is indexed first with social signal pointing to it then Google will (should) know you are the originator.

You will need to systematically go through every single link coming back to the site.  If you have any URLs 301ed to your site then those link profiles will need close examination too.  If you have had a manual penalty you can't mainly get it right, you have to completely clean out anything with a whiff of toxicity.  Keep a close record of all your activities, possibly using a tool like Rmoov, when your have exhausted manual attempts use Disavow then file a reconsideration request.  There are some good threads and posts on here covering all these aspects.

I don't think there is a problem in removing the 301s.  You are not doing anything underhand.  The 301 is in place because for whatever reason you have decided it is better for your visitor to reach the redirected page.  There could be many valid reasons why the original pages are now valid, you aren't always to know a redirect is definitely temporary. Just in the same way you may need to amend a 301 to a new URL.

However, I would introduce the URLs back in small batches and monitor rankings and traffic.

In comparison to the sites overtaking you your domain authority and page authority is low.  You need to seriously work on your link profile which seems full of press releases and syndicated articles.  This strategy will at best not do you any favours.

Sean gives some good points.

What I would also suggest is look for not just fancy keyword sales stuff, but what are they going to do with you under the hood, what's currently wrong with any of the on-page elements, how are they going to fix those, what impact are those changes likely to have on organic ranking.

Having a list of keywords and saying here are some keywords, here are the monthly search volumes, if you get to top 3 for x,y and z you will be getting 'this' amount of traffic is not enough.  There is so much more, what strategy do they have for content development and more importantly how do they approach link development - what is their overall philosophy in these areas.  Can they show you any evidence of overall growth of sites.

Easy to be seduced by keywords, but far more difficult to start over or come back from an algorithmic or manual penalty.

This is all to do with the technical handling of non ASCII characters and how they have to be presented in the URL.  If the browser is set to UTF-8 then it should be able to read the percent encoding. Take a read of these:

http://www.jamesburton.net/url-design-for-international-seo/

http://en.wikipedia.org/wiki/Percent-encoding

http://stackoverflow.com/questions/3907995/how-to-support-utf8-japanese-arabic-spanish-urls-in-php

The client does not want to change the HTTPS structure and insists all URLs must be HTTPS.  So I am back to square one.

I want the homepage to be HTTP (when a user is not logged in) but I am hitting a brick wall with the client. So at present we have good links coming in to the HTTP url but the indexed is HTTPS and the internal redirect in DNN is a 302 to HTTPS.

I would try and map each URL of each community site to a corresponding URL within the regional site where possible, rather than pass all link juice to just the homepage.

Something like:

RewriteCond %{HTTP_HOST} ^www.communityx.com$ [NC]
RewriteRule ^/some-directory/$ http://www.regionaly.com/new-directory{ [R=301,L]

This is a very useful article which I hope you find of help - http://www.blindfiveyearold.com/what-is-a-c-block-ip-address

If they are internal links you can have and do what you like.  However the key is user experience.  Have you taken a look at you analytics to see how popular the links are and the visitor behaviour with them?  At a guess, if they look keyword engineered they can be off-putting.

that's great thanks.  NoFollow in Chrome is not picking up.

but the rel attribute "nofollow" is assigned to an 'a' element?

I just wanted to quickly run this past someone.  I have some footer links I want set to nofollow:

<map <span="" class="webkit-html-tag">name</map>="Map"><area <span="" class="webkit-html-tag">shape="rect" coords="10,39,73,101" href="URL" target="_blank" alt="some text">

should this be

<map <span class="webkit-html-tag">name</map <span>="Map"><area <span class="webkit-html-tag">shape</area <span>="rect" coords="10,39,73,101 <a <span class="webkit-html-tag">href="URL" rel="nofollow" target="_blank" alt="some text"></a <span>

Want to check before I advise. If this is not the way how can I fix?

I don't know what the image is but I would imagine it is likely to be promoting the site somehow. If that's the case it must be no-follow.

"Also, the bounce rate on these referrals was over 50%, compared to a site average of around 20%, so the web visitors didn't enjoy it either."

That's a good point - how do visitor view the links.

Actually some of the content isn't too bad.  Yes my example came from the nabcep-entry-level-training/ page.  The key is writing for your audience.  It seems a paradox but forget the keyword when writing your content, write around and expand on that keyword, or get a specialist copywriter to do the content for you.  Get the keyword in there 3-4 times and the rest of content if well written will semantically back-up the keyword.

9% is high, but don't sweat on precise percentages, rather a well written page with mention of the keyword.

If you are using Wordpress install and use Yoast Wordpress SEO plugin as one of the pass or fail criteria is keyword density.

Did you follow through to this - http://moz.com/blog/googles-multi-week-algorithm-update

May be of help?

I completely agree with Sheldon.  I've been in the position to pre-empt a potential slap by requesting removals (repeatedly) then Disavow.  I am no expert but in my opinion as long as there are removal requests as backup and to put in Sheldon's words the pruning is for health and diseased reasons rather than purely ornamental it shows good intent.