Welcome to the Q&A Forum

Moz can't crawl his root domain (i tested using page grader) or any other URLs on his site,

So while the special chars might not help, i dont think thats the issue here :). Atleast, not the only one.

https://moz.com/researchtools/on-page-grader?keyword=preguntas&url=https%3A%2F%2Fwww.respuestasparaelhombre.com

Its possible your SSL is the issue, do you know if its an SNI cert? Moz crawler doesnt yet support SNI and a lot of recently issued certs are SNI ones...

Also, if your using Cloudflare, those certs are SNI as well.

http://serverfault.com/questions/506177/how-can-i-detect-if-a-server-is-using-sni-for-https

All reference i can find to this error is firewall issues. You will probably need to check with your hosting providers that they arnt blocking moz.

a similar post made last year:

http://moz.com/community/q/moz-tools-are-returning-url-is-inaccessible

DA on a couple of my accounts sank by 2 ~ 3 in the last update, maybe a minor update to the metrics triggered something...

A few more links about this, that i was able to dig out:

http://live.theverge.com/sundar-pichai-google-mwc-2015/

and the full thing:

http://www.mobileworldlive.com/mwl-extra-sundar-pichai-svp-of-products-at-google

Hi James,

Thanks, I know it's not a 1 day fix, it's just somewhat frustrating to find that Moz doesn't yet support technology that is fast becoming standard, and thats been around for years. I look forwards to hearing when Moz does have an ETA for this! The Crawler is the main reason I created a Moz account, so i can't emphasize enough how much of an effect not having a long term solution at least announced soon is going to have on my decision to remain a Moz subscriber in the long term.

If it makes any difference, turns out Bing doesn't support it either, so something else to remember as more people start seeing crawler errors / lack of rankings that they may not understand!

http://webmasters.stackexchange.com/questions/77267/bing-and-lack-of-sni-support

And thank you for confirming the issue :).

Examining the server logs yourself probably wont help your understanding of the issue unless you know what your looking at specifically. On the Yahoo note, i have found Slurp to be really bad in the past, but no legitimate bot should be able to bring down a properly configured web server, especially an 'enterprise-level' one.

I would check your .htaccess and apache settings for bad redirects (or web.conf if on windows) before considering banning the bot. Other things to check would be website code or if a bot hits a massive and horribly optimised Database Query for example, that could bring the server down.

Ask IT exactly what the bot did that caused the server to go down, they should atleast be able to tell you that. If not then they need to run load tests against the website itself to try and reproduce the scenario and thus debug the issue, if indeed there is one.

Tl;dr :- Normally bad config or code / queries are to blame for this kind of thing. I'd review that before blocking a bot that crawls hundreds of thousands of other sites without issue.

The main problem here is that i can't disable SNI unless i pay for a business account ($200 USD per website, per month), except for the enterprise account everything else is Pro which is more than enough for my clients (no way i could justify $200 a month just so moz can crawl).

SNI is a technology thats been supported since IE7 and is now standard in all browsers, so still not supporting it now is just a little insane....

But as i said, this is going to be effecting everyone using SSL under cloudflare (which is probably everyone using cloudflare) so could you guys please make this a priority? its not just an edge case issue.

Hi All,

Recently i've noticed that Moz can no longer crawl any of my wbsites that are behind cloudflare. After speaking to CF support, we have come to the conclusion that a recent change by CF to force SNI for all SSL domains is the issue.

As this is likely to affect almost everyone using Cloudflare could anyone please confirm this?

In the mean time, one of the websites currently blocked is on CF enterprise, so we've requested SNI be turned off. Will test crawl again when thats done to confirm SNI is the issue.

Since im here im going to respond to this with a couple of things.

"While Cloudflare offers many optimization services and DDoS protection, many of them break Magento and Wordpress functionality."

I've not had any issue with this. Typically if something does break in WP or Magneto its pretty simple to fix. Personally, i dont run WP sites without cloudflare, its too much of a security issue.

" Implementing caching and Google PageSpeed with Nginx or Apache shows much better performance,"

You should do this anyway....

" Worse though, Cloudflare doesn't allow you to use your own SSL on any plan under $200/moth"

Sure it does, i run 40 + sites on standard Comodo SSL certs on Cloudflare pro without issue.

"If you're a Pagespeed/YSlow nut, they you'll really be upset to learn Cloudflare adds a security cookie to everything, so there is no way to have serve content on a cookie-free domain when using Cloudflare DNS"

Thats a fair point, though perhaps not such an issue in most situations.

"Customer service is also slow and mostly automated, so don't expect great service on unpaid plans."

Again i dont really find this. Most of my accounts are CF Pro but for those that arn't, i dont notice any difference in support level. I know Business and Enterprise plans get faster support but for $200 + a month, i'd expect that anyway.

Hi All,

Recently i've noticed that Moz can no longer crawl any of my wbsites that are behind cloudflare. After speaking to CF support, we have come to the conclusion that a recent change by CF to force SNI for all SSL domains is the issue.

As this is likely to affect almost everyone using Cloudflare could anyone please confirm this?

In the mean time, one of the websites currently blocked is on CF enterprise, so we've requested SNI be turned off. Will test crawl again when thats done to confirm SNI is the issue.

I think your pretty much spot on. Google -can- crawl queries but they wont rank very well at all.

Your best bet will be to change: (just reading into this that looks like a 'get all jobs' query)

http://www.prospect-health.com/Jobs/?st=0&o3=973&s=1&o4=1215&sortdir=desc&displayinstance=Advanced Search_Site1&pagesize=50000&page=1&o1=255&sortby=CreationDate&o2=260&ij=0

to just

http://www.prospect-health.com/Jobs

There are loads of ways to remove the query string and keep the functionality, depending on the software powering the site, personally i'd fix up the search to POST data so that it keeps the url clean and add in the appropriate routes to create the path.

I have some experience of job search sites (having worked on a couple of the largest in the UK) and breaking URLS down something like this seems to work best. (depending on the data you have obviously)

<domain>/jobs/<location>/</location></domain>

You could also take a look at how other job sites structure their URLS, (monster.co.uk, targetjobs.co.uk, jobsite.co.uk etc)

Let me know if you need a hand and i'll see if i can be more specific. (you'll have to tell me what its running on though)

EDIT:: You should force lower-case on urls as well. Caps wont effect google but they arn't user friendly (miss types etc)

DA on a couple of my accounts sank by 2 ~ 3 in the last update, maybe a minor update to the metrics triggered something...

I don't believe so, or at least any effect should be minimal. a couple of links on the subject:

http://webmasters.stackexchange.com/questions/15393/is-a-url-with-a-query-string-better-or-worse-for-seo-then-one-without-one

http://moz.com/community/q/link-building-does-query-string-matter

The general consensus seems to be that so long as you're using rel=canonical your fine.

Personally i would suggest removing the silly query string as I've said above, as if nothing else it doesn't look very nice for users. (in the past we've had clients say users are asking if they've been hacked etc.. as an example)

Hi Jarno,

A couple of things to look at, did your site structure change when you switched to responsive? How has the responsiveness been done? (Poor implementation might cause you problems if google can no longer follow links correctly).

Are you testing while logged into a google account? Or logged out in an incognito window in Chrome? (The later will give you a much cleaner result as google adjust search results per user based on viewing habits). Or do you see these changes in Moz specifically?

Something to remember, mobile responsiveness has absolutally no known effect on desktop searches, only those carried out on mobile devices and so won't be a ranking factor for a large number of your keyword searches.

I would look at what going responsive changed though. If you like, PM me a link to the website (or link it here) and I’ll take a look to see if i can spot anything with the implementation.

You're correct, you can make it a little more generic though, without seeing all of your .htaccess file, try this:

Replace:

RewriteCond %{HTTP_HOST} ^companyname.co.uk [NC]
RewriteRule ^(.*)$ http://www.companyname.co.uk/$1 [L,R=301]

With:

RewriteCond %{HTTP_HOST} !^www. [NC]
RewriteRule ^ http%{ENV:protossl}://www.%{HTTP_HOST}%{REQUEST_URI} [L,R=301]

This is what could be called a wildcard redirect in that a direct copy paste should work for you with no need to edit. (you dont have to manually add in the correct domain name)

What it does:

• First it checks to see if the requested url has a www in it
• if it does -not- it then runs the rule, otherwise it ignores it.
• The rule first checks for http or https. Then adds in the www. followed by the domain and tld, finally adding the URI (/somepage/page for example).
• the L in square brakets means do not process anything else in the htaccess file
• the R=301 means that it will be a 301 (perminant) redirect.

If that still doesnt work for you, paste up your full .htaccess file, or you can send it to me directly if you'd rather and i'll take another look

"We went from Drupal to WordPress"

That seems like a very backwards move to me, that should be the other way around (My personal opinion anyway)

On the 404s though, any URL changes will cause problems, you will need to add in redirects to deal with those.

On a drupal specific note, if your old site was using the 'URL Redirect' module, you may have had a number of redirects in place in the database, these wouldn't have been in your .htaccess file and will no longer work after moving to wordpress.

We have a couple of major law firms as clients who do exactly this, run one website for each sector that they operate in and then have a central corporate site.

At the end of the day Google wont penalize you so long as the content is unique.

Hmm i cant see anything funny going on, this is essentually what googlebot should see when viewing the page. If you'd been hacked i would expect to see the wrong description here.

http://www.evolvedwebsites.com.au/googlebot/view.php?url=protectachild.com&keywords=evolved+websites%2C+website&submit=VIEW+AS+GOOGLEBOT

If the site has been live a week, you might just be in a bit of a blip. Best thing i can suggest is request a reindex / new crawl in GWMT and wait it out. Google can take 3 ~ 4 weeks to update listings properly when meta information changes (in my experiance atleast)

A couple of improvements i can suggest straight off:

You need to redirect all non-www traffic to www. or vice versa. Currently so far as google can see, you have 2 different websites, one on http://bassilimousineservice.com/ and one on http://www.bassilimousineservice.com/

Your URL structure needs to be improved,for example: http://www.bassilimousineservice.com/index.php/services/rates.html

Should be something like: http://www.bassilimousineservice.com/services/rates