Why can no tool crawl this site?
-
I am trying to perform a crawl analysis on a client's website at https://www.bravosolution.com
I have tried to crawl it with IIS for SEO, Sreaming Frog and Xenu and not one of them makes it further than the home page of the site. There is nothing I can see in the robots.txt that is blocking these agents.
As far as I can see, Google is able to crawl the site although they have noticed a significant drop in organic traffic.
Any advise would be very welcome
Regards
Danny
-
I would look into finding a method to redirect via your server rather than with javascript. This will ensure that bots can properly crawl your site.
I would also add hreflang tags which should help Google with the multiple language versions of the site.
Also in the short term you may want to do something like add a link or a delayed meta refresh just in case someone either has javascript disabled or is using script blocking extensions. This will make sure they at least see something instead of a blank page.
-
Really helpful and much appreciated - many thanks!
Danny
-
Yes that's what I said CleverPhD, I just couldn't type that fast today.
Only joking Thanks for expanding on the subject.
-
To expand on Dean's point.
If you look at the source code on https://www.bravosolution.com/ you get a bunch of JavaScript (shown below). It is basically looking at the users location and the sending them to the appropriate version of your website based on country. This is why here in the US we are sent to https://www.bravosolution.com/cms/us
Many spiders/tools (and Googlebot was not really good at this until recently) are not good at (or do not do any) crawling and executing on JavaScript so they get stuck when they hit your home page.
If you want to evaluate any of your localized sites, just run those URLs through various tools like screaming frog etc. You would then ask, "Well, how do I know that my main https://www.bravosolution.com is working properly for SEO?". I don't have as much background in how to optimize for international SEO, but you can do a several things to start with.
-
Google anything having to do with Aleyda Solis and International SEO. She posts a lot of stuff here at Moz and is pretty sharp on this stuff. There may be a more appropriate way to redirect international clients from your main page that how you are executing.
-
Run your home page through Google Webmaster Tools under Crawl > Fetch as Google. See what the page looks like
-
Double check your robots.txt to make sure you are not blocking any folders that would contain a JavaScript library. Based on the code below, I do not see you referencing any external libraries, but if you are dependent on JS to send Google, it would be worth having your developer check things
-
As with everything on what to do, it all depends. If all of your local country sites are independently ranked and successful, this main website may nor may not be doing you any favors currently if it is just a pass through with no domain authority to start with. Spend time on step #1 to see if there is anything else worth doing.
Cheers!
name="description" />
-
-
Yes, it should redirect you to the correct country version based on your IP. But I still can't crawl the site from the home page
-
Cheers Bryan - much appreciated. It's driving me crazy!
-
Hi Danny,
Have you looked at the site via http://web-sniffer.net/
It would appear that the home page is just a JavaScript redirect.
I was redirected to https://www.bravosolution.com/cms/us which then could seen via Sreaming Frog.
The reason for my (default) redirect is given by web-sniffer as:
DEFAULT CORPORATE if ( path == '' ) { path = '/cms/us
-
Interesting. I verified the robots file and tried running through screaming frog... nothing. I' will dig into this with my dev team to try and get you an answer asap.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Does anyone know the linking of hashtags on Wix sites does it negatively or postively impact SEO. It is coming up as an error in site crawls 'Pages with 404 errors' Anyone got any experience please?
Does anyone know the linking of hashtags on Wix sites does it negatively or positively impact SEO. It is coming up as an error in site crawls 'Pages with 404 errors' Anyone got any experience please? For example at the bottom of this blog post https://www.poppyandperle.com/post/face-painting-a-global-language the hashtags are linked, but they don't go to a page, they go to search results of all other blogs using that hashtag. Seems a bit of a strange approach to me.
Technical SEO | | Mediaholix0 -
How long does it take for Webmaster Tools to index a site?
I submitted my client's site about a week ago. It had 138 links, it's still at 43 links. Should it be taking that long to index? Thanks! Luciana
Technical SEO | | Luciana_BAH1 -
Switching site from http to https. Should I do entire site?
Good morning, As many of you have read, Google seems to have confirmed that they will give a small boost to sites with SSL certificates this morning. So my question is, does that mean we have to switch our entire site to https? Even simple information pages and blog posts? Or will we get credit for the https boost as long as the sensitive parts of our site have it? Anybody know? Thanks in advance.
Technical SEO | | rayvensoft1 -
If I want clean up my URLs and take the "www.site.com/page.html" and make it "www.site.com/page" do I need a redirect?
If I want clean up my URLs and take the "www.site.com/page.html" and make it "www.site.com/page" do I need a redirect? If this scenario requires a 301 redirect no matter what, I might as well update the URL to be a little more keyword rich for the page while I'm at it. However, since these pages are ranking well I'd rather not lose any authority in the process and keep the URL just stripped of the ".html" (if that's possible). Thanks for you help! [edited for formatting]
Technical SEO | | Booj0 -
Another client copies everything to blogspot. Is that what keeps her site from ranking? Or what? Appears to be a penalty somewhere but can't find it.
This client has a brand new site: http://www.susannoyesandersonpoems.com Her previous site was really bad for SEO, yet at one time she actually ranked on the first page for "LDS poems." She came to me because she lost rank. I checked things out and found some shoddy SEO work by a very popular Wordpress webhoste that I will leave unnamed. If you do a backlink analysis you can see the articles and backlinks they created. But there are so few, so I'm not sure if that was it, or it just was because of the fact that her site was so poorly optimized and Google made a change, and down she fell. Here's the only page she had on the LDS poems topic in her old site: https://web.archive.org/web/20130820161529/http://susannoyesandersonpoems.com/category/lds-poetry/ Even the links in the nav were bad as they were all images. And that ranked in position 2 I think she said. Even with her new site, she continues to decline. In fact she is nowhere to be found for main keywords making me think there is a penalty. To try and build rank for categories, I'm allowing google to index the category landing pages and had her write category descriptions that included keywords. We are also listing the categories on the left and linking to those category pages. Maybe those pages are watered down by the poem excerpts?? Here's an example of a page we want to rank: http://susannoyesandersonpoems.com/category/lds-poetry/ Any help from the peanut gallery?
Technical SEO | | katandmouse0 -
Manual Webspam Error. Same Penalty on all sites on Webmaster Tools account.
My URL is: www.ebuzznet.comToday when i checked webmaster tools under manual spam section. I got Manual spam and reason was Thin content with little or no added value. Then I checked other sites from same webmaster tools account, there are 11 sites, all of them received same manual action. I never received any mail, no notification on site messages section regarding this manual action. I just need confirmation whether it is something to do with any error in webmaster tools or all of the sites really received manual spam actions.Most of the article on sites are above 500 words, quality content (no spun or copied).Looking for suggestions, answers
Technical SEO | | ndroidgalaxy0 -
Can the Hosting location of image files have a negative effect if 'off-site' such as on the devs own media server ?
Hi Can the Hosting location of image files have a negative effect if 'off-site' such as if they are on the developers own media server ? As opposed to on the actual websites server or file structure ? In the case i'm looking at the image files are hosted on a totally separate server (a media subdomain of the developers site server) from the subject sites dedicated server. Will engines still attribute the properties of files hosted in this manner to the main website (such as file name, alt attributes, etc etc) ? Or should they really be on the subject sites server own media folder ? Cheers Dan
Technical SEO | | Dan-Lawrence0 -
Young site trying hard, but banging head against the wall -- Site Review
Hi All New to PRO but we're seriously committed to getting this working. And firstly thank you to anyone who offers any useful thoughts and insights. We've launched a new site, unfortunately late to the market for the season and are really struggling to get search engine recognition. Site: http://www.ignitehats.co.uk/ We're continuously adding new content, slowly gathering more links and working hard to promote socially. But even on our clearest search terms like "Ignite hats" we're down on page 4. Both GWT and the Seomoz tools highlight no big problems (a few titles that are too long) but otherwise nothing. Maybe wrongly we requested that the Google spam team review our site incase it was being penalised, but got a template response saying the site was not in their spam system (phew, there wasn't a reason it should be we believe). We're wondering if this is just that our site is just too young? It's been live for 6 weeks. But worry maybe this is not the case. We've had success with another site we run much sooner than this. Any help or pointers would be really appreciated. Similar stories and what others have done, at least to give us some confidence to carry on would be great. Thanks for reading.
Technical SEO | | JHill0