Why can no tool crawl this site?
-
I am trying to perform a crawl analysis on a client's website at https://www.bravosolution.com
I have tried to crawl it with IIS for SEO, Sreaming Frog and Xenu and not one of them makes it further than the home page of the site. There is nothing I can see in the robots.txt that is blocking these agents.
As far as I can see, Google is able to crawl the site although they have noticed a significant drop in organic traffic.
Any advise would be very welcome
Regards
Danny
-
I would look into finding a method to redirect via your server rather than with javascript. This will ensure that bots can properly crawl your site.
I would also add hreflang tags which should help Google with the multiple language versions of the site.
Also in the short term you may want to do something like add a link or a delayed meta refresh just in case someone either has javascript disabled or is using script blocking extensions. This will make sure they at least see something instead of a blank page.
-
Really helpful and much appreciated - many thanks!
Danny
-
Yes that's what I said CleverPhD, I just couldn't type that fast today.
Only joking Thanks for expanding on the subject.
-
To expand on Dean's point.
If you look at the source code on https://www.bravosolution.com/ you get a bunch of JavaScript (shown below). It is basically looking at the users location and the sending them to the appropriate version of your website based on country. This is why here in the US we are sent to https://www.bravosolution.com/cms/us
Many spiders/tools (and Googlebot was not really good at this until recently) are not good at (or do not do any) crawling and executing on JavaScript so they get stuck when they hit your home page.
If you want to evaluate any of your localized sites, just run those URLs through various tools like screaming frog etc. You would then ask, "Well, how do I know that my main https://www.bravosolution.com is working properly for SEO?". I don't have as much background in how to optimize for international SEO, but you can do a several things to start with.
-
Google anything having to do with Aleyda Solis and International SEO. She posts a lot of stuff here at Moz and is pretty sharp on this stuff. There may be a more appropriate way to redirect international clients from your main page that how you are executing.
-
Run your home page through Google Webmaster Tools under Crawl > Fetch as Google. See what the page looks like
-
Double check your robots.txt to make sure you are not blocking any folders that would contain a JavaScript library. Based on the code below, I do not see you referencing any external libraries, but if you are dependent on JS to send Google, it would be worth having your developer check things
-
As with everything on what to do, it all depends. If all of your local country sites are independently ranked and successful, this main website may nor may not be doing you any favors currently if it is just a pass through with no domain authority to start with. Spend time on step #1 to see if there is anything else worth doing.
Cheers!
name="description" />
-
-
Yes, it should redirect you to the correct country version based on your IP. But I still can't crawl the site from the home page
-
Cheers Bryan - much appreciated. It's driving me crazy!
-
Hi Danny,
Have you looked at the site via http://web-sniffer.net/
It would appear that the home page is just a JavaScript redirect.
I was redirected to https://www.bravosolution.com/cms/us which then could seen via Sreaming Frog.
The reason for my (default) redirect is given by web-sniffer as:
DEFAULT CORPORATE if ( path == '' ) { path = '/cms/us
-
Interesting. I verified the robots file and tried running through screaming frog... nothing. I' will dig into this with my dev team to try and get you an answer asap.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Messy older site
I am taking over a website that doesn't have any canonical tags and spotty redirects. It looks like they have http://, https://, www and non-www pages indexed but GA is just set up for the http://non-www home page. Should all versions of the site be set up in GA and Search Console? I think so but wanted to confirm. Thanks in advance.
Technical SEO | | SpodekandCo0 -
Can you keep you old HTTP xml sitemape when moving to HTTPS site wide?
Hi Mozers, I want to keep the HTTP xml sitemape live on my http site to keep track of indexation during the HTTPS migration. I'm not sure if this is doable since once our tech. team forces the redirects every http page will become https. Any ideas? Thanks
Technical SEO | | znotes0 -
Moving site from html to Wordpress site: Should I port all old pages and redirect?
Any help would be appreciated. I am porting an old legacy .html site, which has about 500,000 visitors/month and over 10,000 pages to a new custom Wordpress site with a responsive design (long overdue, of course) that has been written and only needs a few finishing touches, and which includes many database features to generate new pages that did not previously exist. My questions are: Should I bother to port over older pages that are "thin" and have no incoming links, such that reworking them would take time away from the need to port quickly? I will be restructuring the legacy URLs to be lean and clean, so 301 redirects will be necessary. I know that there will be link juice loss, but how long does it usually take for the redirects to "take hold?" I will be moving to https at the same time to avoid yet another porting issue. Many thanks for any advice and opinions as I embark on this massive data entry project.
Technical SEO | | gheh20130 -
Then why my site is not ranking
My website's DA and PAs are good compare with my competitors. Then why my site is not ranking.
Technical SEO | | Somanathan0 -
About google Disavow tool
My website is attacked by spammed link method, so should i use Goolge disavow tool to remove that links? And i have an question that when i use google Disavow to remove backlinks, but i still not remove it on the webpage that placed my links. Does Google index that backlink again? or never?
Technical SEO | | magician0 -
On-site adjustment opinions
Hi folks, I've got a fairly interesting scenario. I'm trying to rank this page (http://www.staysa.co.za/sa/1-2-0-0-1/East-London/accommodation) better for the term, "accommodation east london". The client isn't keen on making many changes and it was built horribly with ASP, half CMS, half not. I have made the following changes today: I introduced two paragraphs of text below the H1 tag. I changed "East London Bed and Breakfast", "East London Conference Venues", "East London Cottage / Chalet" to just "Bed and Breakfast", "Conference Venues", "Cottage / Chalet" as the continual key phrase duplication in my experience is a bad move. I've made a change to the title tag (this is a huge mission as it's not CMS controlled, so I had to teach myself some basic ASP to do so). Meta data.. nightmare to change unfortunately, at least not without rewriting part of the CMS. I'm wondering, are there any other on-site factors that I'm missing? I'm not a fan of site-wide links, so I don't want to put an exact match anchor text link from the sidebar/footer to the page, not unless someone can motivate why I should. Keen to hear everyone's opinions 🙂
Technical SEO | | ChristopherM0 -
Have my site been penalised ?
Hi, I recently hired a link builder (who works for a digital marketing agency and wanted to earn some extra cash ) to do some link building for me on keyword Carpet Cleaner Hire Basically , May my ranking for this work has moved from 29 up to 13 , then down to 500 + and now up to 67 ?. It's been all over the place and I am worried that maybe google thinks its unnatural or to many links etc. I looked at some of the work this chap did and there was a log of bookmarks done , apparently to increase social awareness and some articles on random forums I never heard of but the articles didn't look to good and in my eyes were a little spammy.. I don't have any messages in WMT saying I have been penalised or anything but I am naturally worried that this chap may have caused damaged and that I was manually penalised. I have since asked him to stop immediately.. does this sound normal for the ranking to be all over the place in this manner when link building ? How can I tell if I have been manually penalized in any way ? What;s the general consensus ?. Should I contact google or leave it and see what happens ? Other keywords seems to be okay though. Any advice much appreciated. thanks Sarah.
Technical SEO | | SarahCollins1 -
Does adding Tool Tips to a site hurt it's SEO?
I'm wanting to add tool tips to my site as it's intended for non-technical people that are wanting high tech equipment and services. I thought that by adding tool tips, I could clear any confusion they may have about a particular word right there rather then them having to search for what it means. I did some research online and saw that it may hurt SEO ratings but wanted to verify here first before deciding.
Technical SEO | | sDevik0