Bing Indexation and handling of X-ROBOTS tag or AngularJS
-
Hi MozCommunity,
I have been tearing my hair out trying to figure out why BING wont index a test site we're running.
We're in the midst of upgrading one of our sites from archaic technology and infrastructure to a fully responsive version.
This new site is a fully AngularJS driven site. There's currently over 2 million pages and as we're developing the new site in the backend, we would like to test out the tech with Google and Bing.We're looking at a pre-render option to be able to create static HTML snapshots of the pages that we care about the most and will be available on the sitemap.xml.gz
However, with 3 completely static HTML control pages established, where we had a page with no robots metatag on the page, one with the robots NOINDEX metatag in the head section and one with a dynamic header (X-ROBOTS meta) on a third page with the NOINDEX directive as well. We expected the one without the meta tag to at least get indexed along with the homepage of the test site.
In addition to those 3 control pages, we had 3 pages where we had an internal search results page with the dynamic NOINDEX header. A listing page with no such header and the homepage with no such header.
With Google, the correct indexation occured with only 3 pages being indexed, being the homepage, the listing page and the control page without the metatag.
However, with BING, there's nothing. No page indexed at all. Not even the flat static HTML page without any robots directive.
I have a valid sitemap.xml file and a robots.txt directive open to all engines across all pages yet, nothing.
I used the fetch as Bingbot tool, the SEO analyzer Tool and the Preview Page Tool within Bing Webmaster Tools, and they all show a preview of the requested pages. Including the ones with the dynamic header asking it not to index those pages.
I'm stumped. I don't know what to do next to understand if BING can accurately process dynamic headers or AngularJS content.
Upon checking BWT, there's definitely been crawl activity since it marked against the XML sitemap as successful and put a 4 next to the number of crawled pages.
Still no result when running a site: command though. Google responded perfectly and understood exactly which pages to index and crawl.
Anyone else used dynamic headers or AngularJS that might be able to chime in perhaps with running similar tests?
Thanks in advance for your assistance....
-
Thank you for the update Kavit.
-
Hi Everett and Fellow Mozzers,
I have been away overseas so wasn't able to put up an update.
Eventually, managed to get a hold of someone at BING within the tech team who told me that the reason that they didn't index the pages was simply because of popularity.
It isn't enough to have unique content, design and structure on your site, it is also vital to have traffic, links and mentions as external signals.
We also got word that dynamic sites and pre-render content will be acceptable for BING so we're resting easier at night these days.
Development on the site continues as per schedule and we will be launching the proper site this year on a highly authoritative domain which should yield very different results to the test we put together.
Hopefully, this will help someone else who is on a similar pathway.
Everett, I would like to thank you again for taking the time to read, reply and help us with our analysis.
Thanks!
-
Hi Everett,
Thank you for the analysis and deeper insights.
I did make the changes to the test pages bar the design template.
We added the unique titles, meta descriptions and meta keywords.
We added completely unique content to all three pages with no other instances of this content appearing on the web at all.
The pages are now also interlinked and also linked from the top of the homepage so none of them are orphan pages.
sitemaps have been updated and resubmitted.
The latest version has been out a week so far, but no response from BING as yet.
Thanks,
Kavit.
-
Hello Kavit,
I would suggest putting unique Title tags, meta descriptions and content on those pages. They are very thin as it is, and all of the content is boilerplate.
There are 57,100,000 results on Bing for: "Search for an Australian Business, Government Department or Person" which is the content on the home page you shared.
There are 60,600 results on Bing for: ""There was a table set out under a tree in front of the house, and the March Hare and the Hatter were having tea at it" which is the content on this page: http://wp-seospike-weblbl.naws-sensis.com.au/bing-seo-control/no-metatag.html .
And so on. I can see why Bing wouldn't want to add yet another thin, duplicate, orphan page to their index. My advice would be to build out those test pages with a design template and to put original content, title tags and meta descriptions on all of them. Then repeat your test.
-
Hi Everett,
Thank you for taking the time out to read and respond.
The URL we have setup for testing is: wp-seospike-weblbl.naws-sensis.com.au
We have 3 control pages (all flat HTML pages) that we setup and put online for bing to crawl:
http://wp-seospike-weblbl.naws-sensis.com.au/bing-seo-control/no-metatag.html - no robots metatag and allowed to crawl and index.
http://wp-seospike-weblbl.naws-sensis.com.au/bing-seo-control/metatag.html - page with a noindex metatag not to be crawled and indexed
http://wp-seospike-weblbl.naws-sensis.com.au/bing-seo-control/metatag-header.html - X-Robots meta tag NOINDEX
http://wp-seospike-weblbl.naws-sensis.com.au - homepage with no robots exclusion
Ideally, I expected the homepage and the no-metatag page to be indexed at least.
I am familiar with the builtvisible documentation that they've put out.
My main pain point is that even the flat HTML pages are getting ignored, so I can't even test the deeper AngularJS developed pages since my control group is not delivering results as it should.
a site command on the above domain on bing shows no results.
Thanks again!
-
Is there any chance of getting a URL for the domain in question?
Have you read this yet?
https://builtvisible.com/javascript-framework-seo/What are the URLs like that you're asking Bing to index? Which is closest?
Hashbang
http://www.IWishJSFramworkWebsitesWouldGoAway/#!Escaped Fragment
http://www.IWishJSFramworkWebsitesWouldGoAway/?escaped_fragment=Base URL using Angular's $location service to construct URLs without the #! via the HTML5 History API http://www.IWishJSFramworkWebsitesWouldGoAway/
I know this doesn't answer your question, but hopefully it will get the discussion started.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
H1 tag within on top nav but css class styled appear at bottom
Hi, Sorry the similar question could have been ask previously but I couldn't get an exact answer. Someone put a h1 tag on our homepage (which do not have before) and the is within the top nav. But it has a css style class in that h1 tag to make it appear at the bottom of the page so not everyone would be aware of that small h1 title. I personally do not think that's going to help with seo. However I want to know if this practice is going to be: beneficial, or not at all? or harmful instead? Thanks LM
Web Design | | LauraHT0 -
Is The HREF Link "Title" Tag Needed on Mobile Websites?
Hello To Those Who Are Wiser Than I, I am wondering if the href link "title" tag is needed, or serves any purpose, on mobile websites? Also, does it effect SEO in any way? I ask because generally the href link title tag provides more information to the user when they scroll their mouse over the link - but this action does not happen on mobile! Users have no mouse and thus no extra information would be displayed. I'm really wondering if it still matters for SEO purposes on mobile though. -The UnEnlightened
Web Design | | Stew2220 -
App indexing with several subdomains
Hi, We are currently in the process of getting our app developed. From an SEO point of view I want the app to get indexed and am currently trying to get a step by step guide for it. I've come across this helpful guide: https://moz.com/blog/how-to-get-your-app-content-indexed-by-google but need to go a bit deeper now. Questions I have is related to the "apple-app-site-association" file As far as I could find out the JSON code of that file has the following syntax: {_ “applinks”: {__ “apps”: [],__ “details”: [__ {__ “appID”: “9JA89QQLNQ.com.my.bundle.id“,__ “paths”: [ “/myPath1”, “/myPath2/*” ]__ },__ ]__ }_ } My question is related to the key "appID" I'm wondering how the entry would look like if I have 10 country subdomains and would like to make sure that the deep link clicked in the browser is going to the right subdomain page within the app. Current website structure:
Web Design | | FashionLux
uk.example.com/english-path/
de.example.com/german-path/ Many Thanks in advance0 -
Rel=next and rel=prev meta tags
Hi,
Web Design | | RocoClothing
We have recently implemented the rel=next and rel=prev meta tags on the
category pages of our website. Currently for example if on page 2 we have the following: href="http://www.rococlothing.co.uk/boys/boys-suits/" />
href="http://www.rococlothing.co.uk/boys/boys-suits/?p=1" />
href="http://www.rococlothing.co.uk/boys/boys-suits/?p=3" /> For each page we are using the same canonical tag which is the url for
the 1st page in the category.
Is this the correct way to impletment it or should the canonical tag for
page 2 be as follows: href="http://www.rococlothing.co.uk/boys/boys-suits/?p=2" /> I have also seen some companies ommiting the canonical tag on pages
after page 1 and just using the prev and nexts. Could anyone advise as to what the ideal implementation would be for this? Regards
Patrick0 -
Is it important to keep your website home index page simple to rank better?
My website http://www.endeavourcottage.co.uk/ markets holiday cottages and it's grown from my own singular cottage into a small letting agency and I used to rank at best number 3 for the short tailed keywords like Whitby holiday cottages with its drop-down to position 10 on Google.co.uk. So this week I was looking for a UK business to help me improve my rankings and the first thing they said was my home page is detrimental with the listing too many conflicting info with it advertising all 12 properties on it. They suggested a door entry page into the site keeping it simple but when I run it through the analysing tool here on Moz for "Whitby holiday cottages" as an example it came out looking okay. I do the usual things of title tags and meta descriptions for my keywords etc any suggestions or advice would be very welcome thank you Alan
Web Design | | WhitbyHolidayCottages0 -
Google tag manager on blocked beta site - will it phone home to Google and cause site to get indexed?
We want to develop a beta site, in a directory with the robots.txt blocking bots. We want to include the Google Tag Manager tags and event layer tracking code on this beta site. My question is that by including the Google Tag Manager code, that phones home to Google, will it cause Google to index this beta site when we don't want it indexed?
Web Design | | CFSSEO0 -
Hi, I have a doubt. If we want to hide unwanted text in a web page its possible with "" tag. And my question "does a search engine crawl those text? help me.
I want to hide a lot of text behind my site page. I know its possible with that tag. But in what way a search engine looks at those text? Hidden or they are crawled and indexed.
Web Design | | FhyzicsBCPL0 -
Does **tag on a product description help?**
Hi, Does using the tag on a line of text in the products description help with SEO for that keyword phrase? **See here: http://www.designerboutique-online.com/tops/passarella-death-squad/passarella-death-squad-t-shirt-white/0/ I have bolded the Passarella Death Squad T-Shirt line. Would this help in any way? Cheers Will**
Web Design | | YNWA0