Moz Q&A is closed.
After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.
Bing Indexation and handling of X-ROBOTS tag or AngularJS
-
Hi MozCommunity,
I have been tearing my hair out trying to figure out why BING wont index a test site we're running.
We're in the midst of upgrading one of our sites from archaic technology and infrastructure to a fully responsive version.
This new site is a fully AngularJS driven site. There's currently over 2 million pages and as we're developing the new site in the backend, we would like to test out the tech with Google and Bing.We're looking at a pre-render option to be able to create static HTML snapshots of the pages that we care about the most and will be available on the sitemap.xml.gz
However, with 3 completely static HTML control pages established, where we had a page with no robots metatag on the page, one with the robots NOINDEX metatag in the head section and one with a dynamic header (X-ROBOTS meta) on a third page with the NOINDEX directive as well. We expected the one without the meta tag to at least get indexed along with the homepage of the test site.
In addition to those 3 control pages, we had 3 pages where we had an internal search results page with the dynamic NOINDEX header. A listing page with no such header and the homepage with no such header.
With Google, the correct indexation occured with only 3 pages being indexed, being the homepage, the listing page and the control page without the metatag.
However, with BING, there's nothing. No page indexed at all. Not even the flat static HTML page without any robots directive.
I have a valid sitemap.xml file and a robots.txt directive open to all engines across all pages yet, nothing.
I used the fetch as Bingbot tool, the SEO analyzer Tool and the Preview Page Tool within Bing Webmaster Tools, and they all show a preview of the requested pages. Including the ones with the dynamic header asking it not to index those pages.
I'm stumped. I don't know what to do next to understand if BING can accurately process dynamic headers or AngularJS content.
Upon checking BWT, there's definitely been crawl activity since it marked against the XML sitemap as successful and put a 4 next to the number of crawled pages.
Still no result when running a site: command though. Google responded perfectly and understood exactly which pages to index and crawl.
Anyone else used dynamic headers or AngularJS that might be able to chime in perhaps with running similar tests?
Thanks in advance for your assistance....
-
Thank you for the update Kavit.
-
Hi Everett and Fellow Mozzers,
I have been away overseas so wasn't able to put up an update.
Eventually, managed to get a hold of someone at BING within the tech team who told me that the reason that they didn't index the pages was simply because of popularity.
It isn't enough to have unique content, design and structure on your site, it is also vital to have traffic, links and mentions as external signals.
We also got word that dynamic sites and pre-render content will be acceptable for BING so we're resting easier at night these days.
Development on the site continues as per schedule and we will be launching the proper site this year on a highly authoritative domain which should yield very different results to the test we put together.
Hopefully, this will help someone else who is on a similar pathway.
Everett, I would like to thank you again for taking the time to read, reply and help us with our analysis.
Thanks!
-
Hi Everett,
Thank you for the analysis and deeper insights.
I did make the changes to the test pages bar the design template.
We added the unique titles, meta descriptions and meta keywords.
We added completely unique content to all three pages with no other instances of this content appearing on the web at all.
The pages are now also interlinked and also linked from the top of the homepage so none of them are orphan pages.
sitemaps have been updated and resubmitted.
The latest version has been out a week so far, but no response from BING as yet.
Thanks,
Kavit.
-
Hello Kavit,
I would suggest putting unique Title tags, meta descriptions and content on those pages. They are very thin as it is, and all of the content is boilerplate.
There are 57,100,000 results on Bing for: "Search for an Australian Business, Government Department or Person" which is the content on the home page you shared.
There are 60,600 results on Bing for: ""There was a table set out under a tree in front of the house, and the March Hare and the Hatter were having tea at it" which is the content on this page: http://wp-seospike-weblbl.naws-sensis.com.au/bing-seo-control/no-metatag.html .
And so on. I can see why Bing wouldn't want to add yet another thin, duplicate, orphan page to their index. My advice would be to build out those test pages with a design template and to put original content, title tags and meta descriptions on all of them. Then repeat your test.
-
Hi Everett,
Thank you for taking the time out to read and respond.
The URL we have setup for testing is: wp-seospike-weblbl.naws-sensis.com.au
We have 3 control pages (all flat HTML pages) that we setup and put online for bing to crawl:
http://wp-seospike-weblbl.naws-sensis.com.au/bing-seo-control/no-metatag.html - no robots metatag and allowed to crawl and index.
http://wp-seospike-weblbl.naws-sensis.com.au/bing-seo-control/metatag.html - page with a noindex metatag not to be crawled and indexed
http://wp-seospike-weblbl.naws-sensis.com.au/bing-seo-control/metatag-header.html - X-Robots meta tag NOINDEX
http://wp-seospike-weblbl.naws-sensis.com.au - homepage with no robots exclusion
Ideally, I expected the homepage and the no-metatag page to be indexed at least.
I am familiar with the builtvisible documentation that they've put out.
My main pain point is that even the flat HTML pages are getting ignored, so I can't even test the deeper AngularJS developed pages since my control group is not delivering results as it should.
a site command on the above domain on bing shows no results.
Thanks again!
-
Is there any chance of getting a URL for the domain in question?
Have you read this yet?
https://builtvisible.com/javascript-framework-seo/What are the URLs like that you're asking Bing to index? Which is closest?
Hashbang
http://www.IWishJSFramworkWebsitesWouldGoAway/#!Escaped Fragment
http://www.IWishJSFramworkWebsitesWouldGoAway/?escaped_fragment=Base URL using Angular's $location service to construct URLs without the #! via the HTML5 History API http://www.IWishJSFramworkWebsitesWouldGoAway/
I know this doesn't answer your question, but hopefully it will get the discussion started.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How to prevent development website subdomain from being indexed?
Hello awesome MOZ Community! Our development team uses a sub-domain "dev.example.com" for our SEO clients' websites. This allows changes to be made to the dev site (U/X changes, forms testing, etc.) for client approval and testing. An embarrassing discovery was made. Naturally, when you run a "site:example.com" the "dev.example.com" is being indexed. We don't want our clients websites to get penalized or lose killer SERPs because of duplicate content. The solution that is being implemented is to edit the robots.txt file and block the dev site from being indexed by search engines. My questions is, does anyone in the MOZ Community disagree with this solution? Can you recommend another solution? Would you advise against using the sub-domain "dev." for live and ongoing development websites? Thanks!
Web Design | | SproutDigital0 -
What is the best way to handle annual events on a website?
Every year our company has a user conference with between 300 - 400 attendees. I've just begun giving the event more of a presence on our website. I'm wondering, what is the best way to handle highlights from previous years? Would it be to create an archive (e.g. www.companyname.com/eventname/2015) while constantly updating the main landing page to promote the current event? We also use an event website (cvent) to handle our registrations. So once we have an agenda for the current years event I do a temporary redirect from the main landing page to the registration website. I don't really like this practice and I feel like it might be better to keep all of the info on the main domain. Wondering if anybody has any opinions or feedback on that process as well. Just looking for best practices or what others have done and have had success with.
Web Design | | Brando161 -
Spanish website indexed in English, redirect to spanish or english version if i do a new website design?
Hi MOZ users, i have this problem. We have a website in Spanish Language but Google crawls it on English (it is not important the reasons). We re made the entire website and now we are planning the move. The new website will have different language versions, english, spanish and portuguese. Somebody tells me that we have to redirect the old urls (crawled on english) to the new english versions, not to the spanish (the real language of the firsts). Example: URL1 Language: Spanish - Crawled on English --> redirect to Language English version. the other option will be redirect to the spanish new version, which the visitor is waiting to find. URL1 Language: Spanish - Crawled on English --> redirect to Language Spanish version. What do you think? Which is the better option?
Web Design | | NachoRetta0 -
Spaces at beginning of title tag - negatively affect the optimization of the page?
For some reason, our title tags have a long space after the beginning title tag and before the text appears. The beginning title tag is on one line, then a break, a tab and then the content of the title tag. I'm pretty sure this is not good and is affecting optimization of the page. Am I correct or is this not an issue and does not need to be fixed? Example: | <title></span></p> <p> </p> </td> </tr> <tr> <td class="line-number"> </td> <td class="line-content"> First keyword</td> </tr> </tbody> </table></title> |
Web Design | | CFSSEO0 -
Lots of Listing Pages with Thin Content on Real Estate Web Site-Best to Set them to No-Index?
Greetings Moz Community: As a commercial real estate broker in Manhattan I run a web site with over 600 pages. Basically the pages are organized in the following categories: 1. Neighborhoods (Example:http://www.nyc-officespace-leader.com/neighborhoods/midtown-manhattan) 25 PAGES Low bounce rate 2. Types of Space (Example:http://www.nyc-officespace-leader.com/commercial-space/loft-space)
Web Design | | Kingalan1
15 PAGES Low bounce rate. 3. Blog (Example:http://www.nyc-officespace-leader.com/blog/how-long-does-leasing-process-take
30 PAGES Medium/high bounce rate 4. Services (Example:http://www.nyc-officespace-leader.com/brokerage-services/relocate-to-new-office-space) High bounce rate
3 PAGES 5. About Us (Example:http://www.nyc-officespace-leader.com/about-us/what-we-do
4 PAGES High bounce rate 6. Listings (Example:http://www.nyc-officespace-leader.com/listings/305-fifth-avenue-office-suite-1340sf)
300 PAGES High bounce rate (65%), thin content 7. Buildings (Example:http://www.nyc-officespace-leader.com/928-broadway
300 PAGES Very high bounce rate (exceeding 75%) Most of the listing pages do not have more than 100 words. My SEO firm is advising me to set them "No-Index, Follow". They believe the thin content could be hurting me. Is this an acceptable strategy? I am concerned that when Google detects 300 pages set to "No-Follow" they could interpret this as the site seeking to hide something and penalize us. Also, the building pages have a low click thru rate. Would it make sense to set them to "No-Follow" as well? Basically, would it increase authority in Google's eyes if we set pages that have thin content and/or low click thru rates to "No-Follow"? Any harm in doing this for about half the pages on the site? I might add that while I don't suffer from any manual penalty volume has gone down substantially in the last month. We upgraded the site in early June and somehow 175 pages were submitted to Google that should not have been indexed. A removal request has been made for those pages. Prior to that we were hit by Panda in April 2012 with search volume dropping from about 7,000 per month to 3,000 per month. Volume had increased back to 4,500 by April this year only to start tanking again. It was down to 3,600 in June. About 30 toxic links were removed in late April and a disavow file was submitted with Google in late April for removal of links from 80 toxic domains. Thanks in advance for your responses!! Alan0 -
Does it do harm if you add a rel="canonical" tag on a page that doesn't need it?
If a page is clearly unique and there is obviously no canonical tag needed, does it hurt anything if one has been added?
Web Design | | jaychow0 -
Duplicate H1 tag IF it holds SAME text?
Hello people, I know that majority of SEO gurus (?) claim that H1 tag should only be used once per page. In the landing page design I'm working with, we actually need to repeat our core message stated in H1 & H2 - at the bottom of the page. Now the question is: Can that in any way cause any ranking penalty from big G? In my eyes that is not attempt to over optimize page as it contains SAME info as the H1 & H2 at the top of the page. Confusing, so I'm hope that some SEO gurus here will share some light on this. Thanks in advance!
Web Design | | RetroOnline0 -
Indexing Dynamic Pages
Hi, I am having an issues among others, regarding indexing dynamic pages. Our website, www.me-by-melia, was just put live and I am concerned the bottom naviagtion pages (http://www.me-by-melia.com/#store, http://www.me-by-melia.com/#facebook, etc) will not be indexed and create duplicate pages. Also, when you open these pages in a new tab, it takes you to homepage. The website was created in HTML5. Please advise.
Web Design | | Melia0