Welcome to the Q&A Forum

Hi Everyone

I wanted to see how people submit their urls to Google and ensure they are all being indexed.  I currently have an ecommerce site with 18,000 products.  I have sitemaps setup, but noticed that the various product pages haven't started ranking yet.  If I submit the individual url through the new Google Search Console I see the page ranking in a matter of minutes.

Before the new Google Search Console you could just ask Google to Fetch/Render an XML sitemap and ask it to crawl all the links.  I don't see the same functionality working today on Google Search Console and was wondering if there are any new techniques people could share.

Thanks,
Anthony

Hi Ethan

One thing I have heard of people trying is a plugin that serves dynamic robots.txt files.  I don't use add-on sites so you will probably have to test the behavior.  He is an example of one of the plugins.

https://wordpress.org/plugins/wp-robots-txt/

hope this helps,
Anthony

Hi Ethan

Sorry, I wasn't clear.  I was thinking you could drop the use of the robots.txt all together and just use the Meta Tag approach since it seems that the robots.txt is having a global impact to your sites.  Search engines will still crawl the pages, but it should exclude them from the index.

Hope this helps,
Anthony

Hi Ethan

One approach may be to try using the Robots Meta Tag.  You can use noindex to tell Google not to index.  This won't prevent crawling, but Google should respect the request to not index your site.  I have included a good guide below to get you started.

https://developers.google.com/webmasters/control-crawl-index/docs/robots_meta_tag

Hope this helps,
Anthony B
Biondo Creative
biondocreative.com

Hi Chris

I think two separate sites with unique domains is not a bad thing.  Use the .com for corporate content and campaigns and the UK as you mentioned.  I would just suggest you try to make the content unique on both sites as much as possible. I would also suggest separating your Google Plus and other social properties like Facebook and Twitter, because you will want to tailor your content and social posts to your audience.

Hope this helps,
Anthony B
Biondo Creative
biondocreative.com

Hi Nikola

My thought would be no. I did a quick search for Prismatic and I don't see web pages out there that are indexed. Its great to hear that your pages are being picked up by an app, because it will just get your site in front of more eyes, but it may not have any positive link benefits at this time. I have seen Google promote app indexing and deep linking on their site, but it looks like the app developer needs to do something to take advantage of it. https://developers.google.com/app-indexing/

Hope this helps,
Anthony B
Biondo Creative
biondocreative.com

Hi John

If you are using Google Tag Manager, you should let it fire the events and conversions instead of putting code in-line.  If you have the old Google Analytics tags you will want to comment them out and remove them when you are comfortable with the transition.

Hope this helps,
Anthony B
Biondo Creative
biondocreative.com

Hi Carlos

It sounds funny to me.  Use the content and post regular updates to your site (daily, weekly) and start building an email list where you can get genuine traffic and inbound links coming to your site.  Use social sites to promote your unique content and drive people to your site because you have compelling content.  Like Egol said, if you invest in building up these other sites and Google penalizes them you will have lost your investment.

Hope this helps,
Anthony B
Biondo Creative
biondocreative.com

Hi Josh

This may be a more subjective question, but my vote is for responsive.  Even though you don't have a high volume of traffic today that is mobile, it will most likely increase and when you have a mobile-friendly page it will most likely accelerate. Some companies still go with stripped down mobile sites, but I have been seeing more people moving towards responsive to minimize the need of managing two separate sites and being able to give their mobile users as much content and functionality as possible.

Also on April 21st Google is going to start giving more preference to mobile sites.  You can read more about it in the Moz article - http://moz.com/blog/9-things-about-googles-mobile-friendly-update

Hope this helps,
Anthony B
Biondo Creative
biondocreative.com

Given the technical limitations you have with Webs.com I would suggest recreating the page on your site so it can be accessed and adding a canonical tag if they allow you to add items to the meta-header. See the syntax below. Point the href to the new page you want to rank.

Then on the old page do a redirect after 10 seconds to the correct page. That way it doesn't look like you are trying to do cloaking or something shady.

hope this helps,

Anthony
Biondo Creative LLC.
http://www.biondocreative.com

Hi Josh

This may be a more subjective question, but my vote is for responsive.  Even though you don't have a high volume of traffic today that is mobile, it will most likely increase and when you have a mobile-friendly page it will most likely accelerate. Some companies still go with stripped down mobile sites, but I have been seeing more people moving towards responsive to minimize the need of managing two separate sites and being able to give their mobile users as much content and functionality as possible.

Also on April 21st Google is going to start giving more preference to mobile sites.  You can read more about it in the Moz article - http://moz.com/blog/9-things-about-googles-mobile-friendly-update

Hope this helps,
Anthony B
Biondo Creative
biondocreative.com

Hi John

If you are using Google Tag Manager, you should let it fire the events and conversions instead of putting code in-line.  If you have the old Google Analytics tags you will want to comment them out and remove them when you are comfortable with the transition.

Hope this helps,
Anthony B
Biondo Creative
biondocreative.com

Hi Everyone

I wanted to see how people submit their urls to Google and ensure they are all being indexed.  I currently have an ecommerce site with 18,000 products.  I have sitemaps setup, but noticed that the various product pages haven't started ranking yet.  If I submit the individual url through the new Google Search Console I see the page ranking in a matter of minutes.

Before the new Google Search Console you could just ask Google to Fetch/Render an XML sitemap and ask it to crawl all the links.  I don't see the same functionality working today on Google Search Console and was wondering if there are any new techniques people could share.

Thanks,
Anthony

I don't think where you setup the page has anything to do with the ranking.  I run pages for various travel sites and I haven't seen issues.  I think the thing that matters is that the geo information you add for the page or profile lines up with where your client is located and the information is consistent across sites like Yelp, Google+, Facebook, etc.

Moz Local gives you some good tips for where to register your site if you want to do it on your own.  
https://moz.com/local/search

hope this helps,

Anthony

Hi Nikola

My thought would be no. I did a quick search for Prismatic and I don't see web pages out there that are indexed. Its great to hear that your pages are being picked up by an app, because it will just get your site in front of more eyes, but it may not have any positive link benefits at this time. I have seen Google promote app indexing and deep linking on their site, but it looks like the app developer needs to do something to take advantage of it. https://developers.google.com/app-indexing/

Hope this helps,
Anthony B
Biondo Creative
biondocreative.com

Hi Ethan

One approach may be to try using the Robots Meta Tag.  You can use noindex to tell Google not to index.  This won't prevent crawling, but Google should respect the request to not index your site.  I have included a good guide below to get you started.

https://developers.google.com/webmasters/control-crawl-index/docs/robots_meta_tag

Hope this helps,
Anthony B
Biondo Creative
biondocreative.com