Welcome to the Q&A Forum

Whenever the rogerbot comes back to my site for a crawl it seems to want to crawl urls that dont exist and thus causes errors to be reported...

Example:- The correct url is as follows:

/vw-baywindow/cab_door_slide_door_tailgate_engine_lid_parts/cab_door_seals/genuine_vw_brazil_cab_door_rubber_68-79_10330/

But it seems to want to crawl the following:

/vw-baywindow/cab_door_slide_door_tailgate_engine_lid_parts/cab_door_seals/genuine_vw_brazil_cab_door_rubber_68-79_10330/?id=10330

This format doesn't exist anywhere and never has so I have no idea where its getting this url format from

The user agent details I get are as follows:

IP ADDRESS: 107.22.107.114
USER AGENT: rogerbot/1.0 (http://moz.com/help/pro/what-is-rogerbot-, rogerbot-crawler+pr1-crawler-17@moz.com)

thats a great help cheers

wheres the best place to do an inbound link check?

cheers, i thought as much

We have a test version of a clients web site on a separate server before it goes onto the live server.

Some code from the test site has some how managed to get Google to index the test site which isn't great!

Would simply adding a robots text file to the root of test simply blocking all be good enough or will i have to put the meta tags for no index and no follow etc on all pages on the test site also?

Recently I was asked the question by a client who has some SEO knowledge.....

We built their site and during testing we simply set the home page links to be:

home

rather than:

home

This was mainly to help while in test and dev as they were on different servers to the live site and if we had put the full domain in it kept taking us from dev/test to live.

Is it bad to have just a slash for SEO purposes or is it REALLY that important on home links to have the full domain as a slash takes you back to the full domain?

Thanks

Site is still in testing but as an example the products are for cars and the same product can fit multiple cars, hence why it may be in more than one place

Another option i thought of would be that the other pages I simply add a no index meta tag to so it only indexes the one url.

The reason I say this as from experience of how seomoz crawls sites and some pages which i know are not duplicate content yet i have had many pages for many sites get tagged for duplicate content simply because there isn't enough of a difference on the page.

The product pages wouldb e very much the same as really all that would be different is the breadcrumb trail and possibly some other links 95% of the pages would be the same as they are the same product.

We have an eCommerce site we have built for a customer and the products are allowed to appear in more than one product category within the web site.

Now I know this is a bad idea from a duplicate content point of view, But we are going to allow the customer to select which out of the multiple categories the product appears in will be the default category.

This will mean we will have a way of defining what the default url is for a product.

So am I correct in thinking all the other urls where the product appears we should add a rel canonical to these pages pointing to the default url to stop duplicate content?

Is this the best way?

Many of our e-com sites we build for customers have drop down menus to help the user easily find products without having to click -

Example: http://www.customandcommercial.com/

But this then causes the report to trigger too many on page links

We do have a site map and a google site map

So should I put code in place not to follow the drop down menu link items or leave in place?

cheers

Another option i thought of would be that the other pages I simply add a no index meta tag to so it only indexes the one url.

The reason I say this as from experience of how seomoz crawls sites and some pages which i know are not duplicate content yet i have had many pages for many sites get tagged for duplicate content simply because there isn't enough of a difference on the page.

The product pages wouldb e very much the same as really all that would be different is the breadcrumb trail and possibly some other links 95% of the pages would be the same as they are the same product.