Hi Bob,

You can "suggest" a crawl rate to Google by logging into your webmasters tools on Google and adjusting it there.

As for indexing pages.. I looked at your robots and site. It really looks like you need to employ some No Follow on some of your internal linking, specifically on the product page filters, that alone could reduce the total number of URLS that the crawlers even attempts to look at.

Additionally your sitemap http://premium-hookahs.nl/sitemap.xml shows a change frequency of daily, and probably should be broken out between Pages / Images so you end up using two sitemaps one for images and one for pages. You may also want to review what is in there. Using ScreamingFrog (free) the sitemap I made (link) only shows about 100 urls.

Hope it helps,

Don

Hi,

Something is generating those links, and that is what I would want to fix. It sounds like a module or add-on is incorrectly coded, or installed improperly. As p=1_1 is usually a form of pagination and / or sort order.

You could of course easily fix it by putting a canonical tag on the root page to itself or work out a mod rewrite rule, but that isn't actually fixing the problem. As soon as you add another category you may have to revisit the fix again.

Here is an example of using mod re-write on Stack Overflow: http://stackoverflow.com/questions/14666140/htaccess-rewriting-php-pagination

I hope that helps,

Don

Hi Bob,

About the nofollow vs blocked. In the end I suppose you have the same results, but in practice it works a little differently. When you nofollow a link it tells the crawler as soon as it encounters the link not to request or follow that link path. When you block it via robots the crawler still attempts to access the url only to find it not accessible.

Imagine if I said go to the parking lot and collect all the loose change in all the unlocked cars. Now imagine how much easier that task would be if all the locked cars had a sign in the window that said "Locked", you could easily ignore the locked cars and go directly to the unlocked ones. Without the sign you would have to physically go check each car to see if it will open.

About link juice, if you have a link, juice will be passed regardless of the type of link. (You used to be able to use nofollow to preserve link juice but no longer). This is bit unfortunate for sites that use search filters because they are such a valuable tool for the users.

Don

Ben,

I doubt that crawlers are going to access the robots.txt file for each request, but they still have to validate any url they find against the list of the blocked ones.

Glad to help,

Don

Hello,

Yes, multiple sitemaps are okay, and sometimes even advised!

You can read Google's official response here."..it's fine for multiple Sitemaps to live in the same directory (as many as you want!)..."

And you can see a case study showing how multiple sitemaps has helped traffic here on Moz.

Hope this helps,

Don

Linking to pages in the footer generates more links which dilutes link juice passed to all pages that are linked to. Usually the footer is where non-important SEO pages are located but important pages for customer experience like, accounts / support / contact / privacy / terms of service / legal.

You should always side with the USER EXPERIENCE it should not be frustrating for your users to find what they are looking for. Taking that into consideration you should also keep the links to less important SEO pages sitewide to a minimum and not duplicate your navigation unless it makes sense from a usability standpoint. Other-words don't do it to fill up space.