Truncate page URLs
-
We have some pages (for example a contact us form) for which the URL is modified by the CMS depending on the referring page (this helps to put the form submission in context for the sales reps who get the contact submission).
The SEOmoz crawler considers each URL a new page -- and so numbers like in diagnostics are all inflated as the same page is listed multiple times (e.g. for too many links)
Is there a setting to change what the crawler considers to be the same page?
Here are two URLs for the same page that the reports treat as separate pages:
http://www.spirent.com/About-Us/Contact_us.aspx?referurl=0F528F4D703D8BB3523738D6373AA8AD
http://www.spirent.com/About-Us/Contact_us.aspx?referurl=10ACDA6055244E369395223437FDCF30
The page is actually: http://www.spirent.com/About-Us/Contact_us.aspx
Thanks
Ken
-
As you can see here, this is an issue as Google are indexing many variations of the same page although this means that somewhere is linking to them unless your site is set up so that even a crawler passing through links to your contact page is creating the query parameter in the URL's.
To resolve this, you need to add the following to your robots.txt file:-
Disallow: ?referurl=
This will prevent any URL's passing that query parameter from getting crawled and indexed ensuring that only the originals of the pages will appear in search engines and not flag as duplicate content.
Hopefully, someone from SEOmoz can add as to whether there is an option for obeying robots.txt directives within their crawler so that these URL's are not listed as I'm not sure.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Forums and soical network pages - link analysis
Hi, I am working on reputation management project and i reach the point when i found that forums pages and social pages are hard to analyse their links and page rank. for example if you search these kind of pages on http://www.opensiteexplorer.org/ , it wont show any result And if you try to find the page rank on Moz toolbar it will show 1 pagerank How would you do guys when u are in this situation, Do you consider these kind of pages different than normal pages? Thanks
Moz Pro | | MohammadSabbagh0 -
On-Page SEO Fixes - Are They Relative?
So, I'm implementing on-page fixes for a site that my company runs SEO services for (www.ShadeTreePowersports.com). However, I was wondering if there was a way to rank a pages' SEO quality, in general? As of now, it seems like the only way your recommendations can be consumed and altered is on a keyword basis. However, this seems be the reason I have a good amount of my F-Grades. Since my website sells powersports apparel and accessories, we cover a variety of applicable (but different) keywords like 'Motorcycle parts' or 'snow tubes,' because we sell so many different types of products. But, when I look at my F-Grades - SEOMoz is telling me my homepage is ranking poorly for a multitude of those pertinent keywords - but only because my page isn't catered specifically to each of them (IE: 'Snowmobile Parts' - 'Water Sport Apparel') But, with so many different types of products, catering to a specific one is impossible and would be detrimental. Is there a way to see how a page ranks, without factoring in those keywords? Or a better way that I can use these recommendations more efficiently? Thanks guys!
Moz Pro | | BrandLabs0 -
Mozcape API Batching URLs LIMIT
Guys, there's an example to batching URLs using PHP: http://apiwiki.seomoz.org/php Which is the maximum number of URLs I can add to that batch?
Moz Pro | | Srvwiz0 -
The keyword ranking report takes into account all my website urls? Can I specify the URLs where I want to track the keywords?
I don't know if my weekly reports are reporting the ranking of my keywords correctly. I have added some new keywords, since that all my reports are in red numbers. I don't know if this is happening because I did something wrong, or if is because my rankings are really falling down.
Moz Pro | | hockerty0 -
URL Encoding
HI SEOmoz has finished crawling the site and surprised me with nearly 4k of 301's all the 301 are on my deal pages Example of the 301 http://www.economy-car-leasing.co.uk/van-leasing-deals/ford/transit-lease/transit-lwb-el-minibus-diesel-rwd-high-roof-17-seater-tdci-135ps%3D586165 as you can see from the above URL it returns a 404 but the URL is actually sent as below http://www.economy-car-leasing.co.uk/van-leasing-deals/ford/transit-lease/transit-lwb-el-minibus-diesel-rwd-high-roof-17-seater-tdci-135ps=586165 For some reason SEOmoz crawler is converting the = to %3d and reporting its a 301 even though it returns 404 Is this an error on SEOMOZ part ? or is there an error on my site Googlebot when i do a fetch as Google bot returns all on with the = sign and every other tool i have tried is ok too so not sure why SEOMOZ is seeing it different and then adding the URL as a 301 I am hoping this is just a glitch on the report tool part as im struggling since a recent site 301
Moz Pro | | kellymandingo0 -
How to rank each page correctly in the search engines
Hello everyone, I am new to SEOmoz and SEO in general. I am trying to wrap my head around everything and one of the issues I am having trouble grasping is how each individual page on our website plays into ranking for certain keywords. I will explain in more detail below: 1.) We are not a blog, we have a pretty static website. 2.) We can add as many pages to our website as needed (for SEO purposes) 3.) Our site is fairly small, about 25 pages on the website. Now with that information above, here are our main questions & concerns: 1.) What is the best way to rank in the search engines for 50 of our best converting keywords with only 25 pages on our website? 2.) We have 2-3 competitors that rank in positions 1-3 for about 30 of our keywords we are trying to rank for. But no matter what you type in, for example, "cars" "buy cars" "buy a car online" "truck prices" etc, the results that the search engines display our their homepages. Not the other pages on their websites that actually rank for each one of those individual keywords. Is this normal? That is something that we would like for our rankings to do if we have any control over it. 3.) Should each page try to rank for a unique individual keyword? For example, when using the SEOmoz On-page Optimization tool, should I try ranking 2 pages for "buy cars online" or just 1 page? Say my homepage has an "A" grade for the keyword, "buy cars online" but so does my second page on my website. Are these pages essentially competing with each other? Or will the search engines pick their favorite page and display the one they like the best? Any help is greatly appreciated and I look forward to getting this cleared up. Thanks
Moz Pro | | threebiz0 -
Why am I getting 400 client errors on pages that work?
Hi, I just done the initial crawl on y domain and I sem to have 80 400 client errors. However when I visit the urls the pages are loading fine. Any ideas on why this is happening and how I can resolve the problem?
Moz Pro | | moesian0 -
Use of the tilde in URLs
I just signed up for SEOMoz and sent my site through the first crawl. I use the tilde in my rewritten URLs. This threw my entire site into the Notice section 301 (permanent redirect) since each page redirects to the exact URL with the ~, not the %7e. I find conflicting information on the web - you can use the tilde in more recent coding guidelines where you couldn't in the old. It would be a huge thing to change every page in my site to use an underscore instead of a tilde int he URL. If Google is like SEOMoz and is 301 redirecting every page on the site, then I'll do it, but is it just an SEOMoz thing? I ran my site through Firebug and and all my pages show the 200 response header, not the 301 redirect. Thanks for any help you can provide.
Moz Pro | | fdb0