How do I find out which pages are being indexed on my site and which are not?
-
Hi,
I doing my first technical audit on my site. I am learning how to do an audit as i go and am a lost. I know some page won't be indexed but how do I:
1. Check the site for all pages, both indexed and not indexed
2. Run a report to show indexed pages only (i am presuming i can do this via screaming Frog or webmaster tool)
3. I can do a comparison between the two list and work out which pages are not being indexed.
I'll then need to figure out way. I'll cross this bridge once i get to it
Thanks Ben
-
Hi Ben,
I'd echo what Patrick has said and probably recommend his first suggestion the most. Google Webmaster Tools is a good way of checking indexation and if you have a large site with lots of categories, you can even break down the sitemaps by category so that you can see if certain areas are having problems.
Here is an old, but still relevant post on the topic:
http://www.branded3.com/blogs/using-multiple-sitemaps-to-analyse-indexation-on-large-sites/
In terms of creating the sitemap, Screaming Frog has an option under Advanced Export for creating an XML sitemap file for you which works very well. You just need to make sure you're only including pages that you want indexed in there.
Cheers.
Paddy
-
Hi Patrick,
Thanks for replying.
Can you recommend any tools for creating the site map i've had a look around and the few i've found seem to all deliver different results? One has been submitted previously so i need to go through the process for myself so i can under these basics.
I've had a read up on robot txt so i understand what is happening there from an exclusion perspective and once i understand how the XML site works ill be able to do an audit as mentioned above.
Ben
-
Ben,
You can check a couple things:
- Have you submitted your XML site map to Google? If not, create one and get it submitted so you tell Google what pages you want indexed.
- Submit your domain and all pages through Google Webmasters Tool as well (Login > left side bar > Crawl > Fetch as Google
- Screaming Frog is an awesome software, so yes, if you have it, use it to scan your pages
- Try and do a simple "site:domainname.com" search in Google to see what is being indexed from your domain
Cross reference it all and you will then have a better understanding. I do believe, your sitemap is crucial in telling Google exactly what pages you do and do not want indexed. They will follow that. You're on the right track and hope my input was helpful! - Patrick
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Why does moz give different page authority to the same page if a visit comes from adwords vs organic search?
When clicking on an adwords ad the page the landing page has a page authority of 26. When clicking on organic search to the same exact landing page the page authority is 37. Why is this. Does moz or, more importantly Google see these as the same or separate pages? Thanks Tom
Moz Pro | | ffctas1 -
Source page showsI have 2 h1 tags on my page. I can only find one.
When I grade my page it says I have more than one h1 tag. I view the source page and it shows there are two h1 headings with the same wording. If I delete the one h1 heading I can find, the page source shows I have deleted both of them. I don't know how to get to the other heading to delete it. And I'm off page one of google! Can anybody help? Clay Stephens
Moz Pro | | Coot0 -
Right way to find quality free/non free directories to submit web page
I have one question about link building. What will be right way to find quality free/non free directories to submit my web page. My web page thematic is nail-art. Thanks and best regards
Moz Pro | | komir20040 -
Tool to find compeitor keyword overlap?
I want to know which competitors rank for the same things I do. Is there a tool that can give me this info?
Moz Pro | | ShearingsGroup0 -
Duplicate page report
We ran a CSV spreadsheet of our crawl diagnostics related to duplicate URLS' after waiting 5 days with no response to how Rogerbot can be made to filter. My IT lead tells me he thinks the label on the spreadsheet is showing “duplicate URLs”, and that is – literally – what the spreadsheet is showing. It thinks that a database ID number is the only valid part of a URL. To replicate: Just filter the spreadsheet for any number that you see on the page. For example, filtering for 1793 gives us the following result: | URL http://truthbook.com/faq/dsp_viewFAQ.cfm?faqID=1793 http://truthbook.com/index.cfm?linkID=1793 http://truthbook.com/index.cfm?linkID=1793&pf=true http://www.truthbook.com/blogs/dsp_viewBlogEntry.cfm?blogentryID=1793 http://www.truthbook.com/index.cfm?linkID=1793 | There are a couple of problems with the above: 1. It gives the www result, as well as the non-www result. 2. It is seeing the print version as a duplicate (&pf=true) but these are blocked from Google via the noindex header tag. 3. It thinks that different sections of the website with the same ID number the same thing (faq / blogs / pages) In short: this particular report tell us nothing at all. I am trying to get a perspective from someone at SEOMoz to determine if he is reading the result correctly or there is something he is missing? Please help. Jim
Moz Pro | | jimmyzig0 -
On-Page SEO Fixes - Are They Relative?
So, I'm implementing on-page fixes for a site that my company runs SEO services for (www.ShadeTreePowersports.com). However, I was wondering if there was a way to rank a pages' SEO quality, in general? As of now, it seems like the only way your recommendations can be consumed and altered is on a keyword basis. However, this seems be the reason I have a good amount of my F-Grades. Since my website sells powersports apparel and accessories, we cover a variety of applicable (but different) keywords like 'Motorcycle parts' or 'snow tubes,' because we sell so many different types of products. But, when I look at my F-Grades - SEOMoz is telling me my homepage is ranking poorly for a multitude of those pertinent keywords - but only because my page isn't catered specifically to each of them (IE: 'Snowmobile Parts' - 'Water Sport Apparel') But, with so many different types of products, catering to a specific one is impossible and would be detrimental. Is there a way to see how a page ranks, without factoring in those keywords? Or a better way that I can use these recommendations more efficiently? Thanks guys!
Moz Pro | | BrandLabs0 -
Comparing with Open Site Explorer
Hi, I am trying to compare a website that has a url of e.g. https://mysite.com on Open Site Explorer. Any idea how to do this? It will only compare it when I use www and it also doesn't accept https. So I am comparing www.mysite.com which has redirects on it https://mysite.com but I am worried it's not comparing the right stats? If this makes sense and you can help it would be greatly appreciated. Cheers
Moz Pro | | Hughescov0 -
Domain vs Page
I see a lot of different metrics pointing to domain or page. What is the difference between these two definitions?
Moz Pro | | Gfrink0