How do I find out which pages are being indexed on my site and which are not?
-
Hi,
I doing my first technical audit on my site. I am learning how to do an audit as i go and am a lost. I know some page won't be indexed but how do I:
1. Check the site for all pages, both indexed and not indexed
2. Run a report to show indexed pages only (i am presuming i can do this via screaming Frog or webmaster tool)
3. I can do a comparison between the two list and work out which pages are not being indexed.
I'll then need to figure out way. I'll cross this bridge once i get to it
Thanks Ben
-
Hi Ben,
I'd echo what Patrick has said and probably recommend his first suggestion the most. Google Webmaster Tools is a good way of checking indexation and if you have a large site with lots of categories, you can even break down the sitemaps by category so that you can see if certain areas are having problems.
Here is an old, but still relevant post on the topic:
http://www.branded3.com/blogs/using-multiple-sitemaps-to-analyse-indexation-on-large-sites/
In terms of creating the sitemap, Screaming Frog has an option under Advanced Export for creating an XML sitemap file for you which works very well. You just need to make sure you're only including pages that you want indexed in there.
Cheers.
Paddy
-
Hi Patrick,
Thanks for replying.
Can you recommend any tools for creating the site map i've had a look around and the few i've found seem to all deliver different results? One has been submitted previously so i need to go through the process for myself so i can under these basics.
I've had a read up on robot txt so i understand what is happening there from an exclusion perspective and once i understand how the XML site works ill be able to do an audit as mentioned above.
Ben
-
Ben,
You can check a couple things:
- Have you submitted your XML site map to Google? If not, create one and get it submitted so you tell Google what pages you want indexed.
- Submit your domain and all pages through Google Webmasters Tool as well (Login > left side bar > Crawl > Fetch as Google
- Screaming Frog is an awesome software, so yes, if you have it, use it to scan your pages
- Try and do a simple "site:domainname.com" search in Google to see what is being indexed from your domain
Cross reference it all and you will then have a better understanding. I do believe, your sitemap is crucial in telling Google exactly what pages you do and do not want indexed. They will follow that. You're on the right track and hope my input was helpful! - Patrick
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Duplicate pages coming from links from the login page - what should we do about them?
This is a follow on to an earlier question which was well answered by Dirk Ceuppens regarding abnormal crawl issues. We are seeing that the issues relating to Duplicate Pages are coming from links from the login page which shows information about where the user was redirected from. For example, if the visitor is not logged on and wishes to wish-list an item, they will be redirected to the login page, with the item code and intended action in the url; which can then continue on to the desired page once logged on. The MOZ crawler is seeing these pages as having Duplicated Content whilst they are all the same apart from a piece of information in the URL. Should we be blocking these duplications? Are they a risk to us? What should we be doing? Many thanks, Sarah
Moz Pro | | Mutatio_Digital0 -
MOZ Toolbar 3.0: Can't Find the Meta Description On My Page, Why?
Hey, The new MOZ toolbar is unable to identify the meta description on any of the pages for my doman www.1099pro.com. Is there any reason that someone can see why that would be? The tag should be correct on all pages. Thanks! -Mike
Moz Pro | | Stew2220 -
Is there any report / tool that gives me last cache date for each page on my site ?
my site has several hundred pages, and it is important for me to know last crawl date of each page as well as number of pages cralwed in a particualr period ( from / to date ). is there any report in seomoz that can help for this ? or any other suggestion ?
Moz Pro | | elegantmicroweb0 -
Open site explorer
"Unable to retrieve linking pages on this anchor at this time." This is the notice I get when trying to see links for anchor text. Can someone help?
Moz Pro | | Joseph-Green-SEO0 -
Strange nothing site ranking
Hi There. If you check who ranks for "credit cards" there is a website https://www.woolworthsmoney.com.au/ that is in position #5 This is a highly competitive keyword, but OpenSiteExporer.org cannot give me any backlinks for it. it says "No Data Available for this URL" The same thing happens in Market Samurai - no data 1. What are these guys doing that the others are not? 2. How come OSE can't pull any data for it?
Moz Pro | | SearchProduct0 -
How often does Open Site Explorer Update?
How often does Open Site Explorer Update? Just trying to get a rough idea. Great tool btw.
Moz Pro | | seo3210 -
How to rank each page correctly in the search engines
Hello everyone, I am new to SEOmoz and SEO in general. I am trying to wrap my head around everything and one of the issues I am having trouble grasping is how each individual page on our website plays into ranking for certain keywords. I will explain in more detail below: 1.) We are not a blog, we have a pretty static website. 2.) We can add as many pages to our website as needed (for SEO purposes) 3.) Our site is fairly small, about 25 pages on the website. Now with that information above, here are our main questions & concerns: 1.) What is the best way to rank in the search engines for 50 of our best converting keywords with only 25 pages on our website? 2.) We have 2-3 competitors that rank in positions 1-3 for about 30 of our keywords we are trying to rank for. But no matter what you type in, for example, "cars" "buy cars" "buy a car online" "truck prices" etc, the results that the search engines display our their homepages. Not the other pages on their websites that actually rank for each one of those individual keywords. Is this normal? That is something that we would like for our rankings to do if we have any control over it. 3.) Should each page try to rank for a unique individual keyword? For example, when using the SEOmoz On-page Optimization tool, should I try ranking 2 pages for "buy cars online" or just 1 page? Say my homepage has an "A" grade for the keyword, "buy cars online" but so does my second page on my website. Are these pages essentially competing with each other? Or will the search engines pick their favorite page and display the one they like the best? Any help is greatly appreciated and I look forward to getting this cleared up. Thanks
Moz Pro | | threebiz0 -
SEOMoz Pro and Multiple Niche Sites
Currently I have 15 affiliate marketing sites and three blogs which are monetized. My budget dictates the base pro package so may I get some recommendations on how or if I can do analysis over time for all the sites. I ask this because it seems you would set your campaigns for long-term but I imagine you delete/modify/add campaigns to analyze different key words. Thanks for your help.
Moz Pro | | JavaManOne0