My wepgages aren't crawled by google
-
Most of my webpages aren't crawled by google.
Why is that and what can i do to make google index at least most of my webpages? -
Well, Google does have a crawl budget, they might be using that for your most popular pages. As long as your indexed pages number is going up, that means google is working its way through the backlog.
-
My website is a yellow pages site from Greece www.vreite.gr
It has registered more than 175.000 businesses and every business has 6 profile pages(main page,product page,feed page etc.)
Many visitors engage with these pages and are absolutely dunamic pages.
Is that a problem? -
The only site I can think of that would legitimately have 400,000 pages is Amazon.com. Google probably thinks your site is full of a ton of low quality content. Why in the world do you have that many pages? Are they low quality garbage? Do any visitors actually engage this even a fraction of them?
-
Hi
Yes my site is crawlable.
I checked if robots.txt or noindex tags and canonical urls and everything is fine.
Maybe is it because my website has over 400.000 pages -
Hi
You didn't answer the first part of Zoe's question - are you sure that your site is crawlable and that there are no issues with the robots.txt / noindex tags, ip detection systems, canonicals on all pages pointing to the home and so on. It's not because you can see all pages of your site in a browser that they are accessible/crawlable/indexable by google.
Try a crawl with Screaming Frog and user agent Googlebot to see if your pages can be crawled and indexed.
Backlinks are needed to have your site ranked for keywords - but it's not a prerequisite to have your site crawled. (noticed that a few times when a dev site was indexed by accident)
Without the actual url it's impossible to give a more detailed answer.
Dirk
-
Hi,
Backlinks certainly help, if there's no links at all to your site that could be a reason, but it's hard to say without looking deeper.
Are your internal pages all linked to each other? Does your website have a structured navigation system? This is also really necessary to ensure Google will index your whole site, not just a couple of pages.
Zoe
-
Hi
I added my website to Google Webmaster Tools and i checked my website and i don't have any crawling issues.
I added my website to Dmoz but the backling didn't appear yet.My site is live for about a year and google doesn't crawl most of my webpages yet.
Is it because i don't have quality backlinks?Thank you
-
Hi,
Firstly I'd check that Google can index your website. Have you added your site to Google Webmaster Tools? I'd start there and check for any crawl issues, especially your robots.txt file and any no-indexing of pages.
Secondly, if your website is brand new, I'd add your website to Dmoz & some relevant good quality sites like Yelp, Yell, Yellowpages, Google Plus (where relevant). Make sure the details you add to each match exactly with the details on your website. It will take some time for your site to appear in Google's index- sometimes a few days, sometimes a week or so- you can check by typing site:yourdomainname.com into a Google search to find the pages.
If your website is not new & has been indexed by Google before, I'd investigate whether you have a penalty. This post on penalties from white.net is really useful!
Hope this helps,
Zoe
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How do I ensure that colour variant products aren't flagged for being duplicate?
I have a site with 12 colour variants of 1 style. How do I ensure that these are not flagged as duplicate content as they currently have been?
Technical SEO | | Ashcastle0 -
My Homepage Won't Load if Javascript is Disabled. Is this an SEO/Indexation issue?
Hi everyone, I'm working with a client who recently had their site redesigned. I'm just going through to do an initial audit to make sure everything looks good. Part of my initial indexation audit goes through questions about how the site functions when you disable, javascript, cookies, and/or css. I use the Web Developer extension for Chrome to do this. I know, more recently, people have said that content loaded by Javascript will be indexed. I just want to make sure it's not hurting my clients SEO. http://americasinstantsigns.com/ Is it as simple as looking at Google's Cached URL? The URL is definitely being indexed and when looking at the text-only version everything appears to be in order. This may be an outdated question, but I just want to be sure! Thank you so much!
Technical SEO | | ccox10 -
Webmaster tools not showing links but Moz OSE is showing links. Why can't I see them in the Google Search Console
Hi, Please see attached photos. I have a website that shows external follow links when performing a search on open site explorer. However, they are not recognised or visible in search console. This is the case for both internal and external links. The internal links are 'no follow' which I am getting developer to rectify. Any ideas why I cant see the 'follow' external links? Thanks in advance to those who help me out. Jesse T7dkL5s T7dkL5s OkQmPL4 3qILHqS
Technical SEO | | jessew0 -
Will a Robots.txt 'disallow' of a directory, keep Google from seeing 301 redirects for pages/files within the directory?
Hi- I have a client that had thousands of dynamic php pages indexed by Google that shouldn't have been. He has since blocked these php pages via robots.txt disallow. Unfortunately, many of those php pages were linked to by high quality sites mulitiple times (instead of the static urls) before he put up the php 'disallow'. If we create 301 redirects for some of these php URLs that area still showing high value backlinks and send them to the correct static URLs, will Google even see these 301 redirects and pass link value to the proper static URLs? Or will the robots.txt keep Google away and we lose all these high quality backlinks? I guess the same question applies if we use the canonical tag instead of the 301. Will the robots.txt keep Google from seeing the canonical tags on the php pages? Thanks very much, V
Technical SEO | | Voodak0 -
Weird problems with google's rich snippet markup
Once upon a time, our site was ranking well and had all the markups showing up in the results. We than lost some of our rankings due to dropped links and not so well kept maintenance. Now, we are gaining up the rankings again, but the markups don't show up in the organic search results. When we Google site:oursite.com, the markups show up, but not in the organic search. There are no manual actions against our site. any idea why this would happen?
Technical SEO | | s-s0 -
Best way to handle pages with iframes that I don't want indexed? Noindex in the header?
I am doing a bit of SEO work for a friend, and the situation is the following: The site is a place to discuss articles on the web. When clicking on a link that has been posted, it sends the user to a URL on the main site that is URL.com/article/view. This page has a large iframe that contains the article itself, and a small bar at the top containing the article with various links to get back to the original site. I'd like to make sure that the comment pages (URL.com/article) are indexed instead of all of the URL.com/article/view pages, which won't really do much for SEO. However, all of these pages are indexed. What would be the best approach to make sure the iframe pages aren't indexed? My intuition is to just have a "noindex" in the header of those pages, and just make sure that the conversation pages themselves are properly linked throughout the site, so that they get indexed properly. Does this seem right? Thanks for the help...
Technical SEO | | jim_shook0 -
Will google let me do this
Hi i am working on my site at the moment www.in2town.co.uk and i am adding new sections and was thinking about buying domain names that best describe that section and which people would remember. so for example i am looking at adding a tenerife magazine to my site and would like to know if it would be wise to buy a domain name for example tenerife magazine and then have it directed to the section of my site. would this benefit my site in any way and would google allow this. instead of having in2town.co.uk and then tenerife magazine after it, sorry cannot find the slash as i am on a spanish keyborad at the moment, i would like to have something like tenerifemagazine.co..uk etc If anyone can give me advice on this then that would be great. also can anyone let me know if this is a wise idea or not, to have sub domain names on my main site. i would like to know if i had tenerifemagazine under the in2town domain name would it slow the site down or should i consider building a brand new site just for that and then making people aware that it comes under the in2town umbrella many thanks
Technical SEO | | ClaireH-1848861 -
Site 'filtered' by Google in early July.... and still filtered!
Hi, Our site got demoted by Google all of a sudden back in early July. You can view the site here: http://alturl.com/4pfrj and you may read the discussions I posted in Google's forums here: http://www.google.com/support/forum/p/Webmasters/thread?tid=6e8f9aab7e384d88&hl=en http://www.google.com/support/forum/p/Webmasters/thread?tid=276dc6687317641b&hl=en Those discussions chronicle what happened, and what we've done since. I don't want to make this a long post by retyping it all here, hence the links. However, we've made various changes (as detailed), such as getting rid of duplicate content (use of noindex on various pages etc), and ensuring there is no hidden text (we made an unintentional blunder there through use of a 3rd party control which used CSS hidden text to store certain data). We have also filed reconsideration requests with Google and been told that no manual penalty has been applied. So the problem is down to algorithmic filters which are being applied. So... my reason for posting here is simply to see if anyone here can help us discover if there is anything we have missed? I'd hope that we've addressed the main issues and that eventually our Google ranking will recover (ie. filter removed.... it isn't that we 'rank' poorly, but that a filter is bumping us down, to, for example, page 50).... but after three months it sure is taking a while! It appears that a 30 day penalty was originally applied, as our ranking recovered in early August. But a few days later it dived down again (so presumably Google analysed the site again, found a problem and applied another penalty/filter). I'd hope that might have been 30 or 60 days, but 60 days have now passed.... so perhaps we have a 90 day penalty now. OR.... perhaps there is no time frame this time, simply the need to 'fix' whatever is constantly triggering the filter (that said, I 'feel' like a time frame is there, especially given what happened after 30 days). Of course the other aspect that can always be worked on (and oft-mentioned) is the need for more and more original content. However, we've done a lot to increase this and think our Guide pages are pretty useful now. I've looked at many competitive sites which list in Google and they really don't offer anything more than we do..... so if that is the issue it sure is puzzling if we're filtered and they aren't. Anyway, I'm getting wordy now, so I'll pause. I'm just asking if anyone would like to have a quick look at the site and see what they can deduce? We have of course run it through SEOMoz's tools and made use of the suggestions. Our target pages generally rate as an A for SEO in the reports. Thanks!
Technical SEO | | Go2Holidays0