My wepgages aren't crawled by google
-
Most of my webpages aren't crawled by google.
Why is that and what can i do to make google index at least most of my webpages? -
Well, Google does have a crawl budget, they might be using that for your most popular pages. As long as your indexed pages number is going up, that means google is working its way through the backlog.
-
My website is a yellow pages site from Greece www.vreite.gr
It has registered more than 175.000 businesses and every business has 6 profile pages(main page,product page,feed page etc.)
Many visitors engage with these pages and are absolutely dunamic pages.
Is that a problem? -
The only site I can think of that would legitimately have 400,000 pages is Amazon.com. Google probably thinks your site is full of a ton of low quality content. Why in the world do you have that many pages? Are they low quality garbage? Do any visitors actually engage this even a fraction of them?
-
Hi
Yes my site is crawlable.
I checked if robots.txt or noindex tags and canonical urls and everything is fine.
Maybe is it because my website has over 400.000 pages -
Hi
You didn't answer the first part of Zoe's question - are you sure that your site is crawlable and that there are no issues with the robots.txt / noindex tags, ip detection systems, canonicals on all pages pointing to the home and so on. It's not because you can see all pages of your site in a browser that they are accessible/crawlable/indexable by google.
Try a crawl with Screaming Frog and user agent Googlebot to see if your pages can be crawled and indexed.
Backlinks are needed to have your site ranked for keywords - but it's not a prerequisite to have your site crawled. (noticed that a few times when a dev site was indexed by accident)
Without the actual url it's impossible to give a more detailed answer.
Dirk
-
Hi,
Backlinks certainly help, if there's no links at all to your site that could be a reason, but it's hard to say without looking deeper.
Are your internal pages all linked to each other? Does your website have a structured navigation system? This is also really necessary to ensure Google will index your whole site, not just a couple of pages.
Zoe
-
Hi
I added my website to Google Webmaster Tools and i checked my website and i don't have any crawling issues.
I added my website to Dmoz but the backling didn't appear yet.My site is live for about a year and google doesn't crawl most of my webpages yet.
Is it because i don't have quality backlinks?Thank you
-
Hi,
Firstly I'd check that Google can index your website. Have you added your site to Google Webmaster Tools? I'd start there and check for any crawl issues, especially your robots.txt file and any no-indexing of pages.
Secondly, if your website is brand new, I'd add your website to Dmoz & some relevant good quality sites like Yelp, Yell, Yellowpages, Google Plus (where relevant). Make sure the details you add to each match exactly with the details on your website. It will take some time for your site to appear in Google's index- sometimes a few days, sometimes a week or so- you can check by typing site:yourdomainname.com into a Google search to find the pages.
If your website is not new & has been indexed by Google before, I'd investigate whether you have a penalty. This post on penalties from white.net is really useful!
Hope this helps,
Zoe
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Why don't sites using Drupal have keywords
Why don't the vast majority of sites using Drupal list keywords in the head section? Is there another convention used in Drupal that serves the same purpose for SEO? I noticed most of the Drupal info pages about keywords seem to drop off around 2010
Technical SEO | | fxarechiga0 -
"Yet-to-be-translated" Duplicate Content: is rel='canonical' the answer?
Hi All, We have a partially internationalized site, some pages are translated while others have yet to be translated. Right now, when a page has not yet been translated we add an English-language page at the url https://our-website/:language/page-name and add a bar for users to the top of the page that simply says "Sorry, this page has not yet been translated". This is best for our users, but unfortunately it creates duplicate content, as we re-publish our English-language content a second time under a different url. When we have untranslated (i.e. duplicate) content I believe the best thing we can do is add which points to the English page. However here's my concern: someday we _will_translate/localize these pages, and therefore someday these links will _not _have duplicate content. I'm concerned that a long time of having rel='canonical' on these urls, if we suddenly change this, that these "recently translated, no longer pointing to cannonical='english' pages" will not be indexed properly. Is this a valid concern?
Technical SEO | | VectrLabs0 -
Google will index us, but Bing won't. Why?
Bing is crawling our site, but not indexing it, and we cannot figure out why -- plus it's being indexed fine in Google. Any ideas on what the issue with Bing might be? Here's are some details to let you know what we've already checked/established: We have 4 301’s and the rest of our site checks out We’ve already established our Robots is ok, and that we are fixing our site map/it's in fine shape We do not see anything blocking bingbot access to the site There is no varnish or any load balancers, so nothing on that end that would be blocking the access We also don't see any rules in the apache or the .htaccess config that would be blocking the access
Technical SEO | | Alex_RevelInteractive0 -
I've had a sudden a increase in crawl issues as of yesterday (like 300 from a steady 10, does anyone else have this issue?
the main issue is that it's now indexing both www and http:// - anyone else got this issue or had any changes suddenly on their crawl results?
Technical SEO | | beckyhy0 -
I have a 404 error on my site i can't find.
I have looked everywhere. I thought it might have just showed up while making some changes, so while in webmaster tools i said it was fixed.....It's still there. Even moz pro found it. error is http://mydomain.com/mydomain.com No idea how it even happened. thought it might be a plugin problem. Any ideas how to fix this?
Technical SEO | | NateStewart0 -
Moving articles to new site, can't 301 redirect because of panda
I have a site that is high quality, but was hit by penguin and perhaps panda. I want to remove some of the articles from my old site and put them on my new site. I know I can't 301 redirect them because I will be passing on the bad google vibes. So instead, I was thinking of redirecting the old articles to a page on the old site which explains that the article is moved over to the new site. I assume that's okay? I'm wondering how long I should wait between the time I take them down from the old site to the time I repost them on the new site. Do I need to wait for Google to de-index them in order to not be considered duplicate content/syndication? We'll probably reword them a bit, too - we really want to avoid panda. Thanks!
Technical SEO | | philray
Phil0 -
OpenSite Explorer doesn't show Twitter
When I type competitors sites into opensiteexplorer it shows their Twitter page as one of the tops links back. However, my site, our Twitter page isn't even on the links back. Does twitter even though no follow hold any value? Why can't opensite explorer see it? If OSE can't see it then do you think Google isn't?
Technical SEO | | PhotoGazza0 -
Google indexing less url's then containded in my sitemap.xml
My sitemap.xml contains 3821 urls but Google (webmaster tools) indexes only 1544 urls. What may be the cause? There is no technical problem. Why does Google index less URLs then contained in my sitemap.xml?
Technical SEO | | Juist0