Website Crawl problems
-
I have a feeling that Google doesn't crawl my website.
E.g. this blogpost - I copy a sentence from it and paste it to Google. The page that shows up in search results is www.silvamethodlife.com/page/9/ - which is just a blog page with all the articles listed, not the link to the article itself!
Did anyone ever have this problem? It's definitely some technical issue.
Any advice will be deeply appreciated
Thanks
-
When i use the same tactic as you defined in your question’s explanation i was able to see your page in the search results...
Use this common in search bar to see how many pages are indexed in Google ‘site:example.com’.
May be not all the pages are indexed, for that try and check the following two things:
- Make sure all pages of the website are follow,index for Google and other search engines
- Update your sitemap and submit it to Google webmaster tool and you will see the efficiency in website page index.
Hope this helps!
-
I noticed that your category pages specify noindex,follow and index,follow. Might want to make a decision there.
Google has indexed 676 pages of the 817 on your site and, since their are quite a few URLs that specify nofollow correctly without the additional index like mentioned above, I'd say that is about right.
Did not see the issue you spoke of when I checked some random sentences.
Wouldn't be concerned unless you notice this is coupled with a decrease in organic traffic. If so start looking at off page factors.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Multilingual Version of a Website
Hi All, We created 'EN' and 'FR' version of a website and translated all labels and message from English to France with the help of Google Translator. Lets take an example: English version URL - https://www.sitegeek.com/softlayer France version URL - https://fr.sitegeek.com/softlayer France version also contain same reviews available on English version page. So the reviews content or language is same on both pages. To eliminate the duplicate content issue we put following meta tags on both 'EN" and 'FR' version pages : So My question is that (1.) Is this the correct implementation of Multilingual Version of a Website? (2.) Is Added meta tags work for both Google and Bing Search engine? (as Bing not indexing all pages) (3.) We are translated labels and messages from Google Translator. Is this the issue pages not being Indexed in Bing? (4.) Finally, What would the correct SEO approch if we translate our site in other languages? Rajiv
Intermediate & Advanced SEO | | gamesecure0 -
Can't crawl website with Screaming frog... what is wrong?
Hello all - I've just been trying to crawl a site with Screaming Frog and can't get beyond the homepage - have done the usual stuff (turn off JS and so on) and no problems there with nav and so on- the site's other pages have indexed in Google btw. Now I'm wondering whether there's a problem with this robots.txt file, which I think may be auto-generated by Joomla (I'm not familiar with Joomla...) - are there any issues here? [just checked... and there isn't!] If the Joomla site is installed within a folder such as at e.g. www.example.com/joomla/ the robots.txt file MUST be moved to the site root at e.g. www.example.com/robots.txt AND the joomla folder name MUST be prefixed to the disallowed path, e.g. the Disallow rule for the /administrator/ folder MUST be changed to read Disallow: /joomla/administrator/ For more information about the robots.txt standard, see: http://www.robotstxt.org/orig.html For syntax checking, see: http://tool.motoricerca.info/robots-checker.phtml User-agent: *
Intermediate & Advanced SEO | | McTaggart
Disallow: /administrator/
Disallow: /bin/
Disallow: /cache/
Disallow: /cli/
Disallow: /components/
Disallow: /includes/
Disallow: /installation/
Disallow: /language/
Disallow: /layouts/
Disallow: /libraries/
Disallow: /logs/
Disallow: /modules/
Disallow: /plugins/
Disallow: /tmp/0 -
Questions about websites coupon codes
Hi guys, i have 2 questions about my website of coupon codes: should i do redirect people of google to mywebsite, e.g. Someone is looking coupons for Sony or LG and arrive to brand Sony in my website but i show him offers for Sony in Amazon when he click in some offer, ¿that is correct? Footer images links. I saw many sites that put their logos in footer of online stores to get authority, ¿should i do that? Thank you so much.
Intermediate & Advanced SEO | | pompero990 -
Redesigned Website
Hi, I have redesigned my website in html whereas it was in .asp earlier. I have resubmitted my google sitemap but it still showing me old site pages in search except home page. My question is how i can immediately change my web presence. How i can get the benefit of my .asp page ranks? In addition my website still alive with .asp. What should be the strategy, should i remove this websites or have to redirect all pages to new. If i make 301 redirection then will it cause any issue in SEO, ranking etc ? Thx
Intermediate & Advanced SEO | | 1akal0 -
Moving Content To Another Website With No Redirect?
I've got a website that has lots of valuable content and tools but it's been hit too hard by both Panda and Penguin. I came to the conclusion that I'd be better off with a new website as this one is going to hell no matter how much time and money I put in it. Had I started a new website the first time it got hit by Penguin, I'd be profitable today. I'd like to move some of that content to this other domain but I don't want to do 301 redirects as I don't want to pass bad link juice. I know I'll lose all links and visitors to the original website but I don't care. My only concern is duplicate content. I was thinking of setting the pages to noindex on the original website and wait until they don't appear in Google's index. Then I'd move them over to the new domain to be indexed again. Do you see any problem with this? Should I rewrite everything instead? I hate spinning content...!
Intermediate & Advanced SEO | | sbrault741 -
Does 301 Redirect solve many problems?
Hi, there are many problems with my site. I have a lot of duplicate page titles and a lot of missing meta tags. However, I think most of them are BECAUSE i have a lot of duplicate pages. So I have read some articles and I will 301 redirect all the duplicated pages. Will this solve the problem with duplicate titles and missing meta tags as well? For example, my homepage has like 10 duplicated pages. Since they are duplicated, they have the same titles and they are all missing meta tags. I am planning to fill in meta tags JUST for the canonical page and redirect all duplicated pages to that page. Is this a good practice? Also, just curious, do different title tags and different meta tag description make the pages "not duplicated?" I assume it will still appear as duplicated.... Sorry if this was confusing...
Intermediate & Advanced SEO | | waltergah0 -
Crawl Budget on Noindex Follow
We have a list of crawled product search pages where pagination on Page 1 is indexed and crawled and page 2 and onward is noindex, noarchive follow as we want the links followed to the Product Pages themselves. (All product Pages have canonicals and unique URLs) Orr search results will be increasing the sets, and thus Google will have more links to follow on our wesbite although they all will be noindex pages. will this impact our carwl budget and additionally have impact to our rankings? Page 1 - Crawled Indexed and Followed Page 2 onward - Crawled No-index No-Archive Followed Thoughts? Thanks, Phil G
Intermediate & Advanced SEO | | AU-SEO0 -
Should subdomains be avoided for brand new websites?
When creating a brand new website, will setting it up as a subdomain provide ranking benefits? I understand that if it's an existing domain, it's better to use a subfolder because a subdomain is treated as a different domain. But is there any reason not to start a website with the keyword in the subdomain? For example: keyword.domain.com The SERP's are dominated by websites which contain some variation of the head term, but the disadvantage of doing a similar this is your website looks very similar. Thanks!
Intermediate & Advanced SEO | | JonDavies540