Bot Crawling issues
-
Dear all, Is this cache:www.subhavaastu.com is not working now with Google. Why it is not showing when the site is crawled. What were the new algorithms which Google is adopted? in my searches of all of my site internal links of www.SubhaVaastu.com, I observed only 404 instead of Google visiting time and date. After observing this 404 of each of my site links, I understand Google stopped crawling my site.
Some more examples are shown below with other websites:
cache:www.vastuwebsite.com ("NOT" showing when Google visited this site)
cache:www.vastuconsultantusa.com ("NOT" showing when Google visited this site)
cache:www.shubhavaastu.com ("NOT" showing when Google visited this site)
cache:www.subhavastu.com (Showing when Google visited this site)
To my surprise, I noticed that Google crawled latest links in my site, which I added a new link (https://www.subhavaastu.com/remove-negativity.html) just 10 days back, this new link was clearly crawled by Google. I typed "remove negativity subhavaastu", I saw the results with this new page in the SERP, but on the same way when I typed "cache:www.subhavaastu.com/remove-negativity.html", it is showing again 404.
what is happening with Google, is Google is following any new algorithms now. Is google changed any new concept? or, is my site is penalized in any case, I think it may not be, because if my site is penalized, then Google should not visit the new links and should not show the results with my site. Coming with my site, its pure from viruses, no malicious codes. Indeed it's an article based site, which has a good reputation. This domain is taken in the year 2003. We never spam anywhere we never did any wrong methods.
If my site is penalized, then is it manual penalized or bot penalized. I thoroughly checked webmaster tools and google console, I never found any notices or any note from Google.
Require experts analyzation on this doubt. Thanks in Advance.
-
Dear SeoMAN,
Thanks for your answer. I just tried at google, its shows normal, I just checked google search console and webmaster tools, there is nothing serious shows. Can it be possible to try to find out what was the main cause.
Is I taken the right image, is I searched exactly what you are asking about?
Many thanks.
-
Dear Vijay, Thank You.
You are an expert in this field, I respect your thought, but I am thinking there should be some other reason, that might cause of this problem. I am MOZ PRO subscriber, if there is, canonical issues, I will get such notifications. Long back, there are some canonical issues, we fix them.
Site is totally free from malware and viruses, I just now also observed Google webmaster tools and checked other ways, there is nothing viruses found in my site.
Can it be possible to find any other issues which caused this problem.
Many thanks.
-
You are asking quite a lot of questions here, basically if your site has been around for a reasonable amount of time and your page is accessible and can be crawled by search engines you should be able to Google your domain name (including the TLD) like "mydomain.com", if that doesn't show up you either have a serious technical error or you have had a penalty.
As for telling whether it is bot or manual Google search console should give you a bit more information. Personally as far as I can see a penalty is a penalty and needs to be overcome one way or another, whether it is automatic or manual is a bit beside the point!
-
It can be due to a lot of SEO/technical issues, Here are a couple of observations/suggestions:
- Your site works with https:// and with http:// both and also with www. and without www. , this might be an issue for duplicate content. Please make sure that you have only one version with or without www. or https://
- Check your website for malware and viruses just as a precaution.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
SERP Cannibalization Issue
Hi Moz Community, Glad to be here with you. I need some help from you guys. I have a new client website: [https://deluxemassage.co.uk/](link url) My client runs this type of business offering professional therapeutic health services. It offers massage services on demand ("mobile", therapists visiting clients at home). The client has his business registered, and the therapists listed on the website are carefully vetted, all having training in different types of massage. The client's problem is that until 2 months ago the site was in position 3-4 in Google (somewhat stable) with the main page for the keyword "mobile massage London". Recently I noticed that another page on the site that lists all therapists - i.e. this page [https://deluxemassage.co.uk/therapists/](link url)- is on the first page of Google for the above keyword, although both the title tag, meta description and the page on the site do not target the above keyword.
Algorithm Updates | | Koyel9754
I notice that both pages have serious fluctuations in the SERP for the mentioned keyword, sometimes appearing both in the results.
The Home page is marked as rel=canonical Of course the position in SERP fluctuates from 8, 11, 12, 7. What should I do to solve this problem? Any thoughts here would be appreciated. Thanks0 -
Google Images search traffic and image thumbnail issues
Hi MOZ community! Need a little help with a strange issue we are seeing of late on our project CareerAddict.com. We have seen a sudden and significant drop in image visibility in Search Console from the 27th August onwards. I understand that Google has been updating their filters and other bits in image search, so maybe this could have impacted us? I also noticed that the images which are mapped to our articles are not the full featured article 700px wide images which we provide to Google in the Structured Data. They are instead taking the OG share 450px wide images now on many occasions. You can see this by searching for "careeraddict.com" in images. Any insight or suggestions welcome on both of these. Interested to understand if any other webmasters are experiencing other or similar problems with image visibility in Google also. Thanks!
Algorithm Updates | | dqmedia0 -
Issue with Category Ranking on Page 1 vs. Homepage Ranking on Page 2
A client has a high-volume keyword that is rendering different results, whether it is on page one or page two of Google SERPs. If the keyword is on page one, ONLY the category page is ranking. When the keyword bumps off to page two, BOTH the category AND the homepage are ranking. This is happening on our IP and theirs, incognito and personalized searches. This has been happening since February. Any thought/insights would be greatly appreciated, thank you!!!!
Algorithm Updates | | accpar0 -
Struggling with Google Bot Blocks - Please help!
I own a site called www.wheretobuybeauty.com.au After months and months we still have a serious issue with all pages having blocked URLs according to Google Webmaster Tools. The 404 errors are returning a 200 header code according to the email below. Do you agree that the 404.php code should be changed? Can you do that please ? The current state: Google webmaster tools Index Status shows: 26,000 pages indexed 44,000 pages blocked by robots. In late March, we implemented a change recommended by an SEO expert and he provided a new robots.txt file, advised that we should amend sitemap.xml and other changes. We implemented those changes and then setup a re-index of the site by google. The no of blocked URLs eventually reduced in May and June to 1,000 for a few days – but now the problem has rapidly returned. The no of pages that are displayed in a google search request of www.google.com.au where the query was ‘site:wheretobuybeauty.com.au’ is 37,000: This new site has been re-crawled over last 4 weeks. About the site This is a Linux php site and has the following: 55,000 URLs in sitemap.xml submitted successfully to webmaster tools robots.txt file has been modified several times: Firstly we had none Then we created one but were advised that it needed to have this current content: User-agent: * Disallow: Sitemap: http://www.wheretobuybeauty.com.au/sitemap.xml
Algorithm Updates | | socialgrowth0 -
A Serious drop in Pages crawled per day
On 21st April ,I spotted a sudden decrease in pages crawled per day.Previously it was about 5,000 bust after the drop it reached to 225.From the crawl rate never spiked. Here is my website url - http://www.wpstuffs.com/ 8fQHW2G.png
Algorithm Updates | | vividvilla0 -
Does articles for SEO purposes have a minimal and maximum word count in ordered to be crawled/indexed by Google and other search engines?
Does articles for SEO purposes have a minimal and maximum word count in ordered to be crawled/indexed by Google and other search engines?
Algorithm Updates | | WebRiverGroup0 -
Google Webmaster Tools: Quality Issues on http://www.enakliyat.com.tr/
Specifically, we detected low-quality pages on your site which do not provide substantially unique content or added value. Examples could include thin affiliate pages, doorway pages, automatically generated content, or copied content. We encourage you to make changes to your site so that it meets our quality guidelines. Once you've made these changes, please submit your site for reconsideration in Google's search results. Google Webmaster Tool send me this message I think the low-quality pages is like the this http://www.enakliyat.com.tr/detaylar/bursa-fethiye-ucevler-nakliye-5834 page and we have so many pages like this... Example 1: http://www.enakliyat.com.tr/detaylar/evden-eve-nakliyat-5906 **Example 2 : **http://www.enakliyat.com.tr/detaylar/cekmekoy-izmit-5905 **Example 3: **http://www.enakliyat.com.tr/detaylar/evden-eve-nakliyat-5906 What should I do to these pages HELP 😞
Algorithm Updates | | iskq0 -
Should we crawl user profiles?
Hi there, We have a service whereby we measure people's personalities and reputation. We would like to be able to search: "Reputation of XXX" and you would find the particular reputation profile of that person. However the differences for each page are limited. Effectively the name, and a few numbers that measure such reputation. Would crawling these profiles affect us due to "thin content"? Would you consider any alternative strategy? Thanks!
Algorithm Updates | | josei0