Crawl anamoly issue on Search Console
-
Has anyone checked the crwal anamoly issue under the index section on Search console? We recently move to a new site and I'm seeing a huge list of excluded urls which are classified as crawl anamoly (they all lead to 404 page). Does anyone know that if we need to 301 redirect all the links? Is there any other smarter/ more efficiently way to deal with them like set up canonical link (I thought that's what they're used for isn't it?)
Thanks!
-
Did you keep the old url in webmaster tools? With sitemap of old domain that redirects? You should make sure that was submitted for crawl so google see's the 301's. Also make sure you redirected all versions http, https, www, etc of old domain* to new https//: version.
Are the pages in fact 404 pages? are they pages in your sitemap? be careful too that they are not bad internal links. Did you crawl site with moz?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Search ranking for a term dropped from 1st/2nd to 106th in 3 months
Hello all, Just a couple notes first. I have been advised to be vague on the search term we've dropped on (in case this page ranks higher than our homepage for it). If you search for my name in Google though you should be able to figure out where I work (I'm not the soccer player). While I am looking for an answer, I've also posted this question on a couple other forums (see https://www.webmasterworld.com/google/4934323.htm and https://productforums.google.com/forum/?utm_medium=email&utm_source=footer#!msg/webmasters/AQLD7lywuvo/2zfFRD6oGAAJ) which have thrown up more questions than answers. So I have posted this as a discussion. We've also been told we may have been under a negative SEO attack. We saw in SEMRush a large number of backlinks in October/November/December - at about the same time we disavowed around 1m backlinks (more on this below) but we can't see this reflected in Moz. We just got off a call with someone at Moz to try and work this out and he suggested we post here - so here goes... On 4th October for the search term 'example-term' we dropped from number 2 to number 9 on Google searches (this was confirmed in Google Search Console). We also paid an external SEO consultant to review our site and see why we are dropping on the term 'example-term'. We've implemented everything and we're still dropping, the consultant thinks we may have been penalised in error (as we are a legitimate business and we're not trying to do anything untoward). In search console you could see from the graphs on the term we used to rank 1st and 2nd (you could go back 2 or 3 years and still see this). The thing we do find confusing is that we still rank very highly (if not 1st) for 'example-term + uk' and our brand name - which is very similar to 'example-term'. Timeline of events of changes: 2nd October 2018 midday: Added a CTA using something called Wisepops over the homepage - this was a full screen CTA for people to pledge on a project on our site helping with the tsunami in Indonesia (which may have had render blocking elements on). 4th October: we added a Google MyBusiness page showing our corporate headquarters as being in the UK (we did flag this on the Google MyBusiness forums and both people who responded said adding a MyBusiness page would not affect our drop in rankings). 4th October: dropped from number 2 to number 9 on Google searches (this was confirmed in Google Search Console) 4th October: Removed the Wisepops popup 5th November: Server redirect so anything coming in on / was redirected to a page without a / 12th November: Removed around 200 junk pages (so old pages, test cms pages etc that were live and still indexed). Redirects from any 404s resolved 19th November: Updated site maps and video site maps to reflect new content and remove old content. Reviewed the whole site for duplicate meta tags and titles and updated accordingly with unique ones. Fixed issues in Google Search Console for Google search console for 404 and Mobile usability. Removed embedded YouTube video from homepage. 11th December: Removed old content and content seen as not useful from indexing; 'honey pot' pages, old blog, map pages, user profile pages, project page ‘junk pages which have little SEO value’ (comments, contact project owner, backers, report project) from indexing, added ‘no-follow’ to widgets linking back to us 3rd January 2019: Changed the meta title from to remove 'example-term' (we were concerned it may have been seen as keyword stuffing) 7th January: Disavow file updated to refuse a set of external sites powered by API linking to us (these were sites like example-term.externalsite.co.uk which used to link to us showing projects in local areas - our SEO expert felt may be seen as a ‘link farm’) 11th January: Updated our ‘About us’ page with more relevant content 15th January: Changed homepage title to include 'example-term' again, footer links updated to point to internal pages rather than linking off to Intercom, homepage ordering of link elements on homepage changed (so moving external rating site link further down the page, removing underlines on one item that was not a link, fixed and instance where two h1 tags were used), removed another set of external Subdomains (i.e. https://externalsite.sitename.co.uk) from our system (these were old sites we used to run for different clients which has projects in geographical areas displayed) 18th January: Added the word 'example-term' to key content pages We're at a loss as to why we are still dropping. Please note that the above changes were implemented after we'd been ranking fine for a couple years on the 'example-term' - the changes were to try and address the drop in ranking. Any advice would be greatly appreciated.
White Hat / Black Hat SEO | | Nobody15554510997900 -
HELP!! We are losing search visibility fast and I don't know why?
We have recently moved from http to https - could this be a problem? https://www.thepresentfinder.co.uk As far as I'm aware we are doing everything by SEO best practice and have no manual penalties, all content is unique and we are not doing any link farming etc...
White Hat / Black Hat SEO | | The-Present-Finder0 -
Sitelinks Search Box impact for SEO
I am wondering how the relatively new sitelinks search box impacts the SEO rankings for a specific site or keyword combination - do you guys have any experience or bechmarks on this? Obviously it should help on getting more real estate on the SERP page (due to adding the search box), but do you also get extra goodwill and improved SERP position from adding it? Also, is the impact different on different type of terms, let's say single brand or category term such as "Bestbuy" (or "coupon") or a combination term "Bestbuy Apple" (or "Dixons coupon")? Thanks in advance!
White Hat / Black Hat SEO | | tjr0 -
Controlling crawl speed/delay through dynamic server-code and 503's
Lately i'm experiencing performance trouble caused by bot traffic. Although Googlebot is not the worst (it's mainly bingbot and ahrefsbot), they cause heavy server load from time to time. We run a lot of sites on one server, so heavy traffic on one site impacts other site's performance. Problem is that 1) I want a centrally managed solution for all sites (per site administration takes too much time), which 2) takes into account total server-load in stead of only 1 site's traffic and 3) controls overall bot-traffic in stead of controlling traffic for one bot. IMO user-traffic should always be prioritized higher than bot-traffic. I tried "Crawl-delay:" in robots.txt, but Googlebot doesn't support that. Although my custom CMS system has a solution to centrally manage Robots.txt for all sites at once, it is read by bots per site and per bot, so it doesn't solve 2) and 3). I also tried controlling crawl-speed through Google Webmaster Tools, which works, but again it only controls Googlebot (and not other bots) and is administered per site. No solution to all three of my problems. Now i came up with a custom-coded solution to dynamically serve 503 http status codes to a certain portion of the bot traffic. What traffic-portion for which bots can be dynamically (runtime) calculated from total server load at that certain moment. So if a bot makes too much requests within a certain period (or whatever other coded rule i'll invent), some requests will be answered with a 503 while others will get content and a 200. Remaining question is: Will dynamically serving 503's have a negative impact on SEO? OK, it will delay indexing speed/latency, but slow server-response-times do in fact have a negative impact on the ranking, which is even worse than indexing-latency. I'm curious about your expert's opinions...
White Hat / Black Hat SEO | | internetwerkNU1 -
Pagination for Search Results Pages: Noindex/Follow, Rel=Canonical, Ajax Best Option?
I have a site with paginated search result pages. What I've done is noindex/follow them and I've placed the rel=canonical tag on page2, page3, page4, etc pointing back to the main/first search result page. These paginated search result pages aren't visible to the user (since I'm not technically selling products, just providing different images to the user), and I've added a text link on the bottom of the first/main search result page that says "click here to load more" and once clicked, it automatically lists more images on the page (ajax). Is this a proper strategy? Also, for a site that does sell products, would simply noindexing/following the search results/paginated pages and placing the canonical tag on the paginated pages pointing back to the main search result page suffice? I would love feedback on if this is a proper method/strategy to keep Google happy. Side question - When the robots go through a page that is noindexed/followed, are they taking into consideration the text on those pages, page titles, meta tags, etc, or are they only worrying about the actual links within that page and passing link juice through them all?
White Hat / Black Hat SEO | | WebServiceConsulting.com0 -
Are Links from blogs with person using keyword anchor text a Penguin 2.0 issue?
Hello, I am continuing a complete clean up of a clients link profile and would like to know if Penguin is against links from blogs with the user including keywords as anchor text? So far I have been attempting to get them removed before I go for a disavow. An example would be the work clothing comment at the bottom of: http://www.fashionstyleyou.co.uk/beat-the-caffeine-rush.html/comment-page-1 I am also questioning if we should keep any link directories, so far I have been ruthless, but worry I will be losing a hell of a lot of links. For example I have kept the following: http://www.business-directory-uk.co.uk//clothing.htm Your comments are welcomed!
White Hat / Black Hat SEO | | MarzVentures0 -
404checker.com / crawl errors
I noticed a few strange crawl errors in a Google Webmaster Tools account - further investigation showed they're pages that don't exist linked from here: http://404checker.com/404-checker-log Basically that means anyone can enter a URL into the website and it'll get linked from that page, temporarily at least. As there are hundreds of links of varying quality - at the moment they range from a well known car manufacturer to a university, porn and various organ enlargement websites - could that have a detrimental effect on any websites linked? They are all nofollow. Why would they choose to list these URLs on their website? It has some useful tools and information but I don't see the point in the log page. I have used it myself to check HTTP statuses but may look elsewhere from now on.
White Hat / Black Hat SEO | | Alex-Harford0