What is the appropriate Robot.txt to unblock if Google cannot get all the resources from my homepage?
-
Hello everyone. I did some research as to why my website has decreased in the Google search rankings recently. After reading this Yoast article I believe it's because the robot.txt files I have set up on my wordpress website. The following is a screen shot of the results of a fetch & render query of my webpage.Googlebot couldn't get all resources for this page. Here's a list:
My current robot.txt settings are as follows;
User-agent: * Disallow: /wp-admin/ Disallow: /wp-includes/ Disallow: */xmlrpc.php Disallow: */wp-*.php Disallow: */trackback/ Disallow: *?wptheme= Disallow: *?comments= Disallow: *?replytocom Disallow: */comment-page- Disallow: *?s= Disallow: */wp-content/ Allow: */wp-content/uploads/ ```What to I need to allow/disallow to allow Google spiders to properly read my website? -
Remove the following line:
Disallow: */wp-content/ And add: Allow: /wp-includes/js/
http://fonts.googleapis.com/css?family=Open+Sans:400,600,700,800%7CPT+Sans:400,400italic,700,700italic%7COswald:400,300,700&subset=latin,latin-ext
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Rel canonical on other page instead of duplicate page. How Google responds?
Hi all, We have 3 pages for same topics. We decided to use rel canonical and remove old pages from search to avoid duplicate content. Out of these 3 pages....1 and 2 type of pages have more similar content where 3 type don't have. Generally we must use rel canonical between 1 and 2. But I am wondering what happens if I canonical between 1 and 3 while 2 has more similar content? Will Google respects it or penalise as we left the most similar page and used other page for canonical. Thanks
Algorithm Updates | | vtmoz0 -
Does cached duplicate content hurts seo by Google
If we have duplicate content or pages cached in Google which has been indexed months back, still it hurts the original pages? Old URLs with cache can be seen now in Google when we search for the same URLs.
Algorithm Updates | | vtmoz0 -
Reviews - Google & Third Party
Hi We have reviews on our product pages & service reviews on Feefo, but how important is it to also drive customers to review your company on Google? I'm guessing we should be doing both, but it proves difficult when you already ask them to review your company through a third party? Any tips moz?
Algorithm Updates | | BeckyKey0 -
Anchor name URLs & anchor blocks: how Google sees them?
Hi guys, Anchor name URLs & anchor blocks: how Google sees them? As far as I know Google hasn't ever recommended anchor name URLs and anchor blocks, mostly when you have one page site, but I have ran into an organic result with an hyper-link to an anchor name URL. anchor name link There is a proper link and there aren't on the page and the code the words "Jump to". It means Google has put those words there and it has also taken the header of that block as anchor text. Why has Google placed that link? The query is "faqs umbrella company", so I thought that Google has seen "faqs umbrella company" like "what is the most popular faq about umbrella companies?" and therefore perhaps the correct answer could be "Is an umbrella company the only option I have? What are the alternatives?". Although, IMHO the most popular FAQ on Umbrella Companies should always be "what is an umbrella company". Unfortunately, that page is only worthy of third Google organic result page and there is no hint of rich snippet or any kind of conversational/KBT optimisation on its source code. no-rich-snippet Someone has any idea of why Google shows that link and if it's something that we can optimise in our pages? Cheers Pierpaolo IhwGwkb.jpg VWORt5F.jpg
Algorithm Updates | | madcow780 -
Has Google Authorship been completely removed from SERPs?
Noticed today that when I search (non-personalised search, incognito etc.) some of my pages on Google ALL references to authorship have now been completely removed. Does anyone know when this change occurred? I might be a bit slow this week (or last week) with concentrating on projects. I know like others that photos went some time back but now there are no author details being displayed. Just the page title and description. David
Algorithm Updates | | David-E-Carey1 -
Has anyone used Capterra and will I get penalized for paid links?
Hello - I'm contemplating buying a directory listing on the software promotion website http://www.capterra.com/ . It's a site that gets quite a bit of traffic for people searching for software products and I was interested in promoting my software product there, but I don't want to ruin our very good standing with Google at this time if Google deems Capterra as selling paid links. I'm not interested in this for links but instead as a good source of referral traffic for my software site. If anyone has used Capterra or has advice on whether Capterra might injure my SERP rankings, I would appreciate it. Thanks, Jeff
Algorithm Updates | | DenverDude0070 -
Google Page Rank?
We have had a quality website for 12 years now, and it seems no matter how many more links we get and how much new content we add daily, we have stayed at PR3 for the past 10 years or so. Our SEOMoz domain authority is 52. We have over 950,000 pages linking to us from 829 unique root domains. Is this in line with PR3 or should we be approaching PR4 soon? We do daily blog posts with all unique, fresh quality content that has not been published elsewhere. We try to do everything with 'white hat' methods, and we are constantly trying to provide genuine content and high quality products, and customer service. How can we improve our PR and how important is PR today?
Algorithm Updates | | applesofgold0 -
Google Directory vs DMOZ
What is the difference between the Google Directory and the DMOZ if any?
Algorithm Updates | | BrandonC-2698870