What is the appropriate Robot.txt to unblock if Google cannot get all the resources from my homepage?
-
Hello everyone. I did some research as to why my website has decreased in the Google search rankings recently. After reading this Yoast article I believe it's because the robot.txt files I have set up on my wordpress website. The following is a screen shot of the results of a fetch & render query of my webpage.Googlebot couldn't get all resources for this page. Here's a list:
My current robot.txt settings are as follows;
User-agent: * Disallow: /wp-admin/ Disallow: /wp-includes/ Disallow: */xmlrpc.php Disallow: */wp-*.php Disallow: */trackback/ Disallow: *?wptheme= Disallow: *?comments= Disallow: *?replytocom Disallow: */comment-page- Disallow: *?s= Disallow: */wp-content/ Allow: */wp-content/uploads/ ```What to I need to allow/disallow to allow Google spiders to properly read my website? -
Remove the following line:
Disallow: */wp-content/ And add: Allow: /wp-includes/js/
http://fonts.googleapis.com/css?family=Open+Sans:400,600,700,800%7CPT+Sans:400,400italic,700,700italic%7COswald:400,300,700&subset=latin,latin-ext
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Missing Keywords in Google SERP
We just got this attached image from one of our partners - has anyone seen Google putting 'missing' keywords in SERPs like this before? They said that it was not a plugin or anything and this is a screenshot of their organic search results. google%20screenshot_zpsgmwaf9e2.png
Algorithm Updates | | ReunionMarketing0 -
Why are organic search results vastly different on Bing, Google and Yahoo Search
We searched two words for a client so see how/where their site returned results. Depending on both the browser we used and the search engine, the results were so vastly different we were shocked. The site returned #2 or 3 on Bing and YahooSearch and not until the 3rd page for Google! And it also returned much worse on Chrome than any other browser, a Google product. I know this topic must be covered somewhere, or perhaps someone would be kind enough to chime in and shed some light? We have been working hard to optimize for Google and failing, but doing very well everywhere else. What gives?
Algorithm Updates | | jimmyzig1 -
How to get indexed in yahoo and bing?
Hello all I have uploaded sitemap to bing webmaster on January 21st, 2014. However, the site has not been indexed yet. I see few pages crawled and some crawl page error but it does not show what pages have an error. Can anyone help me please on how to get this done right so i can can our website indexed in Bing and Yahoo quickly. By the way our website address is : http://www.eduniche.com
Algorithm Updates | | Eva20140 -
Google cant read my robots.txt from past 10 days
http://awesomescreenshot.com/08d1s6aybc hi, my robots.txt is http://wallpaperzoo.com/robots.txt google says it cant read and has postponed the crawl.. its been 10 days and no crawl.. please help me in solving this issue.. this is save with http://hdwallpaperzones.com/robots.txt
Algorithm Updates | | toxicpls0 -
Frequency & Percentage of Content Change to get Google to Cache Every Day?
What is the frequency at which your homepage (for example) would have to update and what percentage of the page's content would need to be updated to get cached every day? What are your opinions on other factors.
Algorithm Updates | | bozzie3110 -
Site name appended to page title in google search
Hi there, I have a strange problem concerning how the search results for my site appears in Google. The site is Texaspoker.dk and for some strange reason that name is appended at the end of the page title when I search for it in Google. The site name is not added to the page titles on the site. If I search in Google.dk (the relevant search engine for the country I am targeting) for "Unibet Fast Poker" I get the following page title displayed in the search results: Unibet Fast Poker starter i dag - få €10 og prøv ... - Texaspoker.dk If you visit the actual page you can see that there is no site name added to the page title: http://www.texaspoker.dk/unibet-fast-poker It looks like it is only being appended to the pages that contains rich snippets markup and not he forum threads where the rich snippets for some reason doesn't work. If I do a search for "Afstemning: Foretrukne TOPS Events" the title appears as it should without the site name being added: Afstemning: Foretrukne TOPS Events Anybody have any experience regarding this or an idea to why this is happening? Maybe the rich snippets are automatically pulling the publisher name from my Google+ account... edited: It doesn't seem to have anything to do with rich snippets, if I search for "Billeder og stuff v.2" the site name is also appended and if I search for "bedste poker bonus" the site name is not.
Algorithm Updates | | MPO0 -
Google Update?
We have a website that for the past several weeks has been very consistent at between 13,500 and 14,200 daily visits and this site received 15,600 last Thursday. THIS week, Monday is at 22,200, Tuesday is at 26,200, and at mid-day today (at about our traffic halfway point in the day) we're already at 14,000 today. This was a site that was bringing about 14,000 visits as of May 16th last year and dropped to 11,000 the following week. The traffic to this site this week is so far beyond statistical analysis that there must have been something that happened.
Algorithm Updates | | sourcelinemedia0 -
Is it hurting my seo ranking if robots.txt is forbidden?
robots.txt is forbidden - I have read up on what the robots.txt file does and how to configure it but what about if it is not able to be accessed at all?
Algorithm Updates | | Assembla0
