What is the appropriate Robot.txt to unblock if Google cannot get all the resources from my homepage?
-
Hello everyone. I did some research as to why my website has decreased in the Google search rankings recently. After reading this Yoast article I believe it's because the robot.txt files I have set up on my wordpress website. The following is a screen shot of the results of a fetch & render query of my webpage.Googlebot couldn't get all resources for this page. Here's a list:
My current robot.txt settings are as follows;
User-agent: * Disallow: /wp-admin/ Disallow: /wp-includes/ Disallow: */xmlrpc.php Disallow: */wp-*.php Disallow: */trackback/ Disallow: *?wptheme= Disallow: *?comments= Disallow: *?replytocom Disallow: */comment-page- Disallow: *?s= Disallow: */wp-content/ Allow: */wp-content/uploads/ ```What to I need to allow/disallow to allow Google spiders to properly read my website? -
Remove the following line:
Disallow: */wp-content/ And add: Allow: /wp-includes/js/
http://fonts.googleapis.com/css?family=Open+Sans:400,600,700,800%7CPT+Sans:400,400italic,700,700italic%7COswald:400,300,700&subset=latin,latin-ext
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
What happens if we remove all the links to internal pages from our homepage?
Hi Moz community, We wanna give a try by removing all the links from homepage to internal pages and keep just a free trial button. Will this impact our SEO anyway? We have nearly 15 important internal pages at 2nd and 3rd hierarchy level. They may drop in rankings but we want to risk for few days to understand how it works. Your opinion please! Thanks
Algorithm Updates | | vtmoz0 -
Google creating it own content
I am based in Australia but a US founded search on 'sciatica' shows an awesome answer on the RHS of the SERP https://www.google.com/search?q=sciatica&oq=sciatica&aqs=chrome.0.69i59.3631j0j7&sourceid=chrome&ie=UTF-8 The download on sciatica is a pdf created by google. Firstly is this common in the US? secondly any inputs on where this is heading for rollout would be appreciated. Is google now creating its own content to publish?
Algorithm Updates | | ClaytonJ0 -
Why would google favour overseas retailers? Really weird results..
Why would google favour results from overseas retailers for queries in the UK? It's weird since most won't ship to the UK and the same products are found at dozens of UK retailers. It's not the case that the overseas sites are necessarily bigger brands or better SEO optimised, so having asked the leading agencies in the UK and them being stumped I was curious if this was something anyone else had seen? Our theory is that this can only be a poorly disguised attempt to drive Adwords.
Algorithm Updates | | predatornutrition0 -
Google Local Algorithm Changes?
I was wondering if you have heard about any Google Local algorithm changes. We have about 200 franchise locations. Some of our locations have dropped significantly over the past few weeks. Locations that were showing up in the 1-3 positions are now no longer showing on the first page. This is for very relevant phrases for our main line of business (which is also in our business name)... ‘Phrase, CITY NAME’. These locations have plenty of positive Google reviews. We would typically rank well for a phrase like that based on our relevance. I did some brainstorming. Do you think any of these could have any impact? Google is all about things looking and feeling natural including link building, etc. We have used Yext which made a lot of changes across the web to fix addresses, etc. Do you think Google may be seeing this as unnatural? Too many changes at to many sites in to short a period of time? Along those same lines, do you think Google may be penalizing some of our franchise pages for being to ‘perfect’? It would be ‘natural’ for addresses to have some difference across the web and a bit unnatural to have them all match so perfectly. I know that Google has always stated the business name should be listed in Google Local the way it is listed to the general public. Things such as “Business Name Boston” should be listed as “Business Name”. Each of our franchise locations is named in house to reflect their geo location..... "Business Name Boston", "Business Name St. Louis". Many of our competitors also use the practice of attaching geo terms as well. Do you think we may be getting hit with a penalty now even though we have listed things on Google with the Geo term for years.... and is how WE refer to each location? Is it possible that by working with Yext, we drew attention to this practice? Should we remove our local listings geo term on Google Local? How about across the web? We are in a business that does not require customers to come to our location. Some of our locations have not suppressed the address in their local listings while others have. Many of our competitors have not. Do you think this could play into it? Some of our locations that are not showing in Local have good organic results. Have you heard anything about Google dropping Local if they show in organic? I know Google has been looking at social media more and more and I believe they will continue to do so. If our local pages have no social presence, could this adversely affect things? (I think this is probably not the case…. but wanted to throw it out there) I have noticed that in some cases where Local has dropped, we have multiple offices in that metro area. Is it possible that this could affect things? Have you heard of any Local algorithm changes? I know they are releasing a new dashboard sporadically, could this be in conjunction with a larger Local algorithm change? Our CMS tool does not allow us to change Title/Meta per page (I know... terrible!!). So every page has the same title and same meta description. (We are changing our CMS system! Can't wait!). Could this play into it? Thanks for any feedback!
Algorithm Updates | | MABES1 -
URL is starting to appear capitalized in Google Search Results. How come?
Our domain (www.absoluteautomation.com) has just today started appearing in search results as www.AbsoluteAutomation.com. Any ideas why?
Algorithm Updates | | absoauto0 -
Urgent input needed on huge drop in Google
As of today we got huge drops in SERP across all our pages. We can see a drop between 10 to 80% on most of our pages on this domain: http://www.meresverige.dk Some background info: Never bought any links Yes, did optimize the site, but only in fair way, using SEO moz On-Page Optimization. Most pages get an A-grade No cloaking, all pages do look exactly same to visitors and Google Any input on what this could be? We are hugely grateful for any input that might lead us in the correct direction Have a nice day Fredrik
Algorithm Updates | | Resultify0 -
Google Update?
We have a website that for the past several weeks has been very consistent at between 13,500 and 14,200 daily visits and this site received 15,600 last Thursday. THIS week, Monday is at 22,200, Tuesday is at 26,200, and at mid-day today (at about our traffic halfway point in the day) we're already at 14,000 today. This was a site that was bringing about 14,000 visits as of May 16th last year and dropped to 11,000 the following week. The traffic to this site this week is so far beyond statistical analysis that there must have been something that happened.
Algorithm Updates | | sourcelinemedia0 -
Google's reaction to site updates
Hi, Is it safe to assume as soon as Google indexes updates I've made to my site that any ranking changes the updates effected will happen at that same time, or is there ever a lag time before these changes ( if any ) take effect?
Algorithm Updates | | minutiae0