How to fix and issue with robot.txt ?
-
I am receiving the following error message through webmaster tools
http://www.sourcemarketingdirect.com/: Googlebot can't access your site
Oct 26, 2012
Over the last 24 hours, Googlebot encountered 35 errors while attempting to access your robots.txt. To ensure that we didn't crawl any pages listed in that file, we postponed our crawl. Your site's overall robots.txt error rate is 100.0%.The site has dropped out of Google search.
-
Hi Stacey
What plugins do you have running - any caching plugins such the W3 Total Cache plugin?
Are you able to access your servers error logs to see if you can see anything there?
-
Thanks for your answer.
I have received this message from Google
**http://www.sourcemarketingdirect.com/ **using the Meta tag method (less than a minute ago). Your site's home page returns a status of 500 (Internal server error) instead of 200 (OK)
It looks like the permalink structure has changed but I'm not sure how.
-
I've seen several people ask this very same question over the last week in different forums. I am wondering if the major outages with hurricane Sandy have affected several hosts or DNS's.
Your robots.txt looks fine to me.
I'm guessing that you will completely recover once Google has a chance to fully crawl the site again.
-
just a quick check you have got wordpress visible to search engines set in the admin area? if not it will be set to disallow googlebot to crawl it.
it is in admin - options - privacy and select appropriate box - default is no index, no follow.
-
Thanks Matt.
There is no robots.txt as far as I can see. Is there a plugin I can use for wordpress?
The site was down for 2 days last month while hte original host transfered the site over to me.
Right now a site search says their are 13 pages indexed.
Just concerned that this site has always ranked number 1 for a company name search and now they are not on the first 10 pages in Google.
-
have you made sure your robots.txt is loading in your browser by adding robots.txt after your domain same as a normal page and can you see contents? has your site been down in this period? have you changed the contents of the file just before this issue? are you sure googlebot hasnt come back since that date - whats your analytics say? do an index site: search for your domain to see if it is in google.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Fixing Render Blocking Javascript and CSS in the Above-the-fold content
We don't have a responsive design site yet, and our mobile site is built through Dudamobile. I know it's not the best, but I'm trying to do whatever we can until we get around to redesigning it. Is there anything I can do about the following Page Speed Insight errors or are they just a function of using Dudamobile? Eliminate render-blocking JavaScript and CSS in above-the-fold content Your page has 3 blocking script resources and 5 blocking CSS resources. This causes a delay in rendering your page.None of the above-the-fold content on your page could be rendered without waiting for the following resources to load. Try to defer or asynchronously load blocking resources, or inline the critical portions of those resources directly in the HTML.Remove render-blocking JavaScript: http://ajax.googleapis.com/ajax/libs/jquery/2.1.1/jquery.min.js http://mobile.dudamobile.com/…ckage.min.js?version=2015-04-02T13:36:04 http://mobile.dudamobile.com/…pts/blogs.js?version=2015-04-02T13:36:04 Optimize CSS Delivery of the following: http://fonts.googleapis.com/…:400|Great+Vibes|Signika:400,300,600,700 http://mobile.dudamobile.com/…ont-pack.css?version=2015-04-02T13:36:04 http://mobile.dudamobile.com/…kage.min.css?version=2015-04-02T13:36:04 http://irp-cdn.multiscreensite.com/kempruge/files/kempruge_0.min.css?v=6 http://irp-cdn.multiscreensite.com/…mpruge/files/kempruge_home_0.min.css?v=6 Thanks for any tips, Ruben
Web Design | | KempRugeLawGroup0 -
Breadcrumbs show for some section and dont come for some. What could be the issue
Hello, We have implemented breadcrumbs - which comes up nicely for Blog and Forum in Google Search However, when we look at our 2 other section - New Car Product and Latest News - it does not comes up, is there any issue which can foresee Sample URL of New Car - http://www.mycarhelpline.com/index.php?option=com_newcar&view=product&Itemid=2&id=8&vid=188 Sample URL of News - http://www.mycarhelpline.com/index.php?option=com_latestnews&view=detail&n_id=474&Itemid=10 Its been over 1 month - since breadcrumbs been introduced in both New Car and News. is there any issue like in them as why is is not showing in search. Any recommendations Many Thanks !!
Web Design | | Modi0 -
My 404 page is showing a 4xx error. How can that be fixed?
My actual 404 page is giving a 4xx error.
Web Design | | sbetzen
The page address is http://www.ecowindchimes.com/v/404.asp It loads fine... it is the page all 404's are directed to. Why is it showing a 404 error. The page works. How can this be fixed? Stephen0 -
SEO Issues From Image Hotlinking?
I have a client who is hotlinking their images from one of their domains. I'm assuming the images were originally stored on the first domain (let's call it SiteA.com) and when they were putting together SiteB.com, they decided to just link to the images directly on SiteA.com instead of moving the images to Site B. Essentially hotlinking. Site A is not using the images in any way and in essence is just a gateway for their other sites and in this case a storage for their images. It doesn't use those images at all, so it really doesn't get any benefits of the images being referenced since I read that Google sometimes counts that hotlinking as a "vote" for the original image. But again, since ite A doesn't use the images that are being hotlinked at all, there's no benefit for Site A. My concern is that it's affecting their SEO for Site B because it makes it look like Site B is simply scraping data by hotlinking those images from Site A. Their programmer suggested creating a virtual directory so that it "looked" like it was coming from Site B. My guess is that Google can see this, so then not only will it look like Site B is scaping/hotlinking images, but also trying to hide it which may send up red flags to Google. My suggesstion to them was to just upload the images correctly into their own images directory on Site B. They own the images, so there's not any copyright issue, but that if they want proper SEO credit for that content, it all needs to be housed on the correct server and not hotlinked. Am I correct in this or will the virtual directory serve just as well?
Web Design | | GeorgiaSEOServices1 -
Duplicate content issue
I have recently built a site that has a main page intended to rank for national coverage. This site also has a number of pages targeted at local searches, these pages are slight variations of each other with town specific keywords. Does anyone know if google will see this as spam and quarantine my site from ranking? Thanks
Web Design | | stebutty0 -
Wordpress drop down menu issue...
While I am tweaking my current site with getting rid of my footer links, my site will not allow a drop down menu for my service area parent page. It will do a drop down menu for any other tab. Could not find anything in a google search besides just link to parent for drop down... Any ideas?
Web Design | | greenjoe0 -
Search directory - How to apply robots
Hi. On the site I'm working on, we use a search directory to display our search results. It displays as follows - Mydomain.com/search-results/# With the dynamic search results appearing after the hash tag. Because of the structure of the website, many of the lefthand nav defers back to this directory. I know that most websites "noindex, nofollow" the search results pages, but due to the ease of customers generating them, I'm afraid that if I do this, we'll miss out on the inevitable links customers will provide...and, even though it's just the main search directory, these links will still help my domain. The search is all java-generated so there's nothing for spiders to follow within this directory - save the standard category nav. How should I handle this? Thanks.
Web Design | | Blenny0 -
Landing Page/Home Page issues
Hi. I was speaking with my designer last night (we are setting up a new website) and we were discussing the design of our homepage, now the designer said he wanted the first page of the website to be a sort of landing page page were the visitor has to click and enter, im sure everyone has all come across these before. However, I am concerned as to the SEO implications of this? Any help guys?
Web Design | | CompleteOffice0