Issue with site not being properly found in Google
-
We have a website [domain name removed] that is not being properly found in Google. When we run it through Screaming Frog, it indicates that there is a problem with the robot.txt file.
However, I am unsure exactly what this problem is, and why this site is no longer properly being found.
Any help here on how to resolve this would be appreciated!
-
Note: We've edited and removed select links and images in this thread as requested by the OP for privacy.
-
Hi Thomas,
Thanks for all your help here. You've been fantastic!
We have had an issue generating a sitemap for our website using our usual sitemap creation tools. Could you explain why this is?
-
Moderator's Note: Attached images, along with select links in this thread have been edited and/or removed for privacy at the request of the OP.
--
I noticed your robots.txt is fixed but I would recommend two things to get your site back into the index faster based on the photographs below I am suggesting fetching your site as a Google bot as well as adding your XML site map to Webmaster tools.
Please do not forget to add all four versions of your website to webmaster tools if it has not been added
when I say that I mean add every URL below to Google Webmaster tools with and without www
target the site to the fourth or canonical URL. Choose the one with www.
here is a reference from Google
https://support.google.com/webmasters/answer/34592?hl=en&ref_topic=4564315
I would do two things I would add my site map to my robots.txt file because if you're going to use search tools it's going to help you.
You should set up your robots.txt just like this
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php[Sitemap: https://www.website.com/sitemap_index.xml]
you can reference
https://yoast.com/ultimate-guide-robots-txt/
Allow
directiveWhile not in the original “specification”, there was talk of an
allow
directive very early on. Most search engines seem to understand it, and it allows for simple, and very readable directives like this:Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php
The only other way of achieving the same result without an
allow
directive would have been to specificallydisallow
every single file in thewp-admin
folder.because you don't want your login to be showing up in Google.
after which I would go into Webmaster tools/search console and fetch as a Google bot
Ask Google to re-crawl your URLs
If you’ve recently made changes to a URL on your site, you can update your web page in Google Search with the_Submit to Index_function of the Fetch as Google tool. Thisfunction allows you to ask Google to crawl and index your URL.
See
http://searchengineland.com/how-to-use-fetch-as-googlebot-like-seo-samurai-214292
https://support.google.com/webmasters/answer/6066468?hl=en
Ask Google to crawl and index your URL
- Click Submit to Index, shown next the status of a recent, successful fetch in the Fetches Table.
- Select** Crawl only this URL **to submit one individual URL to the Google for re-crawling. You can submit up to 500 individual URLs in this way within a 30 day period.
- Select** Crawl this URL and its direct links** to submit the URL as well as all the other pages that URL links to for re-crawling. You can submit up to 10 of requests of this kind within a 30 day period.
- Click Submit to let Google know that your request is ready to be processed.
adding your XML site map to Google Webmaster tools
[https://www.website.com/sitemap_index.xml]
will help Google determined that you are back online you should not see any real fallout from this. And submitting a complete XML site map gets a lot of images into Google images.
I hope this helps,
Tom
-
Hi it seems your robots.txt file is blocking Google and all other bots that search the web and obey robots.txt basically the good ones. So if you would like your site to be seen and indexed by Google and other search engines you need to remove the forward slash "/"
Shown here in your robots.txt file
Block all web crawlers from all content
User-agent: * Disallow: /
Go here to see [
https://www.website.com/robots.txt]-
Please read https://moz.com/learn/seo/robotstxt
-
Use to make the file http://tools.seobook.com/robots-txt/generator/
it looks like you're using WordPress so if you're using Apache or Yoast SEO you can go in and set it to use this I added your xml sitemap https://www.brightonpanelworks.com.au/sitemap_index.xml
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php[Sitemap: https://www.website.com/sitemap_index.xml]
You can use tools like this to analyze & fix robots.txt & can allways see it by adding /robots.txt after the .com or tld.
I hope that helps,
Tom ```
-
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Mirror from my site
hi to all i find 2 site they do mirror my site and send back link to all my pages. do you thing its bad for my seo ?? my site is https://android-apk.org mirror sites | Who links the most | fryeboysent.com | 1,342,613 |
Intermediate & Advanced SEO | | moztabliq1
| ficyexp.cl | 934,654 | |0 -
Moving multiple Sites to One Site and SEO Impact/Ideas
Hi there, We are in the process of moving 2 sites with higher page authority to another site we own (that is our company brand), so essentially 3 sites into one. We're at risk of losing a lot of SEO from the original 2 sites that have all the product information. We are doing this since we merged companies a couple years back and need one web precense. Anyhow, the site launch date is in 3 months and the recommendation is to start moving content over prior to that for top pages, which is a big undertaking when we are launching all the pages again with new content, redeisgn and moving sites in 3 months. If it's the right move, we should do it, but I just wanted to get opinions on how others have handled something similiar when moving to a site with lower site authority and trying not to lose rankings.
Intermediate & Advanced SEO | | lauramrobinson320 -
Some site's links look different on google search. For example Games.com › Flash games › Decoration games How can we do our url's like this?
For example Games.com › Flash games › Decoration games How can we do our url's like this?
Intermediate & Advanced SEO | | lutfigunduz0 -
Why is my m-dot site outranking my main site in SERPs?
My client has a WP site and a Duda mobile site that we inherited. For some reason their m-dot site is ranking on P1 of Google for their top KWs instead of the main site which is much more robust. The main site might rank beyond page 5 when the generic home page for their m-dot site appears on P1. Does anyone have any idea why this might be happening?
Intermediate & Advanced SEO | | Etna0 -
Google and JavaScript
Hey there! Recent announcements at Google to encourage webmasters to let Google crawl Java Script http://www.googlewebmastercentral.blogspot.com/2014/05/understanding-web-pages-better.html http://googlewebmastercentral.blogspot.com/2014/05/rendering-pages-with-fetch-as-google.html We have always put JS and CSS behind robots.txt, but now considering taking them out of robots. Any opinions on this?
Intermediate & Advanced SEO | | CleverPhD0 -
Why has my site dropped off the face of Google???
Hi All, My site was ranking well for a long time and suddenly can't be seen at all any more. I have been trying to figure this out for some time and can't get to the bottom of it. Funny thing is also even when searching for my site with a keyword "snowboard gulmarg" my URL www.klinehinmalaya.com does not appear and somewhere way back in the listing my another page www.klinehimalaya.com/packages.php comes up. Any help would be good right now. Thanks in advance, Catherine.
Intermediate & Advanced SEO | | caherinechan0 -
Google bot vs google mobile bot
Hi everyone 🙂 I seriously hope you can come up with an idea to a solution for the problem below, cause I am kinda stuck 😕 Situation: A client of mine has a webshop located on a hosted server. The shop is made in a closed CMS, meaning that I have very limited options for changing the code. Limited access to pagehead and can within the CMS only use JavaScript and HTML. The only place I have access to a server-side language is in the root where a Defualt.asp file redirects the visitor to a specific folder where the webshop is located. The webshop have 2 "languages"/store views. One for normal browsers and google-bot and one for mobile browsers and google-mobile-bot.In the default.asp (asp classic). I do a test for user agent and redirect the user to one domain or the mobile, sub-domain. All good right? unfortunately not. Now we arrive at the core of the problem. Since the mobile shop was added on a later date, Google already had most of the pages from the shop in it's index. and apparently uses them as entrance pages to crawl the site with the mobile bot. Hence it never sees the default.asp (or outright ignores it).. and this causes as you might have guessed a huge pile of "Dub-content" Normally you would just place some user-agent detection in the page head and either throw Google a 301 or a rel-canon. But since I only have access to JavaScript and html in the page head, this cannot be done. I'm kinda running out of options quickly, so if anyone has an idea as to how the BEEP! I get Google to index the right domains for the right devices, please feel free to comment. 🙂 Any and all ideas are more then welcome.
Intermediate & Advanced SEO | | ReneReinholdt0 -
On-Site Optimization Tips for Job site?
I am working on a job site that only ranks well for the homepage with very low ranking internal pages. My job pages do not rank what so ever and are database driven and often times turn to 404 pages after the job has been filled. The job pages have to no content either. Anybody have any technical on-site recommendations for a job site I am working on especially regarding my internal pages? (Cross Country Allied.com) Any help would be much appreciated.
Intermediate & Advanced SEO | | Melia0