Issue with site not being properly found in Google

[Gavo]

We have a website [domain name removed] that is not being properly found in Google. When we run it through Screaming Frog, it indicates that there is a problem with the robot.txt file.

However, I am unsure exactly what this problem is, and why this site is no longer properly being found.

Any help here on how to resolve this would be appreciated!

[Christy-Correll]

Note: We've edited and removed select links and images in this thread as requested by the OP for privacy.

[Gavo]

Hi Thomas,

Thanks for all your help here. You've been fantastic!

We have had an issue generating a sitemap for our website using our usual sitemap creation tools. Could you explain why this is?

[BlueprintMarketing]

Moderator's Note: Attached images, along with select links in this thread have been edited and/or removed for privacy at the request of the OP.

--

I noticed your robots.txt is fixed but I would recommend two things to get your site back into the index faster based on the photographs below I am suggesting fetching your site as a Google bot as well as adding your XML site map to Webmaster tools.

Please do not forget to add all four versions of your website to webmaster tools if it has not been added

when I say that I mean add every URL below to Google Webmaster tools with and without www

target the site to the fourth or canonical URL. Choose the one with www.

1. [http://website.com]
2. [http://www.website.com]
3. [https://website.com]
4. [https://www.website.com]

here is a reference from Google

https://support.google.com/webmasters/answer/34592?hl=en&ref_topic=4564315

I would do two things I would add my site map to my robots.txt file because if you're going to use search tools it's going to help you.

You should set up your robots.txt just like this

Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

[Sitemap: https://www.website.com/sitemap_index.xml]

you can reference

https://yoast.com/ultimate-guide-robots-txt/

Allow directive

While not in the original “specification”, there was talk of an allow directive very early on. Most search engines seem to understand it, and it allows for simple, and very readable directives like this:

Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

The only other way of achieving the same result without an allowdirective would have been to specifically disallow every single file in the wp-admin folder.

because you don't want your login to be showing up in Google.

after which I would go into Webmaster tools/search console and fetch as a Google bot

Ask Google to re-crawl your URLs

If you’ve recently made changes to a URL on your site, you can update your web page in Google Search with the_Submit to Index_function of the Fetch as Google tool. Thisfunction allows you to ask Google to crawl and index your URL.

See

http://searchengineland.com/how-to-use-fetch-as-googlebot-like-seo-samurai-214292

https://support.google.com/webmasters/answer/6066468?hl=en

Ask Google to crawl and index your URL

1. Click Submit to Index, shown next the status of a recent, successful fetch in the Fetches Table.
2. Select** Crawl only this URL **to submit one individual URL to the Google for re-crawling. You can submit up to 500 individual URLs in this way within a 30 day period.
3. Select** Crawl this URL and its direct links** to submit the URL as well as all the other pages that URL links to for re-crawling. You can submit up to 10 of requests of this kind within a 30 day period.
4. Click Submit to let Google know that your request is ready to be processed.

adding your XML site map to Google Webmaster tools

[https://www.website.com/sitemap_index.xml]

will help Google determined that you are back online you should not see any real fallout from this. And submitting a complete XML site map gets a lot of images into Google images.

I hope this helps,

Tom

[BlueprintMarketing]

Hi it seems your robots.txt file is blocking Google and all other bots that search the web and obey robots.txt basically the good ones. So if you would like your site to be seen and indexed by Google and other search engines you need to remove the forward slash "/"

Shown here in your robots.txt file

Block all web crawlers from all content

User-agent: *  Disallow: /

 Go here to see [https://www.website.com/robots.txt]

• Please read https://moz.com/learn/seo/robotstxt

• Use to make the file http://tools.seobook.com/robots-txt/generator/

• Check it http://tools.seobook.com/robots-txt/analyzer/

 it looks like you're using WordPress so if you're using Apache or Yoast SEO you can go in and set it to

use this I added your xml  sitemap https://www.brightonpanelworks.com.au/sitemap_index.xml

Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

[Sitemap: https://www.website.com/sitemap_index.xml]

You can use tools like this to analyze & fix robots.txt & can allways see it by adding /robots.txt after the .com or tld.   

I hope that helps,

 Tom
```