Having issues crawling a website

[Gavo]

We looked to use the Screaming Frog Tool to crawl this website and get a list of all meta-titles from the site, however, it only resulted with the one result - the homepage.

We then sought to obtain a list of the URLs of the site by creating a sitemap using https://www.xml-sitemaps.com/. Once again however, we just go the one result - the homepage.

There is something that seems to be restricting these tools from crawling all pages. If you anyone can shed some light as to what this could be, we'd be most appreciative.

[OlegKorneitchouk]

That robots.txt should be fine.. its not blocking anything.

The reason the crawl is stopping on the homepage is this code:

<meta name="<a class="attribute-value">robots</a>" content="<a class="attribute-value">nofollow</a>">

Which tells bots to not follow any links on the page. Remove that and you should be good.

[Jvickery]

Hi,

I think it is your robots.txt file that is causing the issue. At the moment you have the following:

**User-agent: ***

Disallow:

I would recommend updating it to the following:

**User-agent: ***

Allow: /

Moz also has a good post about what else you can include in your robots.txt file for best practices etc. :

https://moz.com/learn/seo/robotstxt

Hope that helps

Thanks