How get rid of 403 crawl error?

emrekeserr3

My wordpress website has 162 crawl 403 errors. Based on what I read it means that the server is denying crawlers to access the pages. The pages itself will load so guessing it's just an issue with crawlers only. How do I go about fixing this issue?

Sacha-Dodig

A "403" error indicates that access to the resource is forbidden for the crawler. This could be due to several reasons, and identifying the exact cause requires some investigation. Here are a few possibilities:

IP Address Blocking: The server may have automatically blocked the crawler's IP address if it detected too many requests in a short period, considering it a potential threat.
User-Agent Restriction: The server might be configured to deny access to certain User-Agents. Ensure your crawler's User-Agent is recognized and allowed by the server.
Incorrect Credentials: If the resource requires authentication, a "403" error can occur when the crawler fails to provide the correct credentials.
Robots.txt Rules: The website's robots.txt file might explicitly disallow crawling of the resource. Check if the crawler's access is restricted by this file.
Geo-Restrictions: Some websites restrict access to users from certain geographical locations. If your crawler's IP is from a blocked region, this could be the reason.
HTTP Referrer Header: Some servers check the HTTP referrer header for requests. If your crawler doesn't send this header or the value doesn't match the server's expectations, access might be denied.
To resolve this, you will need to identify which of these factors applies to your situation and adjust your crawler's configuration accordingly.

you can use a combination of tools and techniques to investigate the issue:

HTTP Header Analysis Tools: Tools like Redbot or HTTP Header Check can help you analyze the HTTP headers returned by the server. This can give clues about why a request might be blocked (e.g., due to User-Agent restrictions).

IP Reputation Checkers: Services like Talos Intelligence or MXToolbox can tell you if the crawler's IP address has been blacklisted or marked as suspicious.

Robots.txt Validator: Google's Robots Testing Tool (within Google Search Console) allows you to check if the site's robots.txt file is blocking your crawler.

User-Agent Tester: Tools like WhatIsMyBrowser.com can show you how websites perceive your crawler's User-Agent. This can help identify if the User-Agent might be causing issues.

VPN Services and Proxy Checkers: If you suspect geo-blocking, using VPN services to change your crawler's apparent location might help test this theory. Proxy checkers can also help you understand if your IP is considered part of a proxy network, which some sites block access to.

teknobird

@emrekeserr3 did you solve the problem?

webduh

TimKDM

Did you fix it?

Welcome to the Q&A Forum

Browse the forum for helpful insights and fresh discussions about all things SEO.

How get rid of 403 crawl error?

Got a burning SEO question?

Browse Questions

Explore more categories

Related Questions

Unsolved 403 crawl error

Unsolved Moz crawler not crawling on my site

Unsolved URL Crawl Reports providing drastic differences: Is there something wrong?

403 Errors Issue

Can't get my preferred URL, how much does it matter?

High Volume Duplicate Title and Content Errors: Scale of 1-10 How bad is this?

Why that many Crawl Diagnostics errors, false?

Crawl Diagnostics - Duplicate Content and Duplicate Page Title Errors