Tracking a Crawl error
-
Hi All,
If you find a crawl error on your page. How do you find it?
The error only says the URL that is wrong but this is not the location. Can i drill down and find out more information?
Thank you!
-
Hi Martin,
Thanks for coming back to me.
You are spot on... I think I have been sitting at the desk to long today. Zoned out!
Yep, webtools in google shows the URL in full so I can find it now.
Thanks for your help
-
Hi Wayne, Just to clarify you are having issues with a crawl error in Webmaster tools, and when you click the url it works fine? You can use tools such as Google Webmaster tools > Fetch as Google bot and input the url to see how Google Bot see's it in HTML. You can also download a text based browser such as Lynx to view your website very similar to how Google bot see's it. It will be very hard to determine a Crawl Path, as if that information was available we could develop trends and get a little closer to their algorithm. Crawler trackers can, on occasion cause hinderence to the crawler.Hope this helps.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Google crawl drop
the crawl request of my company site: https://www.dhgate.com/ has dropped nearly over 95%, from daily 6463599 requests to 476493 requests at 12:00am on 9th, Oct (GMT+8). This dramatic dropping trend not only showed in our GSC crawl stats report but also our company's own log report. We have no idea what’s going on. We want to know whether there is an update of google about crawlling, or is this the issue of our own site? If something is wrong with our site, in what aspects would you recommend us to check, analyze and accordingly optimize?
Technical SEO | | DHgate_20140 -
How to allow bots to crawl all but WP-content
Hello, I would like my website to remain crawlable to bots, but to block my wp content and media. Does the following robots.txt work? I worry that the * user agent may conflict with the others. User-agent: *
Technical SEO | | Tom3_15
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /wp-content/ User-agent: GoogleBot
Allow: / User-agent: GoogleBot-Mobile
Allow: / User-agent: GoogleBot-Image
Allow: / User-agent: Bingbot
Allow: / User-agent: Slurp
Allow: /0 -
Google crawl rate dropped after we activated CloudFront
Hello! Previously we've been using Amazon CloudFront for our static content (js, css etc). But to be able to reduce load on our origin servers and to be able to give our international users a good user experience we decided to deliver a couple of our sites through CloudFront. We noticed very nice drops in page load time, but when checking Google webmaster tools we noticed that all CloudFront-activated sites got a huge drop in pages crawled per day (from avg ~3500 to ~150). Also one of the sites have issues with the Google sitemaps (just marked as "Pending" in GWT) and no new pages or updated pages seems to be updated in the Google SERP. The rest of the sites gets some updates on the Google SERP, but very few compared to before CloudFront activation. Is there anybody here who have experience in full site delivery through CloudFront (or other CDNs) and effects on SEO/Google? Would be very glad for any insights or suggestions. The risk is that we need to remove CloudFront if this just continues.
Technical SEO | | Ludde0 -
Error in how URLs were set up, how can it be fixed?
Hi, I managed a website port to a WP responsive deisgn for a client, see http://chicagotelephony.com. Unfortunately, he wanted me to work with a graphic designer rather than a web geek, so the resulting website has messed up URLS, i.e. index.php is smack in the middle of almost all the pages. I know that is all wrong, but I also realized that she was not fluent in the way the Genesis framework was set up or how the particular template I selected, operated. So I just wanted to get it out there.... and now it is live, but has all these errors. Do I have to do 301 redirects? Is there a setting or a button inside of the WP template that would put correct slugs but get rid of the index.pho within the URL? For example, http://chicagotelephony.com/index.php/cloud-based-solutions/ and http://chicagotelephony.com/index.php/var-network-value-added-reseller/ should be chicagotelephony.com/cloud-based-solutions/ and chicagotelephony.com/var-netowrk-value-added-reselle/ and so forth.
Technical SEO | | DianeDP0 -
What could be the cause of this duplicate content error?
I only have one index.htm and I'm seeing a duplicate content error. What could be causing this? IUJvfZE.png
Technical SEO | | ScottMcPherson1 -
How do you fix soft 404 errors?
I'm getting soft 404 errors for a gallery on my site: http://chrle.us/MBKs yet when I visit the url: http://www.chrisboar.com/html/galleries/114/portfolio/weddings-1/0 the image/url is there. Not sure how to fix this. note we do redirect the site to the flash site and only use those URLs for SEO purposes. Maybe that is what is causing it. thanks
Technical SEO | | callmeed0 -
Hiring someone to assist us in fixing SEOMOZ Errors
Greetings. We have been using SEOMOZ for about 9 months and we are needing to hire someone to assist us in fixing ERRORS promulgated by our SEOMOZ weekly crawl. Does anyone know of any person or firm that can assist us with this?
Technical SEO | | GreatFence0 -
Google Webmaster tools error?
So I am trying to set the URL preference in google webmaster tools for my site. However when I try to save it it tells me to verify that I own the site. I have already done this so where can I go to verify I own the site exactly? Maybe I am wrong and I have not done this already but even on the homepage of webmaster tools I don't see an option to "verify".
Technical SEO | | ENSO0