Moz Q&A is closed.
After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.
Some URLs were not accessible to Googlebot due to an HTTP status error.
-
Hello I'm a seo newbie and some help from the community here would be greatly appreciated.
I have submitted the sitemap of my website in google webmasters tools and now I got this warning:
"When we tested a sample of the URLs from your Sitemap, we found that some URLs were not accessible to Googlebot due to an HTTP status error. All accessible URLs will still be submitted."
How do I fix this? What should I do?
Many thanks in advance.
-
You need to confirm that the URLs are in fact 100% of your URLs going into the site map are accessible.
if it's a big issue in a big site send me the URL in a private message I will use deep crawl to create a XML sitemap for you. The screaming frog tool is excellent as well though does performance well with extremely large sites.
check your robots.txt file this so great tool if in case you have more than one (it happens)
http://www.internetmarketingninjas.com/seo-tools/robots-txt-generator/
or
http://tools.seochat.com/tools/robots-txt-validator/
so many great free tools are found right here http://tools.seochat.com/tools/
It could be a number of things although it could be Google being finicky. Run the site through Moz crawler, use feedthebot.com using "tools SEO" or download the free version of http://www.screamingfrog.co.uk/seo-spider/ this will tell you if there is an issue. If your site is static you can even create an alternate site map with screaming frog if your site is large use deep crawl or Moz analytics
be certain there are no sitemaps redirecting to each other so no redirects going from the old site map to the new site map. Make certain that the site map is in an XML format e.g. http://example.com/sitemap.xml or if in a different format like https://example.com/sitemap_index.xml make sure the proper format That resolves when you look at the site map is what is going into Webmaster tools. Be certain the site map does not contain over 500 URLs per the site map so example.com/sitemap1.xml and so on keep numbering them appropriately. sometimes Google is overloaded and does not seem to like to play well with certain site maps or the site map is maybe not generating very well on the server and that is fixed later on. If this is a long-term problem speak to your host or developer. My recommendation is if you've done everything I have talked about that you attempt to submit is the sitemap to to Webmaster tools or simply build a new sitemap and submit that.
so if worse comes to worse take the screaming frog and use this URL to send it to Google
http://www.google.com/submityourcontent/business-owner/
I hope that helps,
Thomas
-
Hi, It looks like you have url's placed in your sitemap that have an HTTP status error. You can search for the urls and remove them from your sitemap or make sure they have the right status. Does it say which status error? And does it say which url's? Did you check those url's?When you use Screaming frog spider tool (free), you can search for status error's this is an easy way to find these url's.
Grtz, Leonie
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Subdomain 403 error
Hi Everyone, A crawler from our SEO tool detects a 403 error from a link from our main domain to a a couple of subdomains. However, these subdomains are perfect accessibly. What could be the problem? Is this error caused by the server, the crawlbot or something else? I would love to hear your thoughts.
Technical SEO | | WeAreDigital_BE
Jens0 -
Http to https redirection issue
Hi, i have a website with http but now i moved to https. when i apply 301 redirection from http to https & check in semrush it shows unable to connect with https & similar other tool shows & when i remove redirection all other tools working fine but my https version doesn't get indexed in google. can anybosy help what could be the issue?
Technical SEO | | dhananjay.kumar10 -
Tool to Generate All the URLs on a Domain
Hi all, I've been using xml-sitemaps.com for a while to generate a list of all the URLs that exist on a domain. However, this tool only works for websites with under 500 URLs on a domain. The paid tool doesn't offer what we are looking for either. I'm hoping someone can help with a recommendation. We're looking for a tool that can: Crawl, and list, all the indexed URLs on a domain, including .pdf and .doc files (ideally in a .xls or .txt file) Crawl multiple domains with unlimited URLs (we have 5 websites with 500+ URLs on them) Seems pretty simple, but we haven't been able to find something that isn't tailored toward management of a single domain or that can crawl a huge volume of content.
Technical SEO | | timfrick0 -
Which Sitemap to keep - Http or https (or both)
Hi, Just finished upgrading my site to the ssl version (like so many other webmasters now that it may be a ranking factor). FIxed all links, CDN links are now secure, etc and 301 Redirected all pages from http to https. Changed property in Google Analytics from http to https and added https version in Webmaster Tools. So far, so good. Now the question is should I add the https version of the sitemap in the new HTTPS site in webmasters or retain the existing http one? Ideally switching over completely to https version by adding a new sitemap would make more sense as the http version of the sitemap would anyways now be re-directed to HTTPS. But the last thing i can is to get penalized for duplicate content. Could you please suggest as I am still a rookie in this department. If I should add the https sitemap version in the new site, should i delete the old http one or no harm retaining it.
Technical SEO | | ashishb010 -
Trailing Slashes In Url use Canonical Url or 301 Redirect?
I was thinking of using 301 redirects for trailing slahes to no trailing slashes for my urls. EG: www.url.com/page1/ 301 redirect to www.url.com/page1 Already got a redirect for non-www to www already. Just wondering in my case would it be best to continue using htacces for the trailing slash redirect or just go with Canonical URLs?
Technical SEO | | upick-1623910 -
Urls with or without .html ending
Hello, Can anyone show me some authority info on wheher links are better with or without a .html ending? Thanks is advance
Technical SEO | | sesertin0 -
Products with discrete URLs for each color
here is the issue. i have an ecommerce site that on a category page, shows each individual color for each product sold. and there is a distinct URL for each color. each product page shares the same content, with the only potentially differentiating factor being customer reviews (not nearly enough of these to differentiate anything). so we have URLs like: www.domain.com/product-green www.domain.com/product-yellow www.domain.com/product-red and so on. i am looking for a way to consolidate these URL while still showing all colors on the category page. the first solution i am considering is using the hash tag. so we would create www.domain.com/product#green, www.domain.com/product#yellow, www.domain.com/product#red. if possible, i would set the canonical tag as www.domain.com/product. the second solution would be to use the canonical tag and keep the URLs as is. the issue i see here is that we would need to create www.domain.com/product and show that page somewhere. www.domain.com/product would the URL that the above color URLs would canonicalize to. what would be the preferred solution? or is there something else?
Technical SEO | | rakesh_patel0 -
Use of + in url good or bad?
Hi, I am working on a SEO project for a client.
Technical SEO | | MaartenvandenBos
Some of the urls have a + between the keyword.
like www.example.com/make+me+happy/ Is this good or bad for seo?
Or is it maybe better to use - ? Thanks!0