Sitemaps, 404s and URL structure
-
Hi All!
I recently acquired a client and noticed in Search Console over 1300 404s, all starting around late October this year.
What's strange is that I can access the pages that are 404ing by cutting and pasting the URLs and via inbound links from other sites.
I suspect the issue might have something to do with Sitemaps. The site has 5 Sitemaps, generated by the Yoast plugin. 2 Sitemaps seem to be working (pages being indexed), 3 Sitemaps seem to be not working (pages have warnings, errors and nothing shows up as indexed). The pages listed in the 3 broken sitemaps seem to be the same pages giving 404 errors.
I'm wondering if auto URL structure might be the culprit here. For example, one sitemap that works is called newsletter-sitemap.xml, all the URLs listed follow the structure: http://example.com/newsletter/post-title
Whereas, one sitemap that doesn't work is called culture-event-sitemap.xml. Here the URLs underneath follow the structure http://example.com/post-title.
Could it be that these URLs are not being crawled / found because they don't follow the structure http://example.com/culture-event/post-title? If not, any other ideas?
Thank you for reading this long post and helping out a relatively new SEO!
-
Hi Daniel! Thanks for your question.
It's kind of hard to know what's going on without seeing your site. Feel free to PM it to me.
There's definitely a chance that this is the case, but if it's happening with Yoast it is likely a configuration issue on your site not with Yoast's technology. You may need to adjust your tag permalinks within your WordPress admin so that the URLs are correct in your sitemaps.
John
-
I'll make my question shorter and hopefully more clear...
If my Permalink structure in Wordpress is set up for a given custom post type, lets call it "culture", as: example.com/postname,
Yet with Yoast, a sitemap is automatically generated for posts tagged with "culture" that looks like example.com/culture/postname
Could that explain why posts being tagged as "culture" are showing up as 404s in Search Console?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Same URL names in one domain
Hi All, I have 9 different subdirectories for languages in the same domain example: www.example.com/page.html www.example.com/uk/page-uk.html www.example.com/es/page-es.html we are implementing hreflang tags for the languages. I know it is better to translate URLs, but we won't for now, because all the NON-ASCII characters. But we are thinking to get rid of the dashes on the languages URL: -uk or -es, so it will be: www.example.com/page.html www.example.com/uk/page.html www.example.com/es/page.hrml would this be a problem? to have same page names even if they are in different subdirectories? would we need to add canonical tags, at least for the main domain URLs? www.example.com/page.html Thank you, Rachel
Technical SEO | | RaquelSaiz0 -
"Url blocked by robots.txt." on my Video Sitemap
I'm getting a warning about "Url blocked by robots.txt." on my video sitemap - but just for youtube videos? Has anyone else encountered this issue, and how did you fix it if so?! Thanks, J
Technical SEO | | Critical_Mass0 -
Capitals URLs to Non Capitals...
Hi, I am working on a website which has capital urls and non capital urls which will be generating duplicate content, and I know it is better to use all lower case. The problem is that the page authority is better for the capital versions and I was wondering will it negatively impact the SEO of we 301 redirect the uppercase urls to the lowercase counterparts? Thanks.
Technical SEO | | J_Sinclair0 -
Making URLs automatically clickable
Hi all, I have a PHP function which i use to make all links clickable. Problem is, if some one writes a link in a a-tag, the URL inside the href value is made clickable. Not good. Can someone perhaps help me with the issue? Function: function makeClickableLinks($text)
Technical SEO | | rasmusbang
{
$text = preg_replace('/<//', ' $text = preg_replace('(
)', '
', $text);
$text = preg_replace('!((https?://www.|https?://|www.)(([a-z0-9-]+.)+[a-z]{2,6})(/\S+|/)*)!ie', '"[".shortenurl("\1")."]("".(strtolower('$2'))"', $text);
$text = str_replace('( <a href',="" '<a="" $text);<br="">$text = str_replace(')" target', '" target', $text);
$text = str_replace('):" target', '" target', $text);
$text = str_replace(')..." target', '" target', $text);
$text = str_replace(').." target', '" target', $text);
$text = str_replace(')." target', '" target', $text);
return $text;
}</a> <a href',="" '<a="" $text);<br="">Pleeeeease heeelp 🙂 Can't fix it on my own - been at it for ages. -Rasmus</a>0 -
Question/Concern about URL structure
Hey! I have some doubts concerning structuring a websites URL’s and what would be the best practise for this case. The site has 4 (main) categories with a maximum of 4 products in each category. For example: domain -> category (natural-stones) -> product (flooring) Which I would give the follow url: www.companysite.com//natural-stones/flooring Nothing odd so far, but here is the tricky part: the category isn’t an actual page a user wouldn’t be able to visit. The category is just an item in the mainmenu. If a user hovers over the category in the main menu they will get a dropdown in which they can select a product. E.g. flooring, wall strips etc. My question is: Is the url structure as I suggested: www.companysite.com//natural-stones/flooring the best practise. Even though the category isn’t an actually page. Or would it be better to structure the site: www.companysite.com/flooring My concern with this type of structure would be that the site would seem ‘flat’ with in-depth structure. Or would a third (and maybe best?) option be to create an actual page for the category itself. Thanks for taking the time to help me with my question/concern. If you need more information let me know.
Technical SEO | | RvG0 -
How to structure sub catergorys
due to out database each sub category has to be assigned to a category. i.e. /abc would have to be assigned to the category ''letters' in the url structure; so...it would be www.example.com/letters/abc . The question is does is matter which way round the category and sub are arranged? Assuming we are optimizing for ''abc' , does it matter if we structured as; www.example.com/letters/abc www.example.com/abc/letters Thanks in advance.
Technical SEO | | jg1000 -
Should me URLs be uppercase or lowercase
I'm in the middle of doing a bunch of 301 redirects for me site. Should I make them Lowercase, uppercase, or does it matter? Also, do I want to be using hyphens (-), or underscores (_)? Any other tips? EX: http://www.stupid.com/golf-slippers.html OR http://www.stupid.com/Golf-Slippers.html
Technical SEO | | JustinStupid0 -
Keywords in Vanity URL
If I set up a vanity URL that just 301's to the main site, do the search engines look at the keywords in the vanity URL when determing how to rank the site. For example, if I set up a vanity URL of www.coolnewtechgear.com, and redirect it to www.company.com/products/, would the search engines view the keywords of cool, new, tech, and gear and associate that with the page it's getting redirected to? Or does it ignore the vanity URL and only look at the content of the page itself?
Technical SEO | | ryanwats0