Why xml generator is not detecting all my urls?
-
Hi Mozzers,
After adding 3 new pages to example.com, when generating the xml sitemap, Iwasn't able to locate those 3 new url. This is the first time it is happening. I have checked the meta tags of these pages and they are fine. No meta robots setup!
Any thoughts or idea why this is happening? how to fix this?
Thanks!
-
Thank you Guys!
I resolved the issue! I think it was a caching issue!
-
Like Matt says, could it be that there is a robots.txt file that could block these URLs? Could it be that your XML sitemap generator is using cached results for your site. Can the pages be found on the page via links on other pages?
-
Are they linked to a page that the crawler can find?
What are you generating it with? Online tools like xml-sitemaps.com act like a spider - if they can't crawl from the homepage to the new page, they simply won't find it.
Internal tools, like a Wordpress plugin, should actually find those orphan pages.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Submitted URL has crawl issue - Submitted URL seems to be a Soft 404 - but all looks fine
Google Search Console is showing some pages up as "Submitted URL has crawl issue" but they look fine to me. I have set them as fixed but after a month they were finally re-crawled and google states the issue persists. Examples are: https://www.rscpp.co.uk/counselling/175809/psychology-alcester-lanes-end.html
Technical SEO | | TommyNewmanCEO
https://www.rscpp.co.uk/browse/location-index/889/index-of-therapy-in-hanger-lane.html
https://www.rscpp.co.uk/counselling/274646/psychology-waltham-forest-sexual-problems.html There's also some "Submitted URL seems to be a Soft 404": https://www.rscpp.co.uk/counselling/112585/counselling-moseley-depression.html I also have more which are "pending", but again I couldn't see a problem with them in the first place. I'm at a bit of a loss as to what to do next. Any advice? Thanks in advance.0 -
Should I change by URL's
I started with a static website and then moved to Wordpress. At the time I had a few hundred pages and wanted to keep the same URL structure so I use a plugin that adds .html to every page. Should I change the structure to a more common URL structure and do 301 directs from the .html page to the regular page?
Technical SEO | | JillB20130 -
How to handle temporary campaign URLs
Hi, We have just run a yearly returning commercial campaign for which we have created optimized URL's. (e.g. www.domain.tld/campaign including the category and brand names after the campaign www.domain.tld./campaign/womens This has resulted in 4500+ URL's being indexed in Google including the campaign name, now the campaign is over and these URL's do not exist anymore. How should we handle those URL's? 1.) 301 them to the correct category without the campaign name 2.) Create a static page www.domain.tld/campaign to which we 301 all URL's that have the campaign name in them Do you have any other suggestions on what the best approach would be? This is a yearly commercial campaign so in a year time we will have the same URL's again. Thanks, Chris
Technical SEO | | eCommerceSEO0 -
What is your opinion on ideal url structure?
Which url structure do you think is better... website.com/CO/denver/555-your-address-way-denver-co-55678/98347578 or website.com/classifying-keyword/555_your_address_way-denver_colorado-55678-98347578 orrr website.com/classifying-keyword/555-your-address-way-denver-colorado-55678-98347578
Technical SEO | | jessefriedman0 -
URL specific websites with i-framed application
I have 300 URL specific websites that rank well in Yahoo and Bing. Unfortunately I don't have access to the websites due to a previous marketing agreement (before my time). I do have access to the application that is i-framed into the websites. I was thinking about adding a paragraph below the application with a link to the primary website. How does google look at these links. If I add the link, there will be an additional 300 links showing up at the same time. Not what they want to see from my personal knowledge base. At the same time, its not black hat SEO, I am just trying to link to the other websites which I own which are related. What are people thoughts.
Technical SEO | | FidelityOne0 -
Canonical Tag Pointing To The Same URL
Does it matter if a canonical tag points to the URL in which the tag is on? Example Page: http://www.domain.com Canonical tag: rel="canonical" href="http://www.domain.com" /> I only ask because a client of mine has a CMS that automatically does that to every page on the site and there's no way to remove it. Will this have a negative impact or does it not matter at all? Any insights would be great because I can't find a clear answer anywhere online. Thanks!
Technical SEO | | MichaelWeisbaum0 -
No Sub-Categories in XML Sitemap
I have a couple of sites using 3dcart, the ecommerce platform. Their tech support recently told me that they do not list sub-categories in the XML sitemap, only products and top-tier categories. Am I the only one that sees a problem with this? Thanks
Technical SEO | | poolguy0 -
Best Dynamic Sitemap Generator
Hello Mozers, Could you please share the best Dynamic Sitemap Generator you are using. I have found this place: http://www.seotools.kreationstudio.com/xml-sitemap-generator/free_dynamic_xml_sitemap_generator.php Thanks in advanced for your help.
Technical SEO | | SEOPractices0