Why xml generator is not detecting all my urls?
-
Hi Mozzers,
After adding 3 new pages to example.com, when generating the xml sitemap, Iwasn't able to locate those 3 new url. This is the first time it is happening. I have checked the meta tags of these pages and they are fine. No meta robots setup!
Any thoughts or idea why this is happening? how to fix this?
Thanks!
-
Thank you Guys!
I resolved the issue! I think it was a caching issue!
-
Like Matt says, could it be that there is a robots.txt file that could block these URLs? Could it be that your XML sitemap generator is using cached results for your site. Can the pages be found on the page via links on other pages?
-
Are they linked to a page that the crawler can find?
What are you generating it with? Online tools like xml-sitemaps.com act like a spider - if they can't crawl from the homepage to the new page, they simply won't find it.
Internal tools, like a Wordpress plugin, should actually find those orphan pages.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Redirecting a Few URLS from One Domain to Another
Hello, I have two websites within a similar niche...some of the top organic traffic driving pages on Website B I'd like to redirect to a similar page on Website A. The reason is Website A is a bigger and better and is monetized much better as well. I only want to redirect a few of the main URLS on Website A and also only those that I have similar content on my main Website B. Is this process safe for SEO? What is the best way to go about this process. I am not really concerned with Website B and what happens to it's rankings, but in the meantime, I'd like to redirect the traffic from some of it's main organic traffic driving pages to my main website A and to it's similar pages. I am also concerned with making sure my main website A stays white hat and doesn't receive any negativity from these redirects. Thanks.
Technical SEO | | juicyresults0 -
URL Structure Question
We are building a job board website that will have a decent amount of "career resources" type content and want to make sure we set up our url structure correctly. After researching on Google and here I have an idea how to structure it but would like some insight if we are on the right track. We are using Wordpress for the content part of our website. We will have about 5 content categories (like resume-tips, job-interviews, job-search etc.) The two options we are considering; www.domain.com/career-resources/index.html As content start page www.domain.com/career-resources/resume-tips/index.html category start page www.domain.com/career-resources/resume-tips/top-5-resume-mistakes.html article name is the /career-resources/ folder really needed or can we go something like; www.domain.com/career-resources/index.html As content start page www.domain.com/resume-tips/index.html category start page www.domain.com/resume-tips/top-5-resume-mistakes.html article name Are we on the right track... and is one way better for SEO that the other? Thanks! Shaun
Technical SEO | | aactive0 -
URL removals
Hello there, I found out that some pages of the site have two different URL's pointing at the same page generating duplicate content, title and description. Is there a way to block one of them? cheers
Technical SEO | | PremioOscar0 -
Making URLs automatically clickable
Hi all, I have a PHP function which i use to make all links clickable. Problem is, if some one writes a link in a a-tag, the URL inside the href value is made clickable. Not good. Can someone perhaps help me with the issue? Function: function makeClickableLinks($text)
Technical SEO | | rasmusbang
{
$text = preg_replace('/<//', ' $text = preg_replace('(
)', '
', $text);
$text = preg_replace('!((https?://www.|https?://|www.)(([a-z0-9-]+.)+[a-z]{2,6})(/\S+|/)*)!ie', '"[".shortenurl("\1")."]("".(strtolower('$2'))"', $text);
$text = str_replace('( <a href',="" '<a="" $text);<br="">$text = str_replace(')" target', '" target', $text);
$text = str_replace('):" target', '" target', $text);
$text = str_replace(')..." target', '" target', $text);
$text = str_replace(').." target', '" target', $text);
$text = str_replace(')." target', '" target', $text);
return $text;
}</a> <a href',="" '<a="" $text);<br="">Pleeeeease heeelp 🙂 Can't fix it on my own - been at it for ages. -Rasmus</a>0 -
Overly Dynamic URLs
I have a site that I use to time fitness events and I like to post the results using query strings. I create a link to each event's results/gallery/etc. I don't need these pages crawled and I don't want them to hurt my seo. Can I put a "do not crawl" meta on them or will that hurt my overall positioning? What are my other options?
Technical SEO | | bobbabuoy0 -
Siemap.xml appearing in SERP
My sitemap.xml was appearing in the google serp for certain keywords (& not my actual page onsite). Please see image. I recently blocked my sitemap.xml with a robots.txt exclusion but now the sitemap.xml is not getting crawled in google webmaster. Is this the correct method of excluding the sitemap.xml for the serp? User-agent: * Disallow: /assets/cache/ Disallow: /assets/docs/ Disallow: /assets/export/ Disallow: /assets/import/ Disallow: /assets/modules/ Disallow: /assets/plugins/ Disallow: /assets/snippets/ Disallow: /manager/ Disallow: /sitemap.xml Sitemap: http://bryansryan.ie/sitemap.xml Any suggestions what should be done here? thanks. nQo2g.png
Technical SEO | | Socialdude0 -
URL Structure Question
Hey folks, I have a weird problem and currently no idea how to fix it. We have a lot of pages showing up as duplicates although they are the same page, the only difference is the url structure. They seem to show up like: http://www.example.com/page/ and http://www.example.com/page What would I need to do to force the URLs into one format or the other to avoid having that one page counting as two? The same issue pops up with upper and lower case: http://www.example.com/Page and http://www.example.com/page Is there any solution to this or would I need to forward them with 301s or similar? Thanks, Mike
Technical SEO | | Malarowski0 -
What is the best way of generating links with our exchanging them
I have read that google do not look kindly at sites that exchange links so i am trying to find a way of generating links to a new site. I am building a new site and need to start to get google to index it and build important links to generate traffic. I have looked at link exchange sites but have read that this is not great with google and it is better if you have sites where there is just one way linking. I do not want to buy links and would like to find a way of generating free links which can help build up traffic and the status of my new site. Any help would be great
Technical SEO | | ClaireH-1848860