How to visualize our entire site to discover the origin of URLs?
-
What is a tool to use so that I can visualize all links to all pages on the site so that I can discover how certain duplicate content URLs are being created?
-
SEO Spider from Screaming Frog is a great crawler and a notable alternative to Xenu
-
I have not used the tool in quite some time but if I recall correctly it does offer a referring page URL. I can't imagine it would be useful without that information.
-
Yea, I've seen that before. Will that show me how unwanted pages are being referenced?
-
I believe this tool will meet your needs.
-
What if I need 150k pages crawled, rather than 20k?
-
If you set up a SEOmoz campaign, the web crawl report will offer that information. If not, you can request your site to be crawled: http://pro.seomoz.org/tools/crawl-test
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Site scraped over 400,000 urls
Our business is heavily dependent on SEO traffic from long tail search. We have over 400,000 pieces of content, all of which we found scraped and published by another site based out of Hong Kong (we're in the US). Google has a process for DMCA takedown, but doing so would be beyond tedious for such a large set of urls. The scraped content is outranking us in many searches and we've noticed a drastic decrease in organic traffic, likely from a duplicate content penalty. Has anyone dealt with an issue like this? I can't seem to find much help online.
Technical SEO | | Kibin0 -
Are there any negative side effects of having millions of URLs on your site?
After a site upgrade, we found that we have over 3.7 million URLs on our site. Many of these URLs are due to the facet options. Each facet combination yields a different URL. However, we need to do a deeper analysis into these URLs to see if this is the only reason why so many are returning. Does anyone know if there are any negatives of having so many URLs crawled, other than the fact that Google only spends so much time crawling a site? Is the number of URLs something that should be concerning? Any insight appreciated!
Technical SEO | | Deluxe0 -
Why my site is not indexing in google
In google webmaster i have updated my sitemap in Mar 6th..There is around 22000 links..But google fetched only 5300 links for long time...
Technical SEO | | Rajesh.Chandran
I waited for 1 month till no improvement in google index..So apr6th we have uploaded new sitemap (1200 links totally)..,But only 4 links indexed in google ..
why google not indexing my urls? Is this affect our ranking in SERP? How many links are advisable to submit in sitemap for a website?0 -
I noticed all my SEOed sites are getting attacked constantly by viruses. I do wordpress sites. Does anyone have a good recommendation to protect my clients sites? thanks
We have tried all different kinds of security plugins but none seem to work long term.
Technical SEO | | Carla_Dawson0 -
Friendly URL
Can be Friendly URL installed on a custom made jobsite using mod rewrite / apache without any big interference to the system itself? Thank you.
Technical SEO | | tomaz770 -
Site Categorization?
I know getting site categories to appear under the site are dependent on a lot of factors including site mapping. We have a site that does the categorization thing when you type in the sites url name however more people search for the name of the talent to find the site and the short url on the site is just his name, but shorter. However I was just wondering is their a way to optimize the site so that way we could get categorization to show up under the sites URL when they search for the talents full name I ask because the amount of people looking for the talents full name rather than the short name is a lot larger and I would like to see if we can take advantage of the real estate, but I honestly don't think there is a way, however I figured I would open it up to discussion to see if anyone has any ideas. Example: Site name is ABCD you type this into Google and you get ABCD.com about blog how to contact However the actual person whose site it is is ABCDEF and when you type that in you just get: ABCD.com without any of the categories appearing below the url. And that is what I'm asking about. Thanks as I can't seem to find a lot of information on this. However if there is another spot on the site talking about this please let me know I may just not be searching with the right terms.
Technical SEO | | KateGMaker0 -
HTML url extension
I've read some information about the extension of an url. But i couldn't find a clear answer. What is better for SEO, an extension with html or without? /make-money-online/how-to-make-a-million-dollars-in-1-year/ or /make-money-online/how-to-make-a-million-dollars-in-1-year.html/ Is there a difference between a normal website or a blog?
Technical SEO | | PlusPort0 -
How to setup tumblr blog.site.com to give juice to site.com
Is it possible to get a subdomain blog.site.com that is on tumblr to count toward site.com. I hoped I could point it in webmaster tools like we do www but alas no. Any help would be greatly appreciated.
Technical SEO | | oznappies0