Sitemap links
-
Hi, I´m running a sitemap using pro-sitemaps and I find several pages that shouldn´t be listed. How do I find how are these pages being generated? Can´t find the links the robot is following to get to those pages..
-
Screaming Frog is the best crawler out there imo. Have you talked to your hosting provider? They may be refusing the connection on their end.
-
Hi Thanks for your help! I keep getting "Connection Refused" status from the Screamin Frog..just couldn´t use it..I have tried the proxy, firewall and all recommended steps for solving this issue..
Is there a similar app?
Thanks!
-
Will try doing that thanks!
-
Have you considered using a tool like Screaming Frog SEO Spider?
It can crawl your site and it will list the inbound links to pages all neatly in a tab (exportable to Excel).
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
.xml sitemap showing in SERP
Our sitemap is showing in Google's SERP. While it's only for very specific queries that don't seem to have much value (it's a healthcare website and when a doctor who isn't with us is search with the brand name so 'John Smith Brand,' it shows if there's a first or last name that matches the query), is there a way to not make the sitemap indexed so it's not showing in the SERP. I've seen the "x-robots-tag: noindex" as a possible option, but before taking any action wanted to see if this was still true and if it would work.
Technical SEO | | Kyleroe950 -
Is sitemap required on my robots.txt?
Hi, I know that linking your sitemap from your robots.txt file is a good practice. Ok, but... may I just send my sitemap to search console and forget about adding ti to my robots.txt? That's my situation: 1 multilang platform which means... ... 2 set of pages. One for each lang, of course But my CMS (magento) only allows me to have 1 robots.txt file So, again: may I have a robots.txt file woth no sitemap AND not suffering any potential SEO loss? Thanks in advance, Juan Vicente Mañanas Abad
Technical SEO | | Webicultors0 -
Good alternatives to Xenu's Link Sleuth and AuditMyPc.com Sitemap Generator
I am working on scraping title tags from websites with 1-5 million pages. Xenu's Link Sleuth seems to be the best option for this, at this point. Sitemap Generator from AuditMyPc.com seems to be working too, but it starts handing up, when a sitemap file, the tools is working on,becomes too large. So basically, the second one looks like it wont be good for websites of this size. I know that Scrapebox can scrape title tags from list of url, but this is not needed, since this comes with both of the above mentioned tools. I know about DeepCrawl.com also, but this one is paid, and it would be very expensive with this amount of pages and websites too (5 million ulrs is $1750 per month, I could get a better deal on multiple websites, but this obvioulsy does not make sense to me, it needs to be free, more or less). Seo Spider from Screaming Frog is not good for large websites. So, in general, what is the best way to work on something like this, also time efficient. Are there any other options for this? Thanks.
Technical SEO | | blrs120 -
Page for Link Building
Hello Guys, My question is about a link building process. We all know that some directories/sites do require a reciprocal link. Does it make any sense to creat a page in website exclusively to reciprocal links? And what we do with this webpage in terms of indexing, do folow, crawling...etc. Any sugestions are more then welcome 🙂 Tks in advance! PP
Technical SEO | | PedroM0 -
Site Wide Links
I have a link on pr 3 home page website placed in the side bar. It is on a WordPress website that spans a couple hundred pages and the side bar is on every page. The majority of the pages are not ranked or have any pr. Can this affect me negatively?
Technical SEO | | raph39880 -
HTML Sitemap Pagination?
Im creating an a to z type directory of internal pages within a site of mine however there are cases where there are over 500 links within the pages. I intend to use pagination (rel=next/prev) to avoid too many links on the page but am worried about indexation issues. should I be worried?"
Technical SEO | | DMGoo0 -
Link building question
ok so we paid the top firm in seo to help us build an seo strategy and i think we have a good one. We are changing our link building tactics and making more Pr related links and creating awesome content on blogs or our own site to generate traffic and links to our site. We have data from our engineer which should be interesting and we are going to sponsor events, do some link baiting with some of our articles, get a pr firm to get us some good articles on major sites and go to events around phily where we will have unique content and a unique perspective such as car shows ect. The problem is even though all the content will be linked to our site how do we link them. We got hit by penguin but in these articles or blogs should we use the anchor text for the word we are using. The company says dont do it right now bc we got hit with penguin and should only use the brand. I have no idea how only using the brand and not the keywords will magically make us rank for certain keywords. Anyone have an opinion. Thank you and we do pretty well with seo but we did get little bit of a hit with penguin that we are eliminating links and making a new way of thinking when it comes to link building. We also just hired a designer so we are going to build 100s of pages on the site to increase seo with unique content and that is also a goal of ours for the year. We have two marketers on staff and 4 programmers so we are able to do anything. Our urls are terrible but the rest of the site is pretty good
Technical SEO | | goldjake17880 -
Are multiple links devalued on the same domain?
I'm in negotiations to get links placed on a popular blog with good stats. I'm allowed to pick older posts on the site, and I get to pick the anchor text. Is it best practice to diversify the links by having different keywords pointing to different pages or am I better off pointing as many links as I can at one page (varying anchor text)? Also, is it best to pick a more recent blog post, or is it ok to pick one from say, 2009?
Technical SEO | | MichaelWeisbaum0