Guys & Gals anyone know if urllist.txt is still used?
-
I'm using a tool which generates urllist.txt and looking on the SEO Forums it seems that Yahoo used to use this. What I'd like to know is is it still used anywhere and should we have it on the site?
-
Thanks for the advice, we already create and submit the XML sitemap to Google, that wasn't the question. Would there be any benefit in creating the urllist.txt file?
-
I would just use a sitemap.xml file instead for Google, Bing and Yahoo. Then you can submit the sitemap.xml file within the Google Webmaster Tools and Bing Webmaster Tools (includes Yahoo). You can easily create an XML sitemap at http://www.xml-sitemaps.com/
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Robots.txt in subfolders and hreflang issues
A client recently rolled out their UK business to the US. They decided to deploy with 2 WordPress installations: UK site - https://www.clientname.com/uk/ - robots.txt location: UK site - https://www.clientname.com/uk/robots.txt
Technical SEO | | lauralou82
US site - https://www.clientname.com/us/ - robots.txt location: UK site - https://www.clientname.com/us/robots.txt We've had various issues with /us/ pages being indexed in Google UK, and /uk/ pages being indexed in Google US. They have the following hreflang tags across all pages: We changed the x-default page to .com 2 weeks ago (we've tried both /uk/ and /us/ previously). Search Console says there are no hreflang tags at all. Additionally, we have a robots.txt file on each site which has a link to the corresponding sitemap files, but when viewing the robots.txt tester on Search Console, each property shows the robots.txt file for https://www.clientname.com only, even though when you actually navigate to this URL (https://www.clientname.com/robots.txt) you’ll get redirected to either https://www.clientname.com/uk/robots.txt or https://www.clientname.com/us/robots.txt depending on your location. Any suggestions how we can remove UK listings from Google US and vice versa?0 -
Https & http
I have my website (HTTP://thespacecollective.com) marked on Google Webmaster Tools as being the primary domain, as opposed to https. But should all of my on page links be http? For instance, if I click the Home button on my home page it will take the user to http, but if you type in the domain name in the address bar it will take you to https. Could this be causing me problems for SEO?
Technical SEO | | moon-boots0 -
HTTPS & 301s
Hi We have like most set up a redirect from HTTP to HTTPS. We also changed our website and set up redirects from .ASP pages to PHP pages We are now seeing 2 redirects in place for the whole of the website.
Technical SEO | | Direct_Ram
http.www.domain.com > https.www.domain.com (1) >> oldwebpage.asp >> new webpage.php (2) The question is: Is there anyway of making the redirect 1 and not 2? thanks
Enver0 -
Tags, Categories, & Duplicate Content
Looking for some advice on a duplicate content issue that we're having that definitely isn't unique to us. See, we are allowing all our tag and category pages, as well as our blog pagination be indexed and followed, but Moz is detecting that all as duplicate content, which is obvious since it is the same content that is on our blog posts. We've decided in the past to keep these pages the way they are as it hasn't seemed to hurt us specifically and we hoped it would help our overall ranking. We haven't seen positive or negative signals either way, just the warnings from Moz. We are wondering if we should noindex these pages and if that could cause a positive change, but we're worried it might cause a big negative change as well. Have you confronted this issue? What did you decide and what were the results? Thanks in advance!
Technical SEO | | bradhodson0 -
Can I rely on just robots.txt
We have a test version of a clients web site on a separate server before it goes onto the live server. Some code from the test site has some how managed to get Google to index the test site which isn't great! Would simply adding a robots text file to the root of test simply blocking all be good enough or will i have to put the meta tags for no index and no follow etc on all pages on the test site also?
Technical SEO | | spiralsites0 -
Should I be using use rel=author in this case?
We have a large blog, which it appears one of our regional blogs (managed separately) is simply scraping content off of our blog and adding it to theirs. Would adding rel=author (for all of our guest bloggers) help eliminate google seeing the regional blog content as scraped or duplicate? Is rel=author the best solution here?
Technical SEO | | VistageSEO0 -
Remove a directory using htaccess
Hi, Can someone tell me if there's a way using htaccess to say that everything in a particular directory, let's call it "A", is gone (http 410 code)? i.e. all the links should be de-indexed? Right now, I'm using the robots file to deny access. I'm not sure if it's the right thing to do since Google webmaster tools is showing me the link as indexed still and a 403 error code. Thanks.
Technical SEO | | webtarget0 -
How to re structure a website & Google
I have a website that requires the site structure to be changed. The website doesnt have many backlnks and rankings are fairly low. I have 11,000 products on the website and want to know the best way to change the site structure without causing 404 errors all over the place. Do I 301 redirect every page? drop all 11,000 pages from the index by adding a no follow no index to all pages? I have the following structure www.domain.co.uk/make/model/part/product I want to change this to www.domain.co.uk/Part/make/model/product whats the best way to preserve the SEO, link juice and on a large scale? 11,000 pages. thank you shivun
Technical SEO | | seohive-2227200