How to handle blank, auto generated system pages/urls
-
Hi Guys
Our backend system has been creating listing pages based on out of date and irrelevant data meaning we have hundreds of thousands of pages that are blank but currently indexable and active. They're almost impossible to access from the front end and have 0 traffic pointing at them but you can access these pages if you have the URL and i'm pretty sure due to the site architecture, google is crawling them regardless. For the most part, I think its likely best to 301 these pages to the most closely related page on the site but I'm concerned we're wasting crawl budget here. We don't want these pages to be crawled or found. Would a sound solution be to make them inactive, no-index and create a custom 404 in the event anyone (or the crawler) managed to get to them? Would this enormous increase in 404 pages cause us issues?
Many thanks
-
Thanks for such a speedy reply! Its such a daunting task as there's literally thousands and thousands of pages so we want to be sure we're doing the right thing. I appreciate your help. Now i'll investigate blocking within the robots.txt and using google search console to remove the URLs
-
First, do not 404 them, use a 410 error code instead as that denotes intended permanent deletion. In addition, I would also block the files/folder in robots.txt. Finally, I would use Google Search Console to remove these URLs. Good luck.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Website URL Structure - keyword targeting on homepage vs internal pages
I have developed a few websites before where the homepage contains the content for the keywords I was targeting. This has been reasonably successful as I have found it easy enough to get links to the homepage. I am considering a new site in a totally different industry that I am thinking about structuring like this: mybrand.com (not necessarily targeting any keywords) mybrand.com/important-keyword-1/ (definitely want to target) mybrand.com/important-keyword-2 (equally important as 1st keyword) There will be several (30-ish) other pages targeting keywords but they are not as significant as the two mentioned above, more so they are about publishing informative information. The two important keywords are quite different but industry related. My questions are: should I be careful targeting keywords away from the homepage when the homepage gets the most links? Would I be better off building 2 different websites where the keyword content is captured in the homepage? Thanks,
Intermediate & Advanced SEO | | BGu0 -
Showing on page 25 for url keyword/ brand, is it a penalty?
Hi I have a site which shows on page 25 in G serps for the main brand keyword which is also the url its a .com and as far as I can see has no penalties and has unique content. The keyword itself has no competition and the site should be no1 in G for it. Our site domain is 11 years old.
Intermediate & Advanced SEO | | MoneySite0 -
Changing my pages URL name - HELP NEEDED FAST
Hello, I need to change the URL name for a few pages on my site. The site was launched just recently, so it has no obvious ranking and traffic. My question is, what is the best practice for changing/deleting the page name? after deleting the page, should I go to Google webmaster tool and use URL Removal and remove the old page? I know that I have to also create a new XML sitemap file, but not sure about the old pages in google search result Thanks!
Intermediate & Advanced SEO | | mdmoz0 -
Consistent Ranking Jumps Page 1 to Page 5 for months - help needed
Hi guys and gals, I have a really tricky client who I just can't seem to gain consistency with in their SERP results. The keywords are competitive but what the main issue I have is the big page jumps that happen pretty much on a weekly basis. We go up and down 40 positions and this behaviour has been going on for nearly 6 months.
Intermediate & Advanced SEO | | Jon_bangonline
I felt it would resolve itself in time but it has not. The website is a large ecommerce website. Their link profile is OK in regards to several high quality newspaper publication links, majority brand related anchor texts and the link building we have engaged in has all been very good i.e. content relevant / high quality places. See below for some potential causes I think could be the reason: The on page SEO is good however the way their ecommerce website is setup they have formed a substantial amount of duplicate title tags. So in my opinion this is a potential cause. The previous web developer set-up 301 redirects all to their home page for any 404 errors. I know best practice is to go to the most relevant pages, however could this be a potential issue? We had some server connectivity issues show up in webmasters tools but that was for 1 day about 4 months ago. Since then no issues. they have quite a few 'blocked URLs' in their robots.txt file, e.g. Disallow: /login, Disallow: /checkout/ but to me these seem normal and not a big issue. We have seen a decrease over the last 12 months in Webmasters Tools of 'total indexed web pages' from 5000 to 2000 which is quite an odd statistic. Summary So all in all I am a tad stumped. We have some duplicate content issues in title tags, perhaps not following best practice in the 301 redirects but other than that I dont see any major on page issues, unless I am missing something in the seriousness of what I have listed.
Finally we have also do a bit of a cull of poor quality links, requesting links to be removed and also submitting a 'disavow' of some really bad links. We do not have a manual penalty though. Thoughts, feedback or comments VERY welcome.0 -
Should you replace the url on a damaged page and 301 to it ?
Hi, We have a couple of pages which have been damaged due to an SEO person we hired creating a stupid amount of bookmarks and generally poor links. I've tried to get the links removed where I can but on most of these blogging sites there is no contact webmaster etc so I am struggling. Panda update as also affected traffic by about 35%. My question is , should I consider creating new urls for the "damaged " pages and then doing 301 redirects to them from the damaged page to the new page. Then start to build up good links to the new page whilst google should de-index the old pages over a couple of months ?. Just at my witts end how to get rid of these blogging rubbish etc etc. Thanks Sarah.
Intermediate & Advanced SEO | | SarahCollins0 -
E Commerce product page canonical and indexing + URL parameters
Hi, I'm having some issues on the best way to handle site structure. The technical side of SEO isn't my strong point so I thought I'd ask the question before I make the decision. Two examples for you to look at. This is a new site http://www.tester.co.uk/electrical/multimeters/digital. By selecting another page to see more products you get this url string where/p/2. This page also has the canonical tag relating to this page and not the original page. Now if say for example I exclude this parameter (where) in webmaster tools will I be stopping Google indexing the products on the other pages where/p/2, 3, 4 etc. and the same if I make the canonical point to multimeters/digital/ instead of multimeters/digital/where/p/2 etc.? I have the same question applied to the older site http://www.pat-services.co.uk/digital-multimeters-26.html. which no longer has an canonical tags at all. The only real difference is Google is indexing http://www.pat-services.co.uk/digital-multimeters-26.html?page=2 but not http://www.tester.co.uk/electrical/multimeters/digital/where/p/2 Thanks for help in advance
Intermediate & Advanced SEO | | PASSLtd0 -
In order to improve SEO with silos'urls, should i move my posts from blog directory to pages'directories ?
Now, my website is like this: myurl.com/blog/category1/mypost.html myurl.com/category1/mypage.html So I use silos urls. I'd like to improve my ranking a little bit more. Is it better to change my urls like this: myurl.com/category1/blog/mypost.html or maybe myurl.com/category1/mypost.html myurl.com/category1/mypage.html Thanks
Intermediate & Advanced SEO | | Max840 -
Need help/insights. Site SEO = OK, Authority BLs = OK, Rank page #1\. How to reach pos #1?
Hi there! Some doubts are confusing my head and need some assistence from you to get on the right track. I'll explain my situation and want to hear from you what do you really recommend for med/long term permanent results. 1 - I have a PR2 (.com.br) domain; 2 - I'm talking about little/med competition micro-niche keywords; 3 - I got all pages I want to, indexed (I have a well SEO constructed website with internal link building); 4 - If a keyword has average competition, I'll already start ranking in page #3 on the SERP's; For a few low competition keywords I start on page #1; 5 - I do a little whitehat link building, 1 or 2 backlinks on authority sites and then like 15 days later I came to page #1, generally on position 9/10; And then I got stucked 🙂 No more authority sites where I can get backlinks... I do some posts on the company twitter/facebook page's, but they are no follow, so I don't really now if this can help. (never see a SERP result). I did some "blackhat" stuff to see if it really work: I can say for sure the "profile backlinks" that we can buy from some sites doesn't work (maybe it's just for me). I can't see it on webmaster tool and neither my ranks changed since I bought a pack of 100 links (the links are working, I see it one by one) to test. Maybe the problem is about the domains, cause my site is .com.br and I'm buying .com profile links. I guess google understand backlinks from .com.br more valuable for my sites. Back to whitehat: I wrote some articles and posted it the right way, of course on .com.br articles sites, got it indexed and can see the backlink on webmaster tool, but no change on SERP's. (maybe this can be a long term result and I'm not seeing it yet). I'm really "scratching my hand" to do some blackhat stuff, but I don't want to lose what I already have done... I heard a lot about scrapebox but doesn't fell confortable to spam as hell a lot of blogs. I really want long term permanent results (my sites are totally whitehat/corporate sites). Can you expert guys give me some point to where I need to "walk" now to improve the SERP's? I never reached top #1 and want to try to rank at least one time to understand how this can be made... I'm thinking now to pay someone to rewrite 20 copies of an article and up it on some sites, to see if 20 can improve something. But still no confident, because it will cost like $100 for a good writer do it for me on my language. Maybe I can do better things with 100 bucks. I guess I did the path right: Internal SEO -> got indexed -> backlinking from authorities -> new articles backlinks to me (is it ok at this position or no?) -> (what next ?) I know SEO is a hard/never ending work, but what I'm trying to get cleaned on my head is the path of the work (if a right path really exists). Every word will be apreciated. What do you can suggest to me to try now? (please give me a hint to see SERP's results 🙂 if I feel that something worked, no matter how it can cost to me, but I'll pay for the work happily) Sorry if I'm a little confusing, english isnt' my first language. Thanks.
Intermediate & Advanced SEO | | azaiats20