Attack of the dummy urls -- what to do?
-
It occurs to me that a malicious program could set up thousands of links to dummy pages on a website:
www.mysite.com/dynamicpage/dummy123
www.mysite.com/dynamicpage/dummy456
etc..
How is this normally handled? Does a developer have to look at all the parameters to see if they are valid and if not, automatically create a 301 redirect or 404 not found? This requires a table lookup of acceptable url parameters for all new visitors.
I was thinking that bad url names would be rare so it would be ok to just stop the program with a message, until I realized someone could intentionally set up links to non existent pages on a site.
-
Hello,
I am also having this issue with hundreds of dummy urls that never existed as a part of our website's blog. Do I go into parameters and specify each of the dummy urls to avoid this?
Thanks in advance for any help!!!! (and sorry to piggyback this question Theodore-hope you don't mind!)
-
Thanks Ray. Appreciate the advice!
-
It's great that you've identified issues like this. I also suggest that if you know certain parameters are generated often and not necessary to index, that you go into your Google Webmaster Tools account > Crawl > URL Parameters and proactively set the crawl rate to 'No URLs' is appropriate. I do this with certain custom parameters for sites that are prone to having these extra URLs indexed mistakenly.
-
Hi Ray-pp,
Thanks for your answer. I'm not getting anything significant, but occasionally a bot will come with extra stuff added to the parameter names, so it got me to thinking a malicious program or nasty competitor might want to do that to cause havoc. My understanding is that 404s don't hurt SEO ranking from Google, but I was thinking that the way things are set up now no-one would get a 404 and in fact Google would index the 'bad' pages, so maybe I needed to do something proactively to 404 or 301 such pages so they would never get put into an index at all.
Since my site has lots of dynamically generated pages, I've had my share of surprises, and am just trying to avoid any new ones!
-
Hi Theodore - You pose an interesting problem, are you currently experiencing this issue? I don't see why someone would create a bunch of random non-existent links to your site, but if they did (and the pages were receiving low quality traffic) then I would proactively disavow those domains that created the links. That would be enough to prevent any penalties you're afraid of receiving.
If, however, you're noticing that specific 404 pages are receiving quality traffic (maybe an old page was removed but good traffic is still sent to the page) then you would want to 301 that page to its closest relative page that deserves the traffic and authority.
Does that help? Maybe a little more information around you specific problem would allow me to tailor the advice better.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
My url disappeared from Google but Search Console shows indexed. This url has been indexed for more than a year. Please help!
Super weird problem that I can't solve for last 5 hours. One of my urls: https://www.dcacar.com/lax-car-service.html Has been indexed for more than a year and also has an AMP version, few hours ago I realized that it had disappeared from serps. We were ranking on page 1 for several key terms. When I perform a search "site:dcacar.com " the url is no where to be found on all 5 pages. But when I check my Google Console it shows as indexed I requested to index again but nothing changed. All other 50 or so urls are not effected at all, this is the only url that has gone missing can someone solve this mystery for me please. Thanks a lot in advance.
Intermediate & Advanced SEO | | Davit19850 -
301 Redirects to relative URLs not absolute a problem?
Hi we recently did a migration and a lot of content changed locations see: https://d.pr/i/RvqI81 Basically, the 301 goes to the correct location but its a relative URL (as you can see from the screenshot) rather than absolute URL. Do you think this is a high priority issue from an SEO standpoint, should we get the developer to change the redirects to absolute? Cheers.
Intermediate & Advanced SEO | | cathywix0 -
URL Migration: Better to have .301s processed or 200s?
I'm migrating sub-domains to sub-folders, but this question is likely applicable for most URL migrations. For example: subdomain1.example.com to example.com/subdomain1 and any child pages. Bear with me as it may just be me but I'm having trouble understanding whether internal links (menu, contextual etc and potentially the sitemaps) should be kept as the pre-migration URL (with .301 in place to the new URL) to give Google a chance to process the redirects or if they should be updated straight away to the new URL to provide a 200 response as so many guides suggest. The reason I ask is unless Google specifically visits the old URL from their index (and therefore processes the .301), it's likely to be found by following internal links on the website or similar which if they're updated to reflect the new URL will return a 200. I would imagine that this would be treated as a new page, which is concerning as it would have a canonical pointing toward itself and the same content as the pre-migrated URL. Is this a problem? Do we need to allow proper processing of redirects for migrations or is Google smarter than this and can work it out if they visit the old URL at a later date and put two and two together? What happens in-between? I haven't seen any migration guides suggest leaving .301s in place but to amend links to 200 as soon as possible in all instances. One thought is I guess there's also the Fetch as Google tool within Search Console which could be used with the old URLs - could this be relied on? Apologies if this topic has been covered before but it's quite difficult to search for without returning generic topics around .301 redirects. Hope it makes sense - appreciate any responses!
Intermediate & Advanced SEO | | AmyCatlow0 -
URL Structure For E-commerce Sites
Hi Guys, I was wondering what would be the optimal and best URL structure for sub-categories on a E-commerce site for SEO purposes. Example if my category was dresses and I had multiple sub-categories within dresses would 1 or 2 below be the better URL structure? 1) Domain + Category + Sub-Category be the most suitable URL structure: Sleeveless Dresses URL: clothingstore.com/dresses/sleeveless-dresses Midi Dresses URL: clothingstore.com/dresses/midi-dresses 2) OR would excluding the category be better Domain + Sub-Category like: Sleeveless Dresses URL: clothingstore.com/sleeveless-dresses Midi Dresses URL: clothingstore.com/midi-dresses Do you think it makes much of a difference, is shorter better and more effective in this case? E.g. Rand discuses in this article: https://moz.com/blog/15-seo-best-practices-for-structuring-urls that having the keyword in the URL serves as anchor text, so wouldn't having additional keywords dilute value in this case? Plus he mentions shorter URLs the better. Cheers, Chris
Intermediate & Advanced SEO | | jayoliverwright1 -
Duplicate Content with URL Parameters
Moz is picking up a large quantity of duplicate content, consists mainly of URL parameters like ,pricehigh & ,pricelow etc (for page sorting). Google has indexed a large number of the pages (not sure how many), not sure how many of them are ranking for search terms we need. I have added the parameters into Google Webmaster tools And set to 'let google decide', However Google still sees it as duplicate content. Is it a problem that we need to address? Or could it do more harm than good in trying to fix it? Has anyone had any experience? Thanks
Intermediate & Advanced SEO | | seoman100 -
Ecommerce SEO URL Structure Questions
| I am in the process of developing a new Magento ecommerce store. Take for instance this website is in the apparel industry and i have the following main categories. Clothing Shoes Accessories Beauty Sub categories for clothing would be: Dresses Pants jeans Tops Products would be: Kelly Maxi dresses What is the best SEO Structure for this? Main categories obviously: www.example.com/clothing Sub Categories:
Intermediate & Advanced SEO | | WayneRooney
www.example.com/clothing/dresses Or www.example.com/dresses (Zappos seem to pursue the second type) Products:
www.example.com/clothing/dresses/kelly-maxi-dresses/ Or www.example.com/kelly-maxi-dresses ? Which one would be the best way to structure your site? Also what about filters that available in category pages? Say if i were to filter by color. what would be the best URL? I am sure canonical tag is needed here. New to Ecommerce SEO so i need some guidance! |0 -
URL with a # but no ! being indexed
Given that it contains a #, how come Google is able to index this URL?: http://www.rtl.nl/xl/#/home It was my understanding that Google can't handle # properly unless it's paired with a ! (hash fragment / bang). site:http://www.rtl.nl/xl/#/home returns nothing, but: site:http://www.rtl.nl/xl returns http://www.rtl.nl/xl/#/home in the result set
Intermediate & Advanced SEO | | EdelmanDigital0 -
Where to put a page ID in a URL?
Hello, My company is going to change URLs to example.com/category or example.com/product. When we will change the URLs to product or category pages somehow we have to check whether the requested page is from category table in DB or from products table (this gives much speed to page load time). So we have to choose how to make the different product and category pages.
Intermediate & Advanced SEO | | komeksimas
Programmers said that we need to insert id to URL. So the question is: Which is the better way to place an id to an URL? example.com/product-name?id=111 example.com/product-name/111 example.com/product_name-111 Or maybe we should use some other punctuation mark to separate id from product name? p.s. I have read Dynamic URLs vs. static URLs by Google and it still didn't answered which is the best for all of the pages. Somehow others solve this problem by typing only the names to the URL, but could anyone tell what that technology should be?0