I am assuming the site isn't yours and doesn't belong to you. That makes things trickier and it means you'll be working with estimates, but it doesn't make things completely impossible!

There are tools which monitor Google organic search positions and paid ads (AdWords) in order to specifically estimate the Google traffic which a site or web-page might be earning.

SEMRush is one tool (paid, though you can get some limited data for free):

• https://www.semrush.com/info/wikipedia.org

You can see a little chart here:

• https://d.pr/i/P2GoDU.png

They basically look at search and ad rankings and then estimate traffic by pitching CTR (click through rate) data against ranking position data. It's very ballpark, but it is at least something you could look at. SEMRush also has a "chart tool" whereby you can input sites and metrics and build comparative charts (quite nice). Keep in mind you can only see traffic data by Google region (e.g: USA / Google.com or UK . Google.co.uk) and getting to an all-encompassing global view can be tricky (and time consuming)

Ahrefs costs more but we have found their traffic estimates to be more responsive, accurate and easier to manage (especially if you just want to see flat global data):

• https://ahrefs.com/site-explorer/overview/v2/subdomains/recent?target=wikipedia.org

Again, it's a paid tool with some free data. You want to be looking here:

• https://d.pr/i/4KJVOM.png

The only thing you are seeing there, is organic (SEO) traffic, not PPC (paid search) traffic. Although the data is narrower from one perspective, it's much easier to get global data. In addition to that, I do find the data to be more accurate, especially since the search volumes which Ahrefs uses within its traffic estimates contain click-stream data to enhance accuracy:

• https://help.ahrefs.com/keywords-explorer/faqs-and-troubleshooting/how-accurate-is-keyword-search-volume-in-ahrefs

That being said, nothing is perfect:

• https://help.ahrefs.com/site-explorer/faqs-and-troubleshooting/why-dont-the-organic-traffic-numbers-reported-by-ahrefs-match-those-i-see-in-google-analytics

Some people have even done studies comparing different tools, including SimilarWeb's traffic forecasting which I have little / no experience with:

• https://www.screamingfrog.co.uk/how-accurate-are-website-traffic-estimators/

I heard down the grapevine a few years back, that SearchMetrics (which is quite expensive) is actually pretty good at all this stuff. What I was told was that, they buy ISP (internet service provider) data and use that to forecast and estimate traffic to different sites instead. If that is true, it would be by far the most accurate traffic data available, but would come at a premium.

I believe that earlier on in my career, I had a little play and it was pretty robust. Platforms change though, you'd have to contact them to see if they still offer similar data.

As you can see, there are different options with different strengths and weaknesses. Accurate, digital, legal competitor surveillance technology is seldom free. Even if you're not buying the data from the competitor (who would never sell it to you! Plus it might be fiddled...) - you're still paying someone. Either someone who resells ISP network data, or someone who has 'done something clever' to get more accurate Google-traffic forecasting than the rest.

Hope this helps

FYI!

Only her homepage 301s to her new site.

Some of her deep-link 404-ing URLs do not 301, and those are the ones which Moz picked up:

https://www.poppyporter.com/racing_wings_earrings.html

If you load the above URL, you will notice that in the URL bar it **does not **change to .co.uk. The 301 is homepage (or maybe 'key-page') level only

The domain is in fact, not parked and still hosts custom 404 pages and other content :')

Hi there Poppy!

For my own benefit and for the benefit of others who may wish to look at helping you solve this, just going to pop the spam results URL here:

• https://analytics.moz.com/pro/link-explorer/spam-score?site=poppyporter.co.uk&target=domain

This is the spam score which is worrying you:

• https://d.pr/i/wl6Dz0.png

And these are the links which Moz thinks are bad:

• https://d.pr/i/tNjbse.png

These are the notes from Moz on why linking sites and / or pages get higher spam flags:

• https://d.pr/i/SDKfmc.png

Basically if Moz identifies that a linking site shares common features with other sites that have factually gained Google penalties, then links from those site will raise the spam score higher and higher. For some reason Moz seems to think that Poppyporter.com fits that pattern, hence the high spam scores for those four links

These are the URLs for the offending web pages:

• https://www.poppyporter.com/racing_wings_earrings.html
• https://www.poppyporter.com/two_finger_ring.html
• https://www.poppyporter.com/large_ring_disc_model.html
• https://www.poppyporter.com/large_connections.html

If you visit those URLs, you will see that the site is not in fact parked at all. Although those pages all result in error 404s, if the site were really taken down - how come the Poppy Porter logo and branding are still there? How come featured products are still listed? If the site were parked and taken down, it wouldn't have your stuff still on it. Clearly it does and it is not parked (hence the custom 404 pages are still rendering)

If your site were really parked, the 404s would look like this instead:

• https://d.pr/i/Qqtojd.png

It would be a blank white page with no branded styling. Nothing except for the server's response! The fact that it's still all styled, shows that your old site is still online. Individual pages may have been taken down, but at large that site still exists

Interestingly, if you click any link on your old site, it moves you to the new site

Let's look at one of the 404 pages on your old site, which does not redirect to your new site and is still live:

• https://www.poppyporter.com/racing_wings_earrings.html

As you can see, there's no redirect to the new site:

• https://d.pr/i/gJ7uKT.png

If we look at the source code of a live (but 404-ing) page on poppyporter.com (say if we examine the shop link in the top-line nav):

• https://d.pr/i/dwdVut.png

You can see very clearly that, you have two sites live and one is linking to the other!

I'm not a betting man and I don't have loads of info available, but if I _were _a betting man I'd say, this is what has happened:

1. You didn't shut down your old site properly, leading to a 'broken' old site with loads of 404s and broken styling
2. You tried to 'point' your old site to your new site with some kind of link architecture update, maybe in the DB or CMS
3. Moz saw that the old site was all janky and broken with loads of broken looking and 404-ing pages
4. Moz gave your old site a higher spam score than it used to have because now it's all bork'd (broken)
5. But there are links from the old site to the new site
6. So now your new site is inheriting the negative spam scores generated by the botched move from old site to new
7. It's possible that the old site had legitimately high spam scores, but I can't say for certain. Certainly after this botched attempt to move across, you have ended up with a spammy looking (and acting, in terms of status codes) old site. Because of the links, those negative scores are being pushed from the old site to the new one
8. As stated I don't know if the old site had high spam scores prior to the botched move and you need to investigate that with some urgency, as it will entirely shape your next movements
9. If the old site was very spammy and you were trying to make a 'clean break', you have undone that by interlinking the old site with the new one. Certainly you'd want to perma-kill the old domain as a matter of urgency! Kill the links, kill the pages that are still up, do not redirect from old site to new etc.
10. If the old site had good metrics before the botched move, you want to replace all the links and live-URLs with 301 redirects. Only do this if you are certain that Google hadn't penalised the old site. This can be a bit of a technical thing to do

Anyway the long and the short of it is, yes your old site is still live - yes it is linking excessively to your new site! One way or the other, it needs sorting out...

All of the plugins I can find allow the tag to be deployed on pages, posts etc. You pick from a pre-defined list of existing content, instead of just whacking in a URL and having it inserted (annoying!)

If you put an index.php at that location (the location of the 404), you could put whatever you wanted in it. Might work (maybe test with one). Would resolve a 200 so you'd then need to force a 410 over the top. Not very scalable though...

Hmm, that's interesting - it should work just as you say! This is the point where you need a developer's help rather than an SEO analysts :') sorry!

Google will revisit 410s if it believes there is a legitimate reason to do so, but it's much less likely to revisit them than it is with 404s (which actively tell Google that the content will return).

Plugins are your friends. Too many will overload a site and make it run pretty slowly (especially as PHP has no multi-threading support!) - but this plugin, you would only need it temporarily anyway.

You might have to start using something like PHPMyAdmin to browse your SQL databases. It's possible that the uninstall didn't work properly and there are still databases at work, generating fresh URLs. You can quash them at the database level if required, however I'd say go to a web developer as manual DB edits can be pretty hazardous to a non-expert

Just remember, the only regex character which is supported is "*". Others like "" and "?" are not supported! So it's still very limited. Changing the response from 404 to 410 should really help, but be prepared to give Google a week or two to digest your changes

Yes, it would be tricky to inject those URLs with Meta no index tags, but it wouldn't be impossible. You could create an index.php file at the directory of each page which contained a Meta no-index directive, or use a plugin to inject the tag onto specific URLs. There will be ways, don't give up too early! That being said, this part probably won't add much more than the 410s will

It wouldn't be a bad idea to inject the no-index tags, but do it for 410s and not for 404s (doing it for 404s could cause you BIG problems further down the line). Remember, 404  - "temporarily gone but will come back", 410 - "gone - never coming back". Really all 410s should be served with no-index tags. Google can read dynamically generated content, but is less likely to do so and crawls it less often. Still - it would at least make the problem begin shrinking over time. It would be better to get the tags into to non-modified source code (server side rendering)

By the way, you can send a no-index directive in the HTTP header if you are really stuck!

https://sitebulb.com/hints/indexability/robots-hints/noindex-in-html-and-http-header/

The above post is quite helpful, it shows no-index directives in HTML but also in the HTTP header

In contrast to that example, you'd be serving 410 (gone) not 200 (ok)

You have taken some good measures there, but it does take Google time to revisit URLs and re-index them (or remove them from the index!)

Did you know, 404 just means a URL was temporarily removed and will be coming back? The status code you are looking to serve is 410 (gone) which is a harder signal

Robots.txt (for Google) does in-fact support wild cards. It's not full regex, in-fact the only wildcard supported is "*" (asterisk: matching any character or string of characters). You could supplement with a rule like this:

User-agent: *

Disallow: /*revslider*

That should, theoretically block any URL from indexation if it contains the string "revslider"
Be sure to **validate** any new robots.txt rules using Google Search Console to check they are working right!

Remember that robots.txt affects crawling and **not indexation!** To give Google a directive not to index a URL, you should use the Meta no-index tag:
[https://support.google.com/webmasters/answer/93710?hl=en](https://support.google.com/webmasters/answer/93710?hl=en)

**The steps are:**

• Remove your existing robots.txt rule (which would stop Google crawling the URL and thus stop them seeing a Meta no-index tag or any change in status code)
• Apply status 410 to those pages instead of 404
• Apply Meta no-index tags to the 410'ing URLs
• Wait for Google to digest and remove the pages from its index
• Put your robots.txt rule back to prevent it ever happening again
• Supplement with an additional wildcard rule
• Done!

-

Hope that helps

So you have! According to Ahrefs the site averages around 1,193 visits (all countries) per month from SEO (Google) specifically. These estimates can be inflated or deflated depending upon how well their keyword index covers your particular website. From Ahrefs you have a 'Domain Rating' of 82 which is really strong.

Looking in SEMRush it's less favourable with the tool suggesting you see between 150 and 300 visits a month from Google in the USA. SEMRush's keyword index (and general handling of data) isn't as good as the stuff you get out of Ahrefs so that's pretty much what I was expecting.

The Ahrefs estimated SEO traffic chart stretches back to 2015 so it seems as though traffic from Google was ascertained in a relatively sustainable way.

I think that you could find a buyer for your site. What they'd be willing to spend would depend upon their current resources and ability to 'keep it going'. Someone who could only just afford the purchase would see diminishing returns as they might lack the ability to invest in future content and community management. To someone with deeper pockets, maybe they could afford to spend over ten grand on it.

Certainly the metrics are pretty decent. Obviously I haven't dissected those metrics in a granular way by conducting some kind of forensic analyses, but the top-line signals are pretty good (in my opinion)

That's pretty good. One big red flag is when sites have like, 60+ CF and like 13 TF or in the single digits. If you want you can send your site to the email on my profile and I'll have more of a look. It still won't be full depth but should show if you have a dud or not. Up to you though

I think it could be something that someone would buy, but obviously that would need a more detailed investigation. It's great if Moz's metrics are high, but is that universally agreed by other tools? For example is the Ahrefs Domain Rating similar to Moz's Domain Authority score? Does SEMRush see the site as a link toxicity risk? Does Majestic SEO give reasonable Trust Flow and Citation Flow read-outs?

It also depends what the backlinks are like. Certainly it's great if there are a lot of linking domains (and the number of overall links isn't disproportionately different to the number of linking domains). But some of those links might be great, whilst others might be garbage.

You can get three really nice articles (1,000+ words written by a proper PR person, not a link builder!) posted across three really nice sites (with DAs ranging from 65 to 80+) for about £700 GBP / $890 USD. You have to figure that, since those are one-time posts, the value of each containing site must be significantly higher (in the thousands or tens of thousands of dollars)

When prospecting websites as potential link opportunities, I also look into tools like SEMRush or Ahrefs which give estimated traffic figures for those websites (although the traffic estimates are usually only estimates of what those sites take in from Google). I would certainly do the same thing in a lot more detail if I was going to be buying a site rather than a one-off placement. Some tools give even better traffic data for 3rd-party sites which they buy from ISPs and build a front-end around that

If your site could assure a buyer across different platforms (even if some readouts were variable, if they were all relatively high...) then I don't think you'd have too much trouble selling it

Of course some shady black-hat cats might wait for you to get sick of it and delete your own site. They may then buy your expired domain and use something like an advanced scraper to pull the (historic) site files off the 'WayBack Machine' (webarchive.org) and resurrect your website for $20-$40 USD

Be wary of the necromancers of the web

This is a really fascinating question. It's highly irregular for Google to de-list a site with absolutely no reason given. Even if it's something really bad like serving malware to Google's users, you usually get a hacked content notification

Your assertion that your site has been de-listed by Google due to data you are seeing in various analytics packages is backed up by Google's front-end:

• https://www.google.co.uk/search?q=site%3Awhoseno.com
• https://www.google.com/search?q=site%3Awhoseno.com
• https://www.google.fr/search?q=site%3Awhoseno.com
• https://www.google.bg/search?q=site%3Awhoseno.com

I can't find any pages from your site in Google US, UK, France or Bulgaria. Whatever has happened they seem to have gone fairly thermonuclear!

I performed a 25% crawl of your site using Screaming Frog (rendering / JS enabled), using Google's user agent (Googlebot). Some pages returned an error 404:

• https://www.whoseno.com/number-information
• https://www.whoseno.com/whose-number-is-this
• https://www.whoseno.com/track-location
• https://www.whoseno.com/get-mobile-number-details
• https://www.whoseno.com/phone-number-details
• https://www.whoseno.com/get-complete-details-of-your-ex
• https://www.whoseno.com/track-any-mobile-number
• https://www.whoseno.com/wrong-number
• https://www.whoseno.com/whose-number-is-this-calling-me
• https://www.whoseno.com/phone-number-search
• https://www.whoseno.com/recent-lookups-on-whoseno
• https://www.whoseno.com/get-details-of-any-mobile
• https://www.whoseno.com/get-details-of-any-phone-number-for-free
• https://www.whoseno.com/trace-location-on-map
• https://www.whoseno.com/reverse-directory
• https://www.whoseno.com/track-location-by-phone-number
• https://www.whoseno.com/reverse-phone-lookup
• https://www.whoseno.com/phone-number-lookup
• https://www.whoseno.com/reverse-phone-lookup-service

Although this seems like quite a few broken pages, there were many more which were rendering properly. This just looks like the kind of stuff which Google would flag as crawl errors, rather than taking a site down in its entirety when the majority of pages return 200 (OK).

Some of the URLs like, getting "complete details about your ex" Google may frown upon. People shouldn't really be able to go on a site and get complete details for their ex-partner as that promotes stalking (something which Google is firmly against, and which most first-world governments are moving to take more and more action on). Even if the name of the page is misleading and it doesn't (when working) really supply that functionality, that then makes it a spam page instead (as it looks to satisfy unscrupulous users looking for such information and then fails to deliver).

Out of the pages which are returning 200 (OK), most of them are individual phone number pages. An example might be this page: https://www.whoseno.com/US/2014623561 - the number has been publicly logged as spam. With the advent of GDPR legislation, if you are logging phone numbers and publicly keeping a database of them (without the permission of the phone number's owner) then you may be in breach of new European GDPR legislation (read about it here).

Google wants to continue operating in Europe, so whilst they may be an American company GDPR does heavily impact Google. They want to comply with GDPR

I checked the technical indexation of your pages, there don't seem to be any huge red flags.

• Robots.txt isn't blocking critical pages and resources
• Nor is the Meta no-index tag
• Canonical tags don't seems to be de-indexing real pages and pointing Google to broken ones
• Google's user-agent seems to be able to access most pages properly

I decided to search for your site on Bing to see if they had also de-indexed you:

• https://www.bing.com/search?q=whoseno.com

Bing still holds pages and records of your domain.

One of the results really interested me. There's a Twitter profile listed on those search results, the SERP snippet reads like this:

"Hussɑin Aвduℓℓɑtif (@whoseno) | Twitter

The latest Tweets from Hussɑin Aвduℓℓɑtif (@whoseno_)"_

The Twitter profile has been suspended. This may or may not be your Twitter profile. If it's not your Twitter profile, your digital identity may have accidentally been combined with this person's who may or may not have Twitter ToS or state-level action against them

You need to go to Google's Webmaster support forum here and ask them what the deal is.

It's unlikely to be Penguin / link related and I don't think it's tech related either. It could be GDPR concerns, pollution of your digital identity - combined with a 3rd party who has state-level action against them, or it could be a basic 'Google glitch'

Not a problem, I always try to give a solid answer

Maybe you could do a compromise and have a "categories" entry that breaks down into the main categories (but not all of them) or something like that. Always remember, you can hedge your bets to test a bit!

Having every news category in the top-line nav, breaking down into sub categories could be excessive. It's a shame that triple-expansion nav never caught on (then you could have categoires->category->sub-category on hover). Whilst that is technically feasible, it's not really very user friendly (at all) as it makes menus really jumpy and dysfunctional (in most instances anyway)

On lots of eCommerce sites now I notice that they have auto-completing product (or category) based pseudo-search bars. Like you'll type a bit of a word, and all the relevant products will come back and you click on one (instead of 'entering' the search, and seeing a page of results). Maybe you could innovate and create a similar thing for news stuff. Have people type a bit of a category (or tag / topic) and then pre-fill with a couple of categories and a load of articles (or something like that)

Just throwing out ideas. Not that yours is bad! Just always trying to think "how could this be more?"

To be honest I found Google Data Studio pretty horrible and was able to work much more easily with Microsoft's Power BI, though for corporations it's not free and the price-tag is steep

One feature that Power BI had was that, in addition to connecting to set platforms like Google Analytics, it could also connect to custom objects like an SQL database

If Google have been smart and kept up, I'd expect this to be a feature by now in Data Studio...

https://support.google.com/datastudio/answer/7020436?hl=en

Looks like I was right!

This could make your life really easy as long as your rank tracking supplier provides an API for their tool. You can make requests to the API and build your own SQL database of that same data for your own records. The server holding the SQL database would have to regularly run API-based tasks to pull the data from your ranking supplier, into itself

Once that was done, you could then just connect to your SQL database via GDS

1. You'd need a developer to build the database and the tasks which updated it (API fetches)
2. If your ranking supplier changed their API significantly you'd have to pay to get it updated
3. You'd need to pay to host your database online somewhere

Other than that... Seems like a good solution. Sorry it's not super cheap and simple - though!

Even if your ranking supplier provides back-end exports only and no API, you can still do what you want. You'd just have to create a script that uses a headless browser to export the data and then push the CSV exports to your database. Again you'd assuredly need a developer's help! Using something like Python's Selenium or Windmill modules might be feasible

Hope that helps

This is a pretty good question and the answer can be variable depending upon the level of authority which your domain has currently accrued.

One of our employees always touts this site (not one of our clients) as having superb navigation:

• https://www.virginexperiencedays.co.uk/

There are several aspects to the navigation and URL structure - and how those different elements interact.

Even the faceted navigation impacts the URL structure, for example you can build very specific (niche) URLs like this just using the on-site UI:

• https://www.virginexperiencedays.co.uk/multiple-car-experiences/yorkshire-and-humber/100-250/4-stars

So if you wanted to see driving 'experience days' for multi-car experiences (drive more than one car) within Yorkshire and Humber, which were between £100 and £250 with a 4-star or higher rating... the pages listed there would be the ones!

If you load that page, you can see that all of those things have been set in the sidebar based filtering (faceted) navigation. WordPress has issues in terms of reflecting faceted navigational choices properly in the URL structure, especially when more than faceted navigation filter is active. You then have to choose between pretty permalinks  or having multiple filters accessible to users simultaneously, not ideal. Due to that I'm pretty sure some kind of bespoke (or heavily customised) CMS would have been used to get everything all working in one place as it should.

In addition to all that neat stuff, if you hover over the top menu entries you'll see that they expand and are pretty comprehensive. That's called a 'mega' menu and re-distributes SEO authority very efficiently, so long as all of the links are accessible in the 'base' (non-modified) source code. Surely this is just the way to go for everyone right?

WRONG!

The cited website obviously has the backing of Virgin and some pretty colossal promotional activity going on behind the scenes. That's not insider info, Virgin is a huge multinational corporation! Anyone examining their backlink profile using publicly available data would say the same thing.

If you're not already riding a very high SEO authority stream, chances are that this type of navigation could actually lower your ranking results.

If you have loads of authority, you can tip it down channels to really soak up a lot of long-tail traffic with this very granular type of site-architecture which leverages great navigational/URL-structure interplay.

On the other hand, if you are a smaller site (of any description) - it's like using an irrigation system with only one bucket of water to supply it. That bucket could have kept 2-3 plants alive for a few days, but spread evenly among hundreds of crops it makes literally no difference (and everything dies together, oh no!)

Implementing very advanced navigational structures too early can sometimes cause your small amount of SEO authority to 'bleed out' and achieve relatively little (or nothing) over thousands of pages.

It's great to be ambitious, and all of these great tactics can work (on any site, not just news or experience-day sites) but they all have a proper time of adoption. Adopt too early and you risk flattening your prior achievements in a void of poor results. Adopt too late and you risk being left behind as your peers take a chance and (potentially) make something more of themselves

It's not just what you do, it's when you do it (if that makes sense)

On a news site, or any other site - I'd say start simpler and as you accrue more SEO authority, redistribute it and gradually increase depth of navigation, URL structure and content granularity

Here is your basic decision:

1. Flatter URL structure and condensed nav helps the site build up authority and win a few mid-to-high tier terms
2. Deeper URL structure with expansive nav helps reap the long-tail and gain massive amounts of traffic, but only once your SEO authority reaches a certain threshold. Before that point, you'll just bleed out

Hope that helps

Pages that I like to call 'core' site URLs should go in your sitemap. Basically, unique (canonical) pages which are not highly duplicate, which Google would wish to rank

I would include core addresses

I wouldn't include uploaded documents, installers, archives, resources (images, JS modules, CSS sheets, SWF objects), pagination URLs or parameter based children of canonical pages (e.g: example.com/some-page is ok to rank, but not example.com/some-page?tab=tab3). Parameters are additional funky stuff added to URLs following "?" or "&".

There are exceptions to these rules, some sites use parameters to render their on-page content - even for canonical addresses. Those old architecture types are fast dying out, though. If you're on WordPress I would index categories, but not tags which are non-hierarchical and messy (they really clutter up your SERPs)

Try crawling your site using Screaming Frog. Export all the URLs (or a large sample of them) into an Excel file. Filter the file, see which types of addresses exist on your site and which technologies are being used. Feed Google the unique, high-value pages that you know it should be ranking

I have said not to feed pagination URLs to Google, that doesn't mean they should be completely de-indexed. I just think that XML sitemaps should be pretty lean and streamlined. You can allow things which aren't in your XML sitemap to have a chance of indexation, but if you have used something like a Meta no-index tag or a robots.txt edit to block access to a page - **do not **then feed it to Google in your XML. Try to keep **all **of your indexation modules in line with each other!

No page which points to another, separate address via a canonical tag (thus calling itself 'non-canonical') should be in your XML sitemap. No page that is blocked via Meta no-index or Robots.txt should be in your sitemap.XML either

If you end up with too many pages, think about creating a sitemap XML index instead, which links through to other, separate sitemap files

Hope that helps!

There's a few things you need to marry up if you want to do this. You need the referring page or domain / hostname (to validate that the session came from a backlink you know about). Once you filter the data down like that, you just need to filter by user-agent ("googlebot" - or any user-agent string which contains "googlebot"). Then you just want to look at the IP address field in the tabular data and you have your answers!

Here's the problem, most IP-level data is contained within basic server-side analysis packages (like AWStats which is installed on most sites, within the cPanel) or alternatively you can go to the log files for much of the same data. Most referrer-level data (stuff that deals with attribution) is contained within Analytics suites like Adobe Omniture or Google Analytics.

In GA, you can't usually get to 'individual' IP-level data. There used to be a URL hack to force it to render, but it was killed off (and many people who used it were banned by Google). The reason for that is, Google don't want too much PID (Personally Identifiable Data) harvested by their tool. It creates too many legal issues for Google (and also, whomever is leveraging that data for potentially nefarious marketing purposes)

Since you won't get enough IP-level data from GA, you're going to have to go to log files and log analysis tools instead. Hopefully they will contain at least some referral level data... The issue is, getting all the pieces you want to align in a legally compliant way

Obviously you have your reasons for looking. I'd check if you can find anything on your CPanel in AWStats (if that's installed) or get the log files and analyse them with something like Screaming Frog Log File Analyser

I can't promise this will return the data you want, but it's probably your only hope

There are so many ways to deal with this. If these were indeed spam URLs, someone may have attached negative-SEO links to them (to water down your site's ranking power). As such, redirecting these URLs back to their parents could pull spam metrics 'onto' your site which would be really bad. I can see why you are thinking about using 410 (gone)

Using Canonical tags to stop Google from indexing those bad parameter-based URLs could also be helpful. If you 'canonicalled' those addresses to their non-parameter based parents, Google would stop crawling those pages. When a URL 'canonicals' to another, different page - it cites itself as non-canonical, and thus gets de-indexed (usually, although this is only a directive). Again though, canonical tags interrelate pages. If those spam URLs were backed by negative SEO attacks, the usage of canonical tags would (again) be highly inadvisable (leaving your 410 suggestion as a better method).

Google listens for wildcard rules in your robots.txt file, though it runs very simplified regex (in fact I think only the "*" wildcard is supported). In your robots.txt you could do something like:

User-agent: *
Disallow: /mono.php?*

That would cull Google's crawling of most of those URLs, but not necessarily the indexation. This would be something to do after Google has swallowed most of the 410s and 'got the message'. You shouldn't start out with this, as if Google can't crawl those URLs - it won't see your 410s! Just remember this, so that when the issue is resolved you can smack this down and stop the attack from occurring again (or at least, it will be preemptively nullified)

Finally you have Meta "No-Index" tags. They don't stop Google from crawling a URL, but they will remove those URLs from Google's index. If you can serve the 410s on a custom 410 page which also gives the Meta no-index directive, that will be a very strong signal to Google indeed that those aren't proper pages or fit for indexation

So now we have a bit of an action plan:

• 410 the bad URLs alongside a Meta no-index directive served from the same URL
• Once Google has swallowed all that (may be some weeks or just over 1 month), back-plate it with robots.txt wildcards

With regards to your oriignal question (sorry I took so long to get here) I'd use something like:

Redirect 410 /mono.php?*

I think .htaccess swallows proper regex (I think). The back slashes say "whatever character follows me, treat that character as a value and do not apply its general regex function". It's the regex escape character (usually). This would go in the .htaccess file at the root of your site, not in a subdir .htaccess file

Please sandbox text my recommendation first. I'm really more of a technical data analyst than a developer!

This document seems to suggest that a .htaccess file will properly swallow "" as the escape character:

https://premium.wpmudev.org/forums/topic/htaccess-redirects-with-special-characters

Hope this helps!

Tim Holmes gave a good answer but it does assume your redirects are being applied via a .htaccess file which is the usual method if your website is hosted on a Linux / Apache server.

If your website runs on a Windows / IIS server, then instead of implementing your redirect rules via .htaccess you'd be using web.config instead. Obviously most plugins (especially on common platforms like WordPress) are coded to interact with a .htaccess file. If you're running on IIS instead they could break stuff or at the least fail to function entirely.

On Google you can find many posts complete with web.config instructions:

https://www.google.com/search?num=100&q=https+redirect+for+web.config

This is the one which Google gives the knowledge-graph entry to:

https://www.ssl2buy.com/wiki/http-to-https-redirect-using-htaccess-or-web-config

The second part of the content deals with Windows.

Checking that your SSL certificate is correctly installed, valid and provided by a supplier which Google accepts is highly advisable. If browsing to an HTTPS URL on Chrome yields warnings or 'not secure' messages, it's safe to say that Google has not accepted your SSL certificate. If you can't even browse to HTTPS URLs, something is likely wrong with the install!

Hope that helps

This is probably more of a ranking authority problem, rather than an indexation problem. If you can force Google to render one of your category URLs within its search results, then it's highly likely the page is indeed indexed (it's just not ranking very well for associated keywords)

Follow this link:

https://www.google.co.uk/search?q=site%3Askirtinguk.com%2Fproduct-category%2Fmdf-skirting-board%2F

As you can see, the category URL which you referenced is indexed. Google can render it within their search results!

Although Google know the page exists and it is in their index, they don't bother to keep a cache of the URL: http://webcache.googleusercontent.com/search?q=cache:https%3A%2F%2Fwww.skirtinguk.com%2Fproduct-category%2Fmdf-skirting-board%2F

This probably means that they don't think many people use the page or that it is of low value.

What you have to keep in mind is, lower value long-tail terms (like product keywords or part number keywords) are much easier to achieve. Category terms are worth more in terms of search volume, so competition for them is higher. If your site ranks for product terms but not for category terms, it probably means your authority and / or trust metrics (as well as UX metrics) may be lower. Remember: Google don't consider their ranking results to be a space to advertise lots of companies. They want to render the best results possible for the end-user (that way people keep 'Googling' and Google continue to leverage revenue from Google AdWords etc)

Let's look at your site's domain-level metrics and see if they paint a picture of an 'authoritative' site which should be ranking for such terms...

Domain Level Metrics from Moz

Domain Authority: 24 (low)

Total Inbound Links: 1,200+

Total Referring Domains (much more important than total link count!): 123 - This is too many links from too few domains IMO

Ranking keywords: 38

Domain Level Metrics from Ahrefs

Homepage URL Rating: 11 (very low)

Domain Rating: 11 (very low)

Total Inbound Links: 2,110+

Referring Domains: 149 - Again, the disparity here could be causing problems! Not a diverse backlink profile

Ranking Keywords: 374 (Ahrefs usually finds more, go with this figure)

SEO Traffic Insights: Between 250 and 380 visits (from SEO) a day on average, not much traffic at all from SEO before November 2016 when things improved significantly

SEMRush Traffic Insights (to compare against Ahrefs): Estimates between 100 and 150 visits from SEO per day. This is narrowed to UK only though. Seems to tally with what Ahrefs is saying, the Ahrefs data is probably more accurate

Domain Level Metrics from Majestic SEO

Trust Flow: 5 - This is extremely low and really bad! Basically Majestic track the number of clicks from a seed set of trusted sites, to your site. A low number (it's on a scale of 0 to 100 I think) indicates that trustworthy seed sites aren't linking to you, or that where you are linked - people avoid clicking a link to your site (or visiting it)

Citation Flow: 24 - low but now awful

What do I get from all of this info?

I don't think your site is doing enough digital PR, or making 'enough of a difference to the web' to rank highly for category related terms. Certainly the site looks very drab and 'cookie-cutter' in terms of the template. It doesn't instil a sense of pride in the business behind the website. That can put people off linking to you, which can cause your SEO authority to fall flat on its face leaving you with no ranking power.

A lot of the product images look as if they are fake which probably isn't helping. They actually look at lot like ads which often look a bit cartoony or CGI-generated, with a balance between blue and white (colour deployment). Maybe they're being misinterpreted as spam due to Google PLA (Page Layout Algorithm). Design is not helping you out at all I am afraid!

So who is ranking for MDF skirting board? The top non-PPC (ad-based) result on Google.co.uk is this one:

https://skirtingboardsdirect.com/products/category/mdf-skirting-boards/

Ok so their content is better and deeper than yours (bullet-pointed specs or stats often imply 'granular' content to Google, which Google really likes - your content is just one solid paragraph). Overall though, I'd actually say their design is awful! It's worse than the design of your site (so maybe design isn't such a big factor here after all).

Let's compare some top-line SEO authority metrics on your site against those earned by this competitor

skirtinguk.com

• Domain Authority from Moz: 24
• Referring Domains from Moz: 123
• Ahrefs Homepage URL Rating: 11
• Ahrefs Domain Rating: 11
• Ahrefs Referring Domains: 149
• Majestic SEO Trust Flow: 5
• Majestic SEO Citation Flow: 24

Now the other site...

skirtingboardsdirect.com

• Domain Authority from Moz: 33 (+9)
• Referring Domains from Moz: 464 (+341)
• Ahrefs Homepage URL Rating: 31 (+20)
• Ahrefs Domain Rating: 65 (+54)
• Ahrefs Referring Domains: 265 (+116)
• Majestic SEO Trust Flow: 29 (+24)
• Majestic SEO Citation Flow: 30 (+6)

They beat you in all the important areas! That's not good.

Your category-level URLs aren't Meta no indexed, or blocked in the robots.txt file. Since we have found evidence that Google are in fact indexing your category level URLs, it's actually a ranking / authority problem, cleverly disguised as an indexation issue (I can see why you assumed that). These pages aren't **good enough **to be frequently indexed by Google, for keywords which they know hold lucrative financial value. Only the better sites (or the more authoritative ones) will rank there

A main competitor has similar design standards but has slightly deeper content and much more SEO authority than you do. The same is probably true for other competing sites. In SEO, you have to fight to maintain your positions. Sitting back is equivalent to begging your competitors to steal all of your traffic...

Hope this analysis helps!