Welcome to the Q&A Forum

Hello Michael Johnson and Mozzers,

I have seen Shopify do this a few times, though I do not have clients on that particular platform at the moment. It is frustrating. You're right to want to resolve this issue. Between duplicate content, authority conflicts, and an inflated crawl budget, one issue or another is bound to hold back site performance.

Is this what you would do? Not immediately, no. I want to see those pages canonicalized. That way, your preferred pages get all the juice back from their respective canonical link. Is this an option for you?

**Deindex request... and s_ide effects?**_ Canonical tags would make these part irrelevant (yay less work!). To be thorough though: I'd let Google figure it out unless you have strong evidence your crawl budget is maxed. And I don't see any negative side effects from noindexing duplicate content. If worse comes to worse, you have a good plan.

Shape that content,
CopyChrisSEO and the Vizergy Team

You are most welcome. I'm glad to hear your road to site recovery is coming along. I'm also glad to confirm that, to all of my knowledge, your understanding of the "*" operator and Disallow /?spam string is correct. One more thing:

Fetch as Google and Request Indexing
Apologies, I neglected to mention this step in my answer. It should be included. This is the best tool I'm aware of to ask Google, "hey, crawl me please." Do this after you upload your shiny new robots.txt.

In GSC, under Crawl, select Fetch as Google. Then, select Fetch and Render. When status is partial or complete, click Request Indexing. There is no guarantee here, and my experience is Google does what it wants. Even so, I've seen results in less than 2 hours (full disclosure: the longest I've waited has been 3 days).

Penalty Free I agree. They cannot possibly be penalizing your site. At least, not purposefully. You have taken all recommended actions and then some to resolve site issues. Even if you do have a few bad back links floating around out there from some blackhat t3 site PBN, Penguin 4.0 should discredit that bad link juice. Your site doesn't even have the offending pages. It's just a matter of time before Google's index lines back up with your live site.

Good Work Sir,
Wipe the Index Clean,
CopyChrisSEO and the Vizergy Team

Hello Miamirealestatetrendsguy and fellow Mozers,

It sounds like you have had a crazy time handling this hack. Good news is, as far as I can tell from the given information, you are close to resolution. Googlebot should correct the indexed pages over time. I'm certain you would like to expedite that process. Here are three recommendations that come to mind: Remove URLs via GSC, block the offending URLs via robots.txt, and suggest edits in Google's SERPs.

Remove URLs via GSC
In GSC, under Google Index, select Remove URLs. This suppression is temporary however. Click on more information for more about that. My experience with it as been suppression for a few months. Don't worry about the time though. Our next step should take affect before your time is up.

Block the Offending URLs via Robots.txt
Before you do this, be very certain what you are doing. After you are confident, list your offending URLs, edit the offending URLs as noindex nofollow in your robots.txt, and upload it. Hopefully, you can find commonalities to shorten this list and save your time.

Note: I have purposefully avoided the details on how to this here because it is vital SEOs learn how to do it with full knowledge of potential risks as well as how to avoid those risks. Here are some resources:
          • Google Support           • Moz's Robots.txt Rundown 
          • Search Engine Land's Deeper Look

Suggest Edits in Google's SERPs This one is iffy, and I really don't trust Google using this feedback. However, I have done it and it worked more than once. Find your offending results and send specific feedback.

Wipe that Index Clean,
CopyChrisSEO and the Vizergy Team

You are most welcome. And thank you for the followup.

You are right. More results are better. The point I meant to establish was that OSE is not bad even though it could be better. It, like the other tools, provides an accurate slice of a site's back links. The size of that slice depends on the tool's resources.

Equipped to Act Having back link profiles from these tools leaves us equipped to act. I said this in my answer as a blanket statement for link building, disavowing*, content development, and outreach. Here are some quick examples:

Link building — If OSE (or any tool) returns 4 domains for you and 150 for your competition, It's time to get some links. Plug in some competitors to see if there are any gaps you can target, hit up Fresh Web Mentions to focus your outreach, and explore content opportunities.

Disavow* — If your tool returns 4 root domains, 15 total links and one of those is complete spam level 10, I'd probably want to disavow* it and audit my site for more spammy links using GSC. I'd know from the slice of back links Moz provided that my site was likely to have more.

Footnote on Disavowing* Penguin 4.0 has presented some changes for us as to the when and what to disavow. I have not had a recent and significant enough experience with spammy back links to formulate my response yet. If you find yourself the victim of bad back links, do some research before you throw the SEO book at the issue. It might save time.

Cultivate those links,
CopyChrisSEO and the Vizergy Team

Hello Eric and fellow Mozers,

Yes, I believe you should... probably*. Even though you have invested your time to review all your business listings, Moz local goes a step further. It pushes your listing information out to the main data aggregators. Most businesses and SEOs (myself included) are not able to consistently influence this data for a few reasons, and consistency is key.

Moz Local helps keep the aggregators' data consistent. Without it or your continued time investment, aggregators like Infogroup and Localeze will eventually change your listing information. That bad data will then, in turn, be disseminated to your edited listings.

*Probably?
The effect data aggregators have on your listing varies. I believe this depends on your location and how many times that location's data has changed over time. You might be fairly far removed from these changes if you are well established or a new business in a newly developed area. Still, the cost is so negligible. Just do it and save yourself time.

Why Aggregators Change Our Listing Data
I've not had a unbiased answer to why this happens. The best answer I've heard (in my opinion because I've seen correlations that back it up) is from Vizergy's rep at Yext*. They say, aggregators switch or rollback listing info when they lack information or are presented with strong, conflicting data. Most often, the published data is reverted back to older, more verifiable data. This often winds up being a relatively data phone-book entry.

*Yext and a quick note on bias. I'm not personally bias to Yext or Moz Local. I like what works for my clients' needs. However, Yext is bias to their own product, and it is in their best interest to cast doubt on data aggregators.

Keep it consistent,
CopyChrisSEO and the Vizergy Team

Hello TaylorRHawkins and fellow Mozers,

I have had the pleasure of using all sorts of backlink tools, including OSE and RavenTools. In doing so, I've run into this issue. Clients notice this difference too and frequently bring it to my attention. Here's what I say: These are different tools with different limitations. They give different results because of these limitations, which always come down to resources. Whether it's the software, indexing process, or bots, all of these tools are limited by their resources.

Tools with larger resources typically return more results. However, even more well-funded tools, including Ahref's beefy bot and index, won't get them all. You can see this in action by comparing your site's links—as reported per your chosen tool—to Google Search Console, which Google has pinky-promised us is the full list. You'll see the difference right away.

So is this a bad thing? No, not at all.

The tools I've used (OSE, RavenTools, Ahrefs, SEM Rush, and Brightedge) all do a pretty great job of delivering an accurate slice of my competitor's backlinks across the internet. Take this data and extrapolate the backlink profile using other metrics like domain authority. You'll get a good picture of what they have—and be equipped to act on it—even if you can only see 1/20th of their backlink pie.

Keep crawling,
CopyChrisSEO and the Vizergy Team

Hello Eric and fellow Mozers,

Yes, I believe you should... probably*. Even though you have invested your time to review all your business listings, Moz local goes a step further. It pushes your listing information out to the main data aggregators. Most businesses and SEOs (myself included) are not able to consistently influence this data for a few reasons, and consistency is key.

Moz Local helps keep the aggregators' data consistent. Without it or your continued time investment, aggregators like Infogroup and Localeze will eventually change your listing information. That bad data will then, in turn, be disseminated to your edited listings.

*Probably?
The effect data aggregators have on your listing varies. I believe this depends on your location and how many times that location's data has changed over time. You might be fairly far removed from these changes if you are well established or a new business in a newly developed area. Still, the cost is so negligible. Just do it and save yourself time.

Why Aggregators Change Our Listing Data
I've not had a unbiased answer to why this happens. The best answer I've heard (in my opinion because I've seen correlations that back it up) is from Vizergy's rep at Yext*. They say, aggregators switch or rollback listing info when they lack information or are presented with strong, conflicting data. Most often, the published data is reverted back to older, more verifiable data. This often winds up being a relatively data phone-book entry.

*Yext and a quick note on bias. I'm not personally bias to Yext or Moz Local. I like what works for my clients' needs. However, Yext is bias to their own product, and it is in their best interest to cast doubt on data aggregators.

Keep it consistent,
CopyChrisSEO and the Vizergy Team

Hello TaylorRHawkins and fellow Mozers,

I have had the pleasure of using all sorts of backlink tools, including OSE and RavenTools. In doing so, I've run into this issue. Clients notice this difference too and frequently bring it to my attention. Here's what I say: These are different tools with different limitations. They give different results because of these limitations, which always come down to resources. Whether it's the software, indexing process, or bots, all of these tools are limited by their resources.

Tools with larger resources typically return more results. However, even more well-funded tools, including Ahref's beefy bot and index, won't get them all. You can see this in action by comparing your site's links—as reported per your chosen tool—to Google Search Console, which Google has pinky-promised us is the full list. You'll see the difference right away.

So is this a bad thing? No, not at all.

The tools I've used (OSE, RavenTools, Ahrefs, SEM Rush, and Brightedge) all do a pretty great job of delivering an accurate slice of my competitor's backlinks across the internet. Take this data and extrapolate the backlink profile using other metrics like domain authority. You'll get a good picture of what they have—and be equipped to act on it—even if you can only see 1/20th of their backlink pie.

Keep crawling,
CopyChrisSEO and the Vizergy Team

Hello Miamirealestatetrendsguy and fellow Mozers,

It sounds like you have had a crazy time handling this hack. Good news is, as far as I can tell from the given information, you are close to resolution. Googlebot should correct the indexed pages over time. I'm certain you would like to expedite that process. Here are three recommendations that come to mind: Remove URLs via GSC, block the offending URLs via robots.txt, and suggest edits in Google's SERPs.

Remove URLs via GSC
In GSC, under Google Index, select Remove URLs. This suppression is temporary however. Click on more information for more about that. My experience with it as been suppression for a few months. Don't worry about the time though. Our next step should take affect before your time is up.

Block the Offending URLs via Robots.txt
Before you do this, be very certain what you are doing. After you are confident, list your offending URLs, edit the offending URLs as noindex nofollow in your robots.txt, and upload it. Hopefully, you can find commonalities to shorten this list and save your time.

Note: I have purposefully avoided the details on how to this here because it is vital SEOs learn how to do it with full knowledge of potential risks as well as how to avoid those risks. Here are some resources:
          • Google Support           • Moz's Robots.txt Rundown 
          • Search Engine Land's Deeper Look

Suggest Edits in Google's SERPs This one is iffy, and I really don't trust Google using this feedback. However, I have done it and it worked more than once. Find your offending results and send specific feedback.

Wipe that Index Clean,
CopyChrisSEO and the Vizergy Team

You are most welcome. I'm glad to hear your road to site recovery is coming along. I'm also glad to confirm that, to all of my knowledge, your understanding of the "*" operator and Disallow /?spam string is correct. One more thing:

Fetch as Google and Request Indexing
Apologies, I neglected to mention this step in my answer. It should be included. This is the best tool I'm aware of to ask Google, "hey, crawl me please." Do this after you upload your shiny new robots.txt.

In GSC, under Crawl, select Fetch as Google. Then, select Fetch and Render. When status is partial or complete, click Request Indexing. There is no guarantee here, and my experience is Google does what it wants. Even so, I've seen results in less than 2 hours (full disclosure: the longest I've waited has been 3 days).

Penalty Free I agree. They cannot possibly be penalizing your site. At least, not purposefully. You have taken all recommended actions and then some to resolve site issues. Even if you do have a few bad back links floating around out there from some blackhat t3 site PBN, Penguin 4.0 should discredit that bad link juice. Your site doesn't even have the offending pages. It's just a matter of time before Google's index lines back up with your live site.

Good Work Sir,
Wipe the Index Clean,
CopyChrisSEO and the Vizergy Team

You are most welcome. And thank you for the followup.

You are right. More results are better. The point I meant to establish was that OSE is not bad even though it could be better. It, like the other tools, provides an accurate slice of a site's back links. The size of that slice depends on the tool's resources.

Equipped to Act Having back link profiles from these tools leaves us equipped to act. I said this in my answer as a blanket statement for link building, disavowing*, content development, and outreach. Here are some quick examples:

Link building — If OSE (or any tool) returns 4 domains for you and 150 for your competition, It's time to get some links. Plug in some competitors to see if there are any gaps you can target, hit up Fresh Web Mentions to focus your outreach, and explore content opportunities.

Disavow* — If your tool returns 4 root domains, 15 total links and one of those is complete spam level 10, I'd probably want to disavow* it and audit my site for more spammy links using GSC. I'd know from the slice of back links Moz provided that my site was likely to have more.

Footnote on Disavowing* Penguin 4.0 has presented some changes for us as to the when and what to disavow. I have not had a recent and significant enough experience with spammy back links to formulate my response yet. If you find yourself the victim of bad back links, do some research before you throw the SEO book at the issue. It might save time.

Cultivate those links,
CopyChrisSEO and the Vizergy Team