Screaming Frog tool left me stumped
-
Hi there again,
I found a major cloaking hack in our client's website that is really well camouflaged and all the seo tools that I tried to help me check for cloaking couldn't find it.
I know that screaming frog is a great tool and I want to use it to help me, however, I can't seem to get my way around their system that I downloaded.
Can you help me with the screaming frog program? Do you know where I can make a full site check for cloaking, maybe there are more links that I wasn't notified about?
I would really appreciate if you could help me with that.
Thanks so much, Ruchy
-
Hi Ruchy,
That's fantastic deep crawl is a great tool.
I was talking about two different types of cloaking one using an URL with a C name for hidden cloaking e.g. https://www.rebrandly.com/
Thank you for the kind words. Giving good answers and thumbs up or thumbs down definitely does make a difference I have outlined a lot of it below the official MozPoints URLhttp://moz.com/community/mozpoints
Yes marking anything as a giving thumbs-up is telling somebody they have helped you and in the event of thumbs up is their response was valuable and maybe contributed to answering your question.
In the case, of giving good answer person has answered your question giving as well as thumbs-up is the way anything is a way of saying thank you.
Welcome to the most community, and I look forward to seeing you here.
All the best,
Tom
-
Hi Tom
we finally solved the issue. Thanks for your clear answers, the info really helped me. We bought DeepCrawl and it really was helpful.
What was the C name you were talking about?
Thank you, Ruchy
P.S. i marked good answer by the wrong one, does it make a difference?
-
Hey just wanted to check was any answer helpfull to you?
Is there anything we might've missed or failed to help you solve the issue?
All the best,
Tom
-
-
Thanx Andy
That was a helpful piece of information.
I am still a little confused, can you give me some further advice?
I there any way for me to do a full site cloaking check?
Ruchy
-
Hi Ruchy,
Screaming Frog is an excellent tool. Dan is a great guy to update and the on that.
In my opinion, I have been able to find formation using https://www.deepcrawl.com/
The cost is higher deepcrawl is also a hosted platform.
Still, it can do remarkable things that no other tool that I know of can do.
There are built just for checking cloaking this one is free
http://www.seotools.com/seo-cloaking-checker/
If you have access to the DNS will want to examine this thoroughly.
I have laid out how you would find c name DNS records if you do not have access to the DNS as well.
One other method you might have to use to determine whether or not domain or subdomain has been cloaked as shown here: http://www.ghosturl.com/ if the cloaking happens in a manner described in the URL for this sentence.
So the scenario would be the cloaked URL would be something like
http://www.cloaked.example.com/?=ku2b4B30ijbasT47720sb534Nbq6
( please know I used example.com is the domain because I did not want to pointed that any live site not because of anything to do with cloaking)
Therefore they're using most likely a C name inside the DNS to pointed to http://www.example.com
You will need to use a tool that would be able to look at the IP or C name being created by the DNS.
The free tool I know if that can look for a C name or A record that should not be there is https://www. cloudflare.com the way you would do this is run the site through the DNS wizard that pulls the current DNS including all the records (it can miss some, but it does an excellent job of catching most of all DNS records. (9 times out of 10 will get all of them)
Because most tools make it impossible to see the C name when looking at DNS, it is important to remember there is a free service that is not designed for this purpose but can be used to discover the rogue cloaked URL.
https://www.cloudflare.com/a/sign-up (the free-tier offers the same ability to get DNS information)
http://i.imgur.com/Wm7W0aM.mp4
https://cl.ly/0A3k3d2j401n/Screen Recording 2017-01-26 at 07.09 AM.mov
http://i.imgur.com/lIN9Gsm.png
Hope This helps,
Tom
-
Dan from Screaming Frog has just had a look at this, and although he is on the move at the moment, he said this...
On the road to a meeting, but they can switch user-agent & change to JavaScript rendering mode to test for cloaking. Depends how it's cloaked, though. Some may do via referrer which you can't test for within the Spider (yet!). Will reply a bit later
-Andy
-
Hi there
You can spider the site with screaming frog and then do a bulk export of all the outlinks into a CSV and run through and see if there are links in there that shouldn't be
Have you downloaded the latest version?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Is a recent hack or the disavow tool causing my alarming dropping in rankings!
My business site has been very successful organically for many years. Just recently we got hit with a spam hack and it was resolved within 3 days. However now my rankings are plummeting and I am so stressed out! So here is some timeline information any info would help: Sept. 4th hack first detected on Google Sept. 7th site completely clean, reconsideration accepted, spam content and links removed. Manual actions cleared. Rankings at this time have not been affected. Sept. 11th disavowed a few incoming links that were completely spam. (In hindsight I know this could have been the beginning of the end using this tool) Sept. 21st start to notice first significant drop in rankings and I went into GWT and downloaded latest 1000 links, I realized ALL of these were either hacked sites as well with spam content linking to our now delete spam content or inappropriate adult content. Sept. 22 Disavowed the 1000 domains (there are still probably 1000-2000 more) As of today rankings have SIGNIFICANTLY dropped, I have resubmitted sitemaps, image sitemaps, fetch and rendered as google. I'm stressing out incredibly and feel like I have made an error and that my site will never recover. I've worked using ALL white hat seo and the site used to rank very well top of page one for almost all my keywords. I feel lost and don't know what else I can do - and I know many say wait but it feels like forever. Is it possible that I didn't make a mistake using the disavow and that Google just took a while to penalize for the hack? Please any advice or experiences I would love to hear and appreciate so much anyone who takes the time to respond.
White Hat / Black Hat SEO | | seounicorn0 -
Submitting a page to Google Search Console or Bing Webmaster Tools with nofollow tags
Hello, I was hoping someone could help me understand if there is any point to submit a domain or subdomain to Google Search Console (Webmaster Tools) and Bing Webmaster Tools if the pages (on the subdomain for example) all have nofollow/noindex tags ... or the pages are being blocked by the robots.txt file). There are some pages on a data feed onto a subdomain which I manage that have these above characteristics ... which I cannot change ... but I am wondering if it is better to simply exclude from submitting those from GWT and BWT (above) thereby eliminating generating errors or warnings ... or is it better to tell Google and Bing about them anyway then perhaps there is a chance those nofollow pages may be indexed/contextualised in some way, making it worth the effort? Many thanks!
White Hat / Black Hat SEO | | uworlds
Mark0 -
Why There is No link Data Available in my Webmaster Tools even the site has lots of links and webmastert tools account setup properly
i have few account in my webmaster tools that are not showing any link data even the has lots of links. i checked the setup and its everything is good. is some one tell me why there is no data coming through? Thanks
White Hat / Black Hat SEO | | OnlineAssetPartners1 -
What is left ethical? What is working for offpage SEO? Very long write up in here and my take on things.
Hello,
White Hat / Black Hat SEO | | MarketingOfAmerica
Please ignore misspells and grammar, this was typed quickly as I am spending my time researching not writing a perfect book on it. My goal is to find ethical very hard to get links unlike guest posts which are now dead according to Matt Cutt's blog here http://www.mattcutts.com/blog/guest-blogging/. My journey started with a quick message to Rand Fishkin, he responded the following "Hi Matthew - thankfully, there's literally hundreds of link building methodologies that are still completely legit. Check out http://moz.com/blog/category/link-building and you'll find tons and tons of them. The key is that none are easy, none are particularly scalable, and all of them require doing work that will add value for searchers, for your brand, and for your overall marketing - which is exactly what Google wants to count. Wish you all the best," Thanks Rand Fishkin! So I started my search looking for links that are hard to get other than those that are directories, forum links that are dead and spammy, blog comments which are overused, guest posts, or any type of black hat link. I figured I would start to check what other popular SEO companies were doing and that have been at the top through many of the updates. After running an analysis on the term SEO services I found the following Test 1. I analyzed Main Street Host to start with. If you type in SEO services in Google you can see they are rank 1 for it. After a quick analysis it's easy to see that they have 100's of footer links on clients that they have, some with exact match anchors and some without. My question is, is why is this a viable tactic? Lets take for example the following. If you pull up their http://www.opensiteexplorer.org/ stats and look at the inbound links you will come to an exact match anchor right away that says SEO Marketing Company. I went to the weebly link that they have and found that they have put their name at the bottom of this page. Issue 1 - Why is it ok for this type of link, but it's not ok for a template link? Aren't these links suppose to be penalized? Issue 2 - Nothing on the page is even relevant to their link at all. As we have read before, you need to have links surrounding relevant text. Take a look at their backlinks you and you will find almost all of their high quality links are exact match anchors coming from their clients surrounded by irrelevant text. Why is this working? How is this different than a network? What stops someone from just starting a network and dedicating 1 footer link to a full site and putting up dummy info... Anyone can go to Godaddy and purchase a DA 40+ site or so and throw up $20 of content and a footer link. As I dove deeper into finding what is ethical and working I discovered many of the top SEO companies use this. Not just one, but over 20 of them use this same method. Lets use another example. So I started to look at what they did for their clients. How did I know who they worked for? Simple I assume that since they have their link at the bottom of the page and claim that they do SEO for them, they are indeed working for them. So I analyzed the site we talked about a while ago on the Weebly that they had their link on. It's the Valley Art Weebly link if your checking yourself. I quickly found that they are using a network to rank up some of their clients as well. For example http://firesidebookshop.com/index.html Take a look at the link on this page leading to the art place. At first glance the site doesn't look spammy, but try to buy a book, or even order one. Who has an online book store, but doesn't sell books lol? Who also puts interesting links on their home page? This screams network to me. I am willing to bet the following will happen - Matt Cutts and his spam team will ad something like the following to the algorithm or whatever you would like to call it "ignore link if total outbound dofollow links on full site = x amount or higher" = internal Google disavow tool = bye to guest blogging. So what is everyone going to do? Okay it's time to figure out what that number is right? Lets do some tests and lets say that magic number is 5 to 10 links on a whole site. What does this do? This drives the price of quick SEO up again evening the playing field for others using ethical SEO like myself. How do I figure this? Lets face it black hat SEO will never end as long as someone is able to do it. Now since guest posts are gone, the quick link on quality sites surrounded by enough text to count is gone. This means that it will cost extra money, because everyone will be forced to put a max of x amount of links to be safe and for the links to get noticed on a website. So now they have to purchase an established domain that is high enough quality to pass the correct link juice through to a clients site that they want to rank up. Lets figure a few dollars for a unique IP, another few for the hosting, $40 to $100 for the domain if your lucky on Godaddy auctions, and then $40 for the content to make it look realistic if your getting it for $0.01 a word. Plus the time it takes to setup your site. This price of that $30 Odesk guest post backlink just went up to a min of $100 or so. Diving deeper into what's working and moving past the networks, because I feel this will only work temporarily as well if you are brave enough to use this and I know I am not. It doesn't seem to ethical to me at the end of the day even though some may argue, you are just creating more relevant websites which can maximize your traffic streams. The problem is I have stopped here and am stuck. Sure I have looked at http://moz.com/blog/category/link-building and read the most recent post where it talks about 31 types of links. Most of those links don't apply or are outdated and you shouldn't use them. Some of them talk about forum links,directories, bookmarks.. Those have been tactics for years and sure you may find 1 out of 1000 that are good, but the rest are just spam. I have been over to search engine land, and a handful of other sites. I have talked to many other SEO's as well. They are emailing me asking what they should do after guest posts, because they are unsure. The question is, what is ethical? Let say you have a plumber, or a roofer, .gov links are nearly impossible for them and quite frankly that seems spammy to me to even post them on one. I know what many are going to say, build links as if your not worried about Google and you will grow.. Where are you going to build the links to if everything is unethical? As we know clients will walk if they don't see improvements quickly. What's quickly? I would say around the 3 to 6 month period using ethical SEO. Sure there is onpage, a great blog, etc., but what is there left truly ethical for offpage SEO besides some good press releases, some social profile links like a pinterst, and the normal? I must be missing something! I am not looking for the easy way, I am not afraid to get my hands dirty and work hard. If anyone can show me a quick example of a truly ethical link I would be grateful to see this. I can't seem to wrap my head around something that I can do that will last at this point. If you don't want to share it to the world, please PM me. [edited for formatting by Keri Morgret]0 -
Website not listing in google - screaming frog shows 500 error? What could the issue be?
Hey, http://www.interconnect.org.uk/ - the site seems to load fine, but for some reason the site is not getting indexed. I tried running the site on screaming frog, and it gives a 500 error code, which suggests it can't access the site? I'm guessing this is the same problem google is having, do you have any ideas as to why this may be and how I can rectify this? Thanks, Andrew
White Hat / Black Hat SEO | | Heehaw0 -
Negative SEO and when to use to Dissavow tool?
Hi guys I was hoping someone could help me on a problem that has arisen on the site I look after. This is my first SEO job and I’ve had it about 6 months now. I think I’ve been doing the right things so far building quality links from reputable sites with good DA and working with bloggers to push our products as well as only signing up to directories in our niche. So our backlink profile is very specific with few spammy links. Over the last week however we have received a huge increase in backlinks which has almost doubled our linking domains total. I’ve checked the links out from webmaster tools and they are mainly directories or webstat websites like the ones below | siteinfo.org.uk deperu.com alestat.com domaintools.com detroitwebdirectory.com ukdata.com stuffgate.com | We’ve also just launched a new initiative where we will be producing totally new and good quality content 4-5 times a week and many of these new links are pointing to that page which looks very suspicious to me. Does this look like negative Seo to anyone? I’ve read a lot about the disavow tool and it seems people’s opinions are split on when to use it so I was wondering if anyone had any advice on whether to use it or not? It’s easy for me to identify what these new links are, yet some of them have decent DA so will they do any harm anyway? I’ve also checked the referring anchors on Ahrefs and now over 50% of my anchor term cloud are totally unrelated terms to my site and this has happened over the last week which also worries me. I haven’t seen any negative impact on rankings yet but if this carries on it will destroy my link profile. So would it be wise to disavow all these links as they come through or wait to see if they actually have an impact? It should be obvious to Google that there has been a huge spike in links so then the question is would they be ignored or will I be penalised. Any ideas? Thanks in advance Richard
White Hat / Black Hat SEO | | Rich_9950 -
Potential Implications of using the Disavow tool to remove thousands of links
So here's the situation. My companies site has over 30 thousand backlinks from Rippling.info These links all point to 3 product pages, some of which are no longer in production. Apparently a former employee was experimenting with some link farm ideas. My questions are; 1. does anyone here have experience with rippling.info? Is it legit? It seems like a link farm but Google allows adsense ads??? I thought Google was against link farms... 2. if I use the Disavow tool in Webmaster Tools to tell Google these 30k+ incoming links are to be ignored, will there be any consequences? -Google Analytics shows zero referral traffic since jan 1st 2012.
White Hat / Black Hat SEO | | mjmorse0 -
Should we add our site to Google Webmaster Tools
Hello, Should we add our site nlpca(dot)com to google webmaster tools? Everything's very white hat but we do have a section on each of our 4 sites for "Our other Sites" that link to the others. It's been there for many years. We're looking for clues as to why we've dropped in rank Thanks!
White Hat / Black Hat SEO | | BobGW0