Screaming Frog tool left me stumped
-
Hi there again,
I found a major cloaking hack in our client's website that is really well camouflaged and all the seo tools that I tried to help me check for cloaking couldn't find it.
I know that screaming frog is a great tool and I want to use it to help me, however, I can't seem to get my way around their system that I downloaded.
Can you help me with the screaming frog program? Do you know where I can make a full site check for cloaking, maybe there are more links that I wasn't notified about?
I would really appreciate if you could help me with that.
Thanks so much, Ruchy
-
Hi Ruchy,
That's fantastic deep crawl is a great tool.
I was talking about two different types of cloaking one using an URL with a C name for hidden cloaking e.g. https://www.rebrandly.com/
Thank you for the kind words. Giving good answers and thumbs up or thumbs down definitely does make a difference I have outlined a lot of it below the official MozPoints URLhttp://moz.com/community/mozpoints
Yes marking anything as a giving thumbs-up is telling somebody they have helped you and in the event of thumbs up is their response was valuable and maybe contributed to answering your question.
In the case, of giving good answer person has answered your question giving as well as thumbs-up is the way anything is a way of saying thank you.
Welcome to the most community, and I look forward to seeing you here.
All the best,
Tom
-
Hi Tom
we finally solved the issue. Thanks for your clear answers, the info really helped me. We bought DeepCrawl and it really was helpful.
What was the C name you were talking about?
Thank you, Ruchy
P.S. i marked good answer by the wrong one, does it make a difference?
-
Hey just wanted to check was any answer helpfull to you?
Is there anything we might've missed or failed to help you solve the issue?
All the best,
Tom
-
-
Thanx Andy
That was a helpful piece of information.
I am still a little confused, can you give me some further advice?
I there any way for me to do a full site cloaking check?
Ruchy
-
Hi Ruchy,
Screaming Frog is an excellent tool. Dan is a great guy to update and the on that.
In my opinion, I have been able to find formation using https://www.deepcrawl.com/
The cost is higher deepcrawl is also a hosted platform.
Still, it can do remarkable things that no other tool that I know of can do.
There are built just for checking cloaking this one is free
http://www.seotools.com/seo-cloaking-checker/
If you have access to the DNS will want to examine this thoroughly.
I have laid out how you would find c name DNS records if you do not have access to the DNS as well.
One other method you might have to use to determine whether or not domain or subdomain has been cloaked as shown here: http://www.ghosturl.com/ if the cloaking happens in a manner described in the URL for this sentence.
So the scenario would be the cloaked URL would be something like
http://www.cloaked.example.com/?=ku2b4B30ijbasT47720sb534Nbq6
( please know I used example.com is the domain because I did not want to pointed that any live site not because of anything to do with cloaking)
Therefore they're using most likely a C name inside the DNS to pointed to http://www.example.com
You will need to use a tool that would be able to look at the IP or C name being created by the DNS.
The free tool I know if that can look for a C name or A record that should not be there is https://www. cloudflare.com the way you would do this is run the site through the DNS wizard that pulls the current DNS including all the records (it can miss some, but it does an excellent job of catching most of all DNS records. (9 times out of 10 will get all of them)
Because most tools make it impossible to see the C name when looking at DNS, it is important to remember there is a free service that is not designed for this purpose but can be used to discover the rogue cloaked URL.
https://www.cloudflare.com/a/sign-up (the free-tier offers the same ability to get DNS information)
http://i.imgur.com/Wm7W0aM.mp4
https://cl.ly/0A3k3d2j401n/Screen Recording 2017-01-26 at 07.09 AM.mov
http://i.imgur.com/lIN9Gsm.png
Hope This helps,
Tom
-
Dan from Screaming Frog has just had a look at this, and although he is on the move at the moment, he said this...
On the road to a meeting, but they can switch user-agent & change to JavaScript rendering mode to test for cloaking. Depends how it's cloaked, though. Some may do via referrer which you can't test for within the Spider (yet!). Will reply a bit later
-Andy
-
Hi there
You can spider the site with screaming frog and then do a bulk export of all the outlinks into a CSV and run through and see if there are links in there that shouldn't be
Have you downloaded the latest version?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
I Mistakenly uploaded Disavow File to Non WWW version of Webiste in Webmaster Tools...is this a Problem???
Hey guys and gals, I need some advice on this please. I recently had someone perform a negative S.E.O campaign on my site and I was inundated with 13,000 + spammy links pointing to my website and I had to perform a Disavow in Google Webmaster Tools but for some reason, it is showing that I uploaded the Disavow text file to the Non WWW version of the website but the WWW version of my website is the preferred domain and I have all NON WWW queries being 301 redirected to www.pcmedicsoncall.com My question is should I correct this and upload the Disavow text to the preferred domain in Google Webmaster Tools??? Please advise on how I should proceed with this situation.... Thank you. Cam
White Hat / Black Hat SEO | | CamMcArthur0 -
What is left ethical? What is working for offpage SEO? Very long write up in here and my take on things.
Hello,
White Hat / Black Hat SEO | | MarketingOfAmerica
Please ignore misspells and grammar, this was typed quickly as I am spending my time researching not writing a perfect book on it. My goal is to find ethical very hard to get links unlike guest posts which are now dead according to Matt Cutt's blog here http://www.mattcutts.com/blog/guest-blogging/. My journey started with a quick message to Rand Fishkin, he responded the following "Hi Matthew - thankfully, there's literally hundreds of link building methodologies that are still completely legit. Check out http://moz.com/blog/category/link-building and you'll find tons and tons of them. The key is that none are easy, none are particularly scalable, and all of them require doing work that will add value for searchers, for your brand, and for your overall marketing - which is exactly what Google wants to count. Wish you all the best," Thanks Rand Fishkin! So I started my search looking for links that are hard to get other than those that are directories, forum links that are dead and spammy, blog comments which are overused, guest posts, or any type of black hat link. I figured I would start to check what other popular SEO companies were doing and that have been at the top through many of the updates. After running an analysis on the term SEO services I found the following Test 1. I analyzed Main Street Host to start with. If you type in SEO services in Google you can see they are rank 1 for it. After a quick analysis it's easy to see that they have 100's of footer links on clients that they have, some with exact match anchors and some without. My question is, is why is this a viable tactic? Lets take for example the following. If you pull up their http://www.opensiteexplorer.org/ stats and look at the inbound links you will come to an exact match anchor right away that says SEO Marketing Company. I went to the weebly link that they have and found that they have put their name at the bottom of this page. Issue 1 - Why is it ok for this type of link, but it's not ok for a template link? Aren't these links suppose to be penalized? Issue 2 - Nothing on the page is even relevant to their link at all. As we have read before, you need to have links surrounding relevant text. Take a look at their backlinks you and you will find almost all of their high quality links are exact match anchors coming from their clients surrounded by irrelevant text. Why is this working? How is this different than a network? What stops someone from just starting a network and dedicating 1 footer link to a full site and putting up dummy info... Anyone can go to Godaddy and purchase a DA 40+ site or so and throw up $20 of content and a footer link. As I dove deeper into finding what is ethical and working I discovered many of the top SEO companies use this. Not just one, but over 20 of them use this same method. Lets use another example. So I started to look at what they did for their clients. How did I know who they worked for? Simple I assume that since they have their link at the bottom of the page and claim that they do SEO for them, they are indeed working for them. So I analyzed the site we talked about a while ago on the Weebly that they had their link on. It's the Valley Art Weebly link if your checking yourself. I quickly found that they are using a network to rank up some of their clients as well. For example http://firesidebookshop.com/index.html Take a look at the link on this page leading to the art place. At first glance the site doesn't look spammy, but try to buy a book, or even order one. Who has an online book store, but doesn't sell books lol? Who also puts interesting links on their home page? This screams network to me. I am willing to bet the following will happen - Matt Cutts and his spam team will ad something like the following to the algorithm or whatever you would like to call it "ignore link if total outbound dofollow links on full site = x amount or higher" = internal Google disavow tool = bye to guest blogging. So what is everyone going to do? Okay it's time to figure out what that number is right? Lets do some tests and lets say that magic number is 5 to 10 links on a whole site. What does this do? This drives the price of quick SEO up again evening the playing field for others using ethical SEO like myself. How do I figure this? Lets face it black hat SEO will never end as long as someone is able to do it. Now since guest posts are gone, the quick link on quality sites surrounded by enough text to count is gone. This means that it will cost extra money, because everyone will be forced to put a max of x amount of links to be safe and for the links to get noticed on a website. So now they have to purchase an established domain that is high enough quality to pass the correct link juice through to a clients site that they want to rank up. Lets figure a few dollars for a unique IP, another few for the hosting, $40 to $100 for the domain if your lucky on Godaddy auctions, and then $40 for the content to make it look realistic if your getting it for $0.01 a word. Plus the time it takes to setup your site. This price of that $30 Odesk guest post backlink just went up to a min of $100 or so. Diving deeper into what's working and moving past the networks, because I feel this will only work temporarily as well if you are brave enough to use this and I know I am not. It doesn't seem to ethical to me at the end of the day even though some may argue, you are just creating more relevant websites which can maximize your traffic streams. The problem is I have stopped here and am stuck. Sure I have looked at http://moz.com/blog/category/link-building and read the most recent post where it talks about 31 types of links. Most of those links don't apply or are outdated and you shouldn't use them. Some of them talk about forum links,directories, bookmarks.. Those have been tactics for years and sure you may find 1 out of 1000 that are good, but the rest are just spam. I have been over to search engine land, and a handful of other sites. I have talked to many other SEO's as well. They are emailing me asking what they should do after guest posts, because they are unsure. The question is, what is ethical? Let say you have a plumber, or a roofer, .gov links are nearly impossible for them and quite frankly that seems spammy to me to even post them on one. I know what many are going to say, build links as if your not worried about Google and you will grow.. Where are you going to build the links to if everything is unethical? As we know clients will walk if they don't see improvements quickly. What's quickly? I would say around the 3 to 6 month period using ethical SEO. Sure there is onpage, a great blog, etc., but what is there left truly ethical for offpage SEO besides some good press releases, some social profile links like a pinterst, and the normal? I must be missing something! I am not looking for the easy way, I am not afraid to get my hands dirty and work hard. If anyone can show me a quick example of a truly ethical link I would be grateful to see this. I can't seem to wrap my head around something that I can do that will last at this point. If you don't want to share it to the world, please PM me. [edited for formatting by Keri Morgret]0 -
Negative SEO and when to use to Dissavow tool?
Hi guys I was hoping someone could help me on a problem that has arisen on the site I look after. This is my first SEO job and I’ve had it about 6 months now. I think I’ve been doing the right things so far building quality links from reputable sites with good DA and working with bloggers to push our products as well as only signing up to directories in our niche. So our backlink profile is very specific with few spammy links. Over the last week however we have received a huge increase in backlinks which has almost doubled our linking domains total. I’ve checked the links out from webmaster tools and they are mainly directories or webstat websites like the ones below | siteinfo.org.uk deperu.com alestat.com domaintools.com detroitwebdirectory.com ukdata.com stuffgate.com | We’ve also just launched a new initiative where we will be producing totally new and good quality content 4-5 times a week and many of these new links are pointing to that page which looks very suspicious to me. Does this look like negative Seo to anyone? I’ve read a lot about the disavow tool and it seems people’s opinions are split on when to use it so I was wondering if anyone had any advice on whether to use it or not? It’s easy for me to identify what these new links are, yet some of them have decent DA so will they do any harm anyway? I’ve also checked the referring anchors on Ahrefs and now over 50% of my anchor term cloud are totally unrelated terms to my site and this has happened over the last week which also worries me. I haven’t seen any negative impact on rankings yet but if this carries on it will destroy my link profile. So would it be wise to disavow all these links as they come through or wait to see if they actually have an impact? It should be obvious to Google that there has been a huge spike in links so then the question is would they be ignored or will I be penalised. Any ideas? Thanks in advance Richard
White Hat / Black Hat SEO | | Rich_9950 -
Has anyone had experience with the Disavow Links tool? If so did you notice positive results from it?
I recently noticed a large number of backlinks from low authority directories coming in for one of my clients. These links were either purchased from a competitor or from a directory service site that knows we might be willing to pay to have bad links removed. I've contacted the website admin and they require a payment of $.30 per link to have them removed from their directory. Has anyone had a similar experience? I'm also considering using the disavow tool but I've heard the outcome of using this tool is usually bad. I'd appreciate any feedback, thanks!
White Hat / Black Hat SEO | | Leadhub1 -
When to NOT USE the disavow link tool
Im not here to say this is concrete and should never do this, and please if you disagree with me then lets discuss. One of the biggest things out there today especially after the second wave of Penguin (2.0) is the fear striken web masters who run straight to the disavow tool after they have been hit with Penguin or noticed a drop shortly after. I had a friend who's site who never felt the effects of Penguin 1.0 and thought everything was peachy. Then P2.0 hit and his rankings dropped of the map. I got a call from him that night and he was desperately asking me for help to review his site and guess what might have happened. He then tells me the first thing he did was compile a list of websites back linking to him that might be the issue and create his disavow list and submitted it. I asked him "How long did you research these sites before you came the conclusion they were the problem?" He Said "About an hour" Then I asked him "Did you receive a message in your Google Webmaster Tools about unnatural linking?" He Said "No" I said "Then why are you disavowing anything?" He Said "Um.......I don't understand what you are saying?" In reading articles, forums and even here in the Moz Q/A I tend to think there is some misconceptions about the disavow tool from Google that do not seem to be clearly explained. Some of my findings with the tool and when to use it is purely based on logic IMO. Let me explain When to NOT use the tool If you spent an hour reviewing your back link profile and you are to eager to wait any longer to upload your list. Unless you have less than 20 root domains linking to you, you should spend a lot more than an hour reviewing your back link profile You DID NOT receive a message from GWT informing you that you had some "unnatural" links Ill explain later If you spend a very short amount of time reviewing your back link profile. Did not look at each individual site linking to you and every link that exists, then you might be using it WAY TO SOON. The last thing you want to do is disavow a link that actually might be helping you. Take the time to really look at each link and ask your self this question (Straight from the Google Guidelines) "A good rule of thumb is whether you'd feel comfortable explaining what you've done to a website that competes with you, or to a Google employee" Studying your back link profile We all know when we have cheated. Im sure 99.9% of all of us can admit to it at one point. Most of the time I can find back links from sites and look right at the owner and ask him or her "You placed this back link didn't you?" I can see the guilt immediately in their eyes 🙂 Remember not ALL back links you generate are bad or wrong because you own the site. You need to ask yourself "Was this link necessary and does it apply to the topic at hand?", "Was it relevant?" and most important "Is this going to help other users?". These are some questions you can ask yourself before each link you place. You DID NOT receive a message about unnatural linking This is were I think the most confusing takes place (and please explain to me if I am wrong on this). If you did not receive a message in GWT about unnatural linking, then we can safely say that Google does not think you contain any "fishy" spammy links in which they have determined to be of a spammy nature. So if you did not receive any message yet your rankings dropped, then what could it be? Well it's still your back links that most likely did it, but its more likely the "value" of previous links that hold less or no value at all anymore. So obviously when this value drops, so does your rank. So what do I do? Build more quality links....and watch you rankings come back 🙂
White Hat / Black Hat SEO | | cbielich1 -
Manage two domains with Webmaster tools
Dear all, I want to create an additional domain in order to: Rank better for a very specific keyword with an exact match domain (I already ask about that, but I did not have my ideas clear at the time); Offer to the user usefull infomation about the topic, without duplicating the content I have in my main domain, just additional and very specific information; Use this domain as landing page, offering a tutorial on "how to" use a specific section of my main domain, including a video tutorial; Link to the related section of my main domain. So, the main idea is, if an user type in google "this specific keyword of ours", they will have in the results "thisspecifickeywordofus.es", they will click and go to the site, where they will find unique and specific information, complementing what I have in my mainsite, and showing how to use my site, so trying to use it for conversion. I want to do only white hat SEO, so first at all, I would like to ask you if you think it is a good idea. The keyword is difficult to rank for, and if I can take advantage of this exact match domain (even if it is nowdays no more so big an advantage), would be great. Second, do you see any problem in managing different domains from the same google account? Newbee question, sorry. Thanks in advance for your help, Daniel
White Hat / Black Hat SEO | | te_c0 -
Webmaster Tools Showing Bad Links Removed Over 60 Days Ago
Hello, One of my clients received the notorious message from Google about unnatural links late last March. We've removed several hundred (if not thousands) of links, and resubmitted several times for reconsideration, only to continue with responses that state that we still have unnatural links. Looking through the "links to your site" in google webmaster tools, there are several hundred sites / pages listed, from which we removed our link over 60 days ago. If you click each link to view the site / page, they contain nothing, viewable or hidden, regarding our website / address. I was wondering if this (outdated / inaccurate) list is the same as the one their employees use to analyze the current status of bad links, and if so how long it will take to reflect up-to-date information. In other words, even though we've removed the bad links, how long do we need to wait until we can expect a clean resubmission for reconsideration. Any help / advice would be greatly appreciated -
White Hat / Black Hat SEO | | Bromtec0 -
Are there tools out there to determine when a link linked to your site? I want to know when a link farm was done a site.
In Webmaster Tools I discovered that a client of mine with signed up for or hired another company to get links. The links are poor quality and from other countries, so it looks like a link farm was done. I want to know when they links were linked to the site, and not sure how to find that information out. Does anyone know how to find this out?
White Hat / Black Hat SEO | | StrategicEdgePartners0