Screaming Frog tool left me stumped
-
Hi there again,
I found a major cloaking hack in our client's website that is really well camouflaged and all the seo tools that I tried to help me check for cloaking couldn't find it.
I know that screaming frog is a great tool and I want to use it to help me, however, I can't seem to get my way around their system that I downloaded.
Can you help me with the screaming frog program? Do you know where I can make a full site check for cloaking, maybe there are more links that I wasn't notified about?
I would really appreciate if you could help me with that.
Thanks so much, Ruchy
-
Hi Ruchy,
That's fantastic deep crawl is a great tool.
I was talking about two different types of cloaking one using an URL with a C name for hidden cloaking e.g. https://www.rebrandly.com/
Thank you for the kind words. Giving good answers and thumbs up or thumbs down definitely does make a difference I have outlined a lot of it below the official MozPoints URLhttp://moz.com/community/mozpoints
Yes marking anything as a giving thumbs-up is telling somebody they have helped you and in the event of thumbs up is their response was valuable and maybe contributed to answering your question.
In the case, of giving good answer person has answered your question giving as well as thumbs-up is the way anything is a way of saying thank you.
Welcome to the most community, and I look forward to seeing you here.
All the best,
Tom
-
Hi Tom
we finally solved the issue. Thanks for your clear answers, the info really helped me. We bought DeepCrawl and it really was helpful.
What was the C name you were talking about?
Thank you, Ruchy
P.S. i marked good answer by the wrong one, does it make a difference?
-
Hey just wanted to check was any answer helpfull to you?
Is there anything we might've missed or failed to help you solve the issue?
All the best,
Tom
-
-
Thanx Andy
That was a helpful piece of information.
I am still a little confused, can you give me some further advice?
I there any way for me to do a full site cloaking check?
Ruchy
-
Hi Ruchy,
Screaming Frog is an excellent tool. Dan is a great guy to update and the on that.
In my opinion, I have been able to find formation using https://www.deepcrawl.com/
The cost is higher deepcrawl is also a hosted platform.
Still, it can do remarkable things that no other tool that I know of can do.
There are built just for checking cloaking this one is free
http://www.seotools.com/seo-cloaking-checker/
If you have access to the DNS will want to examine this thoroughly.
I have laid out how you would find c name DNS records if you do not have access to the DNS as well.
One other method you might have to use to determine whether or not domain or subdomain has been cloaked as shown here: http://www.ghosturl.com/ if the cloaking happens in a manner described in the URL for this sentence.
So the scenario would be the cloaked URL would be something like
http://www.cloaked.example.com/?=ku2b4B30ijbasT47720sb534Nbq6
( please know I used example.com is the domain because I did not want to pointed that any live site not because of anything to do with cloaking)
Therefore they're using most likely a C name inside the DNS to pointed to http://www.example.com
You will need to use a tool that would be able to look at the IP or C name being created by the DNS.
The free tool I know if that can look for a C name or A record that should not be there is https://www. cloudflare.com the way you would do this is run the site through the DNS wizard that pulls the current DNS including all the records (it can miss some, but it does an excellent job of catching most of all DNS records. (9 times out of 10 will get all of them)
Because most tools make it impossible to see the C name when looking at DNS, it is important to remember there is a free service that is not designed for this purpose but can be used to discover the rogue cloaked URL.
https://www.cloudflare.com/a/sign-up (the free-tier offers the same ability to get DNS information)
http://i.imgur.com/Wm7W0aM.mp4
https://cl.ly/0A3k3d2j401n/Screen Recording 2017-01-26 at 07.09 AM.mov
http://i.imgur.com/lIN9Gsm.png
Hope This helps,
Tom
-
Dan from Screaming Frog has just had a look at this, and although he is on the move at the moment, he said this...
On the road to a meeting, but they can switch user-agent & change to JavaScript rendering mode to test for cloaking. Depends how it's cloaked, though. Some may do via referrer which you can't test for within the Spider (yet!). Will reply a bit later
-Andy
-
Hi there
You can spider the site with screaming frog and then do a bulk export of all the outlinks into a CSV and run through and see if there are links in there that shouldn't be
Have you downloaded the latest version?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Apparent Bot Queries and Impressions in Webmaster Tools
I've been noticing some strange stats in Google Webmaster Tools for my forum, which has been getting spam queries with impressions and no clicks. See the queries in the attached images. This might be a motive for the spammers or scrapers. I set the date range to just 22 Aug - 22 Nov and I see very obviously the spike is due to impressions. Questions: What should/can I do? Is Google doing something about this? How to avoid this? o6gKB
White Hat / Black Hat SEO | | SameerBhatia0 -
Can the disavow tool INCREASE rankings?
Hi Mozzers, I have a new client who has some bad links in their profile that are spammy and should be disavowed. They rank on the first page for some longer tail keywords. However, we're aiming at shorter, well-known keywords where they aren't ranking. Will the disavow tool, alone, have the ability to increase rankings (assuming on-site / off-site signals are better than competition)? Thanks, Cole
White Hat / Black Hat SEO | | ColeLusby0 -
Tool to check google index status for backlinks?
I would like to check to see which backlink urls are indexed in Google. Is there a tool that can automate this work or will I have to do it manually?
White Hat / Black Hat SEO | | Choice0 -
What is left ethical? What is working for offpage SEO? Very long write up in here and my take on things.
Hello,
White Hat / Black Hat SEO | | MarketingOfAmerica
Please ignore misspells and grammar, this was typed quickly as I am spending my time researching not writing a perfect book on it. My goal is to find ethical very hard to get links unlike guest posts which are now dead according to Matt Cutt's blog here http://www.mattcutts.com/blog/guest-blogging/. My journey started with a quick message to Rand Fishkin, he responded the following "Hi Matthew - thankfully, there's literally hundreds of link building methodologies that are still completely legit. Check out http://moz.com/blog/category/link-building and you'll find tons and tons of them. The key is that none are easy, none are particularly scalable, and all of them require doing work that will add value for searchers, for your brand, and for your overall marketing - which is exactly what Google wants to count. Wish you all the best," Thanks Rand Fishkin! So I started my search looking for links that are hard to get other than those that are directories, forum links that are dead and spammy, blog comments which are overused, guest posts, or any type of black hat link. I figured I would start to check what other popular SEO companies were doing and that have been at the top through many of the updates. After running an analysis on the term SEO services I found the following Test 1. I analyzed Main Street Host to start with. If you type in SEO services in Google you can see they are rank 1 for it. After a quick analysis it's easy to see that they have 100's of footer links on clients that they have, some with exact match anchors and some without. My question is, is why is this a viable tactic? Lets take for example the following. If you pull up their http://www.opensiteexplorer.org/ stats and look at the inbound links you will come to an exact match anchor right away that says SEO Marketing Company. I went to the weebly link that they have and found that they have put their name at the bottom of this page. Issue 1 - Why is it ok for this type of link, but it's not ok for a template link? Aren't these links suppose to be penalized? Issue 2 - Nothing on the page is even relevant to their link at all. As we have read before, you need to have links surrounding relevant text. Take a look at their backlinks you and you will find almost all of their high quality links are exact match anchors coming from their clients surrounded by irrelevant text. Why is this working? How is this different than a network? What stops someone from just starting a network and dedicating 1 footer link to a full site and putting up dummy info... Anyone can go to Godaddy and purchase a DA 40+ site or so and throw up $20 of content and a footer link. As I dove deeper into finding what is ethical and working I discovered many of the top SEO companies use this. Not just one, but over 20 of them use this same method. Lets use another example. So I started to look at what they did for their clients. How did I know who they worked for? Simple I assume that since they have their link at the bottom of the page and claim that they do SEO for them, they are indeed working for them. So I analyzed the site we talked about a while ago on the Weebly that they had their link on. It's the Valley Art Weebly link if your checking yourself. I quickly found that they are using a network to rank up some of their clients as well. For example http://firesidebookshop.com/index.html Take a look at the link on this page leading to the art place. At first glance the site doesn't look spammy, but try to buy a book, or even order one. Who has an online book store, but doesn't sell books lol? Who also puts interesting links on their home page? This screams network to me. I am willing to bet the following will happen - Matt Cutts and his spam team will ad something like the following to the algorithm or whatever you would like to call it "ignore link if total outbound dofollow links on full site = x amount or higher" = internal Google disavow tool = bye to guest blogging. So what is everyone going to do? Okay it's time to figure out what that number is right? Lets do some tests and lets say that magic number is 5 to 10 links on a whole site. What does this do? This drives the price of quick SEO up again evening the playing field for others using ethical SEO like myself. How do I figure this? Lets face it black hat SEO will never end as long as someone is able to do it. Now since guest posts are gone, the quick link on quality sites surrounded by enough text to count is gone. This means that it will cost extra money, because everyone will be forced to put a max of x amount of links to be safe and for the links to get noticed on a website. So now they have to purchase an established domain that is high enough quality to pass the correct link juice through to a clients site that they want to rank up. Lets figure a few dollars for a unique IP, another few for the hosting, $40 to $100 for the domain if your lucky on Godaddy auctions, and then $40 for the content to make it look realistic if your getting it for $0.01 a word. Plus the time it takes to setup your site. This price of that $30 Odesk guest post backlink just went up to a min of $100 or so. Diving deeper into what's working and moving past the networks, because I feel this will only work temporarily as well if you are brave enough to use this and I know I am not. It doesn't seem to ethical to me at the end of the day even though some may argue, you are just creating more relevant websites which can maximize your traffic streams. The problem is I have stopped here and am stuck. Sure I have looked at http://moz.com/blog/category/link-building and read the most recent post where it talks about 31 types of links. Most of those links don't apply or are outdated and you shouldn't use them. Some of them talk about forum links,directories, bookmarks.. Those have been tactics for years and sure you may find 1 out of 1000 that are good, but the rest are just spam. I have been over to search engine land, and a handful of other sites. I have talked to many other SEO's as well. They are emailing me asking what they should do after guest posts, because they are unsure. The question is, what is ethical? Let say you have a plumber, or a roofer, .gov links are nearly impossible for them and quite frankly that seems spammy to me to even post them on one. I know what many are going to say, build links as if your not worried about Google and you will grow.. Where are you going to build the links to if everything is unethical? As we know clients will walk if they don't see improvements quickly. What's quickly? I would say around the 3 to 6 month period using ethical SEO. Sure there is onpage, a great blog, etc., but what is there left truly ethical for offpage SEO besides some good press releases, some social profile links like a pinterst, and the normal? I must be missing something! I am not looking for the easy way, I am not afraid to get my hands dirty and work hard. If anyone can show me a quick example of a truly ethical link I would be grateful to see this. I can't seem to wrap my head around something that I can do that will last at this point. If you don't want to share it to the world, please PM me. [edited for formatting by Keri Morgret]0 -
Access Denied - 2508 Errors - 403 Response code in webmaster tools
Hello Fellow members, From 9th may I am getting this error messages & these crawl errors is increasing daily. Google is not able to crawl my URLS & getting 403 response code & saying ACCESS Denied Errors in GWT. My all Indexed pages are de-indexed. Why I am receiving this errors ? My website is working fine but why Google is not able to crawl my pages. PLEASE TELL ME what is the ISSUE, I need to resolve ASAP on 9th may I got a message in GWT as well for "http://www.mysitename.co.uk/ Increase in authorization permission errors " Google detected a significant increase in the number of URLs we were blocked from crawling due to authorization permission errors. After this all problem started. Kindly tell what is the issue & how can I solve this. WGsu8pU
White Hat / Black Hat SEO | | sourabhrana390 -
Potential Implications of using the Disavow tool to remove thousands of links
So here's the situation. My companies site has over 30 thousand backlinks from Rippling.info These links all point to 3 product pages, some of which are no longer in production. Apparently a former employee was experimenting with some link farm ideas. My questions are; 1. does anyone here have experience with rippling.info? Is it legit? It seems like a link farm but Google allows adsense ads??? I thought Google was against link farms... 2. if I use the Disavow tool in Webmaster Tools to tell Google these 30k+ incoming links are to be ignored, will there be any consequences? -Google Analytics shows zero referral traffic since jan 1st 2012.
White Hat / Black Hat SEO | | mjmorse0 -
Manage two domains with Webmaster tools
Dear all, I want to create an additional domain in order to: Rank better for a very specific keyword with an exact match domain (I already ask about that, but I did not have my ideas clear at the time); Offer to the user usefull infomation about the topic, without duplicating the content I have in my main domain, just additional and very specific information; Use this domain as landing page, offering a tutorial on "how to" use a specific section of my main domain, including a video tutorial; Link to the related section of my main domain. So, the main idea is, if an user type in google "this specific keyword of ours", they will have in the results "thisspecifickeywordofus.es", they will click and go to the site, where they will find unique and specific information, complementing what I have in my mainsite, and showing how to use my site, so trying to use it for conversion. I want to do only white hat SEO, so first at all, I would like to ask you if you think it is a good idea. The keyword is difficult to rank for, and if I can take advantage of this exact match domain (even if it is nowdays no more so big an advantage), would be great. Second, do you see any problem in managing different domains from the same google account? Newbee question, sorry. Thanks in advance for your help, Daniel
White Hat / Black Hat SEO | | te_c0 -
Should we add our site to Google Webmaster Tools
Hello, Should we add our site nlpca(dot)com to google webmaster tools? Everything's very white hat but we do have a section on each of our 4 sites for "Our other Sites" that link to the others. It's been there for many years. We're looking for clues as to why we've dropped in rank Thanks!
White Hat / Black Hat SEO | | BobGW0