Can anyone suggest ...
How to detect or prevent malware in encrypted traffic without depending on a security tool
I am aware that Cisco comes with Encrypted Traffic Analytics (ETA), which monitors network packet metadata to detect malicious traffic even if its encrypted , but i would like to know any other suggestions for detection and prevention control
You could also choose to decrypt traffic at edge firewall, or at the IPS/web proxy in the middle.
Do take consideration on the additional resource overhead it will put on the device so it wont affect its original function.
With the majority of web and Internet traffic encrypted, you are right to be concerned about it being an avenue to malware.
Most web proxy filters (Bluecoat, Zscaler) and firewalls will also filter web pages that are from known bad sites without decryption. Unless you are willing/able to decrypt the communications (e.g using man-in-the-middle techniques) you are pretty much limited to site-level reputation filtering, (e.g. known bad site, young DNS registration, caught hosting malware, etc). With decryption, it becomes possible to delve deeper, such as allowing chat, but to denying file transfer; and also to AV scan individual files. For example, Facebook is "social networking", but with decryption, you get the ability to allow chat while blocking file transfer.
Also useful is to watch for hosts going to known malware"phone home" sites. This gives an indication of which hosts may already be infected and need remediation.
For now, I have only seen Cisco ETA really working when it comes to encrypted traffic analysis. Although some other vendors claimed the ability to detect 0-day threads (e.g. Darktrace or Greycortex) from our testing the best results came from Cisco Stealthwatch with ETA.
Some other options tough are able to do partial job in malware protection as well. But the majority of the function is based on reputation database for destination IPs/domains... (what is unable to discover malware communicating to twitter or instagram for example).
So aside the Cisco ETA, you basically have another two good options:
1. enhance end-point protection, where the communication is initiated and the payload is processed unencrypted (Cisco AMP for endpoints does great job as it is tracking all operations and communication of the endpoint and the infection can be even discovered afterwards and you posses great data for retrospective analysis).
2. implement decryption at proxy in order to inspect payload.
solution at the end point / proxy seems a good choice but, decrypting the traffic has a an impact in terms of time, performance and cost and in some areas is simply not possible because the necessary cryptographic keys aren't available.
As others here have mentioned decryption at the edge device is a great way to accomplish this. If you go this route keep the following things in mind:
* there is an impact to the througput/speed (although we have found it to be unnoticeable with the properly sized hardware)
* You will want to take into consideration things that you should *not* decrypt (HIPPA, etc)
* Thick client apps that use certificate pinning / hard coded certs will not play nice and will end up requiring exceptions