HUNTING MALWARE in the vast virtual planes of the internet and amid the thousands of files on even the most basic PC is a challenge for the best cyber security boffins.
Threat researchers can usually be found at cyber security companies sifting through data, analysing files and tracing the origins of malware down rabbit warrens of scripts, file paths and rogue code hiding behind seemingly harmless executable files. Once malware is found it can be squashed and measures can be put into place to protect other systems from similar attacks.
This 'threat intelligence' is an important part of protecting individuals and organisations against software vulnerabilities, viruses and hackers.
“These services prioritise vulnerabilities and predict threats, enabling security
teams to rapidly take action. More advanced services also integrate vulnerability alerting with real-world threat intelligence covering geopolitical and business intelligence,” said the UK Computer Emergency Response Team.
As important as it is, threat intelligence takes a fair whack of effort. There are threat tools on the market that dramatically simplify the process of unravelling and responding to attacks, but the process can still be time-consuming and resource-intensive. Enter machine learning.
Machine learning was pretty much the precursor to artificial intelligence and can be used in threat intelligence to sniff out malware and malicious code far more quickly than humans.
“Machine learning and automation is kind of the only way to address threat intelligence because the volume of the threats and the vastness of the internet make the problem too big for human threat researchers or human classifiers,” Hal Lonas, chief technology officer at cloud-based security firm Webroot, told the INQUIRER.
Webroot’s own foray into machine learning came from its acquisitions of endpoint security specialist Prevx and threat intelligence firm BrightCloud, which the firm integrated into its own security and software-as-a-service products.
At its most basic Webroot’s BrightCloud threat intelligence is trained by researchers to identify a variety of threats ranging from malware-ridden files and dodgy URLs to suspicious IP addresses and malicious executables.
“The theory of machine learning is that you can teach machines to do classification by giving machines examples of things that are in the category and not in the category you’re looking for,” said Lonas.
“By taking those examples the machine can learn which parameters are indicative of malware and which are indicative of not malware.”
Machine learning models are built around this process and then applied to unknown files which in turn generates a score that indicates how likely a file is to contain malware.
Researchers then check the results for any mistakes the system makes, such as false positives or mistaking malware for something benign, and feed it back to the system, which learns from its mistakes as well as from classification parameters.
Keeping the human element
Lonas explained that Webroot’s machine learning acts as a combination of specialist threat intelligence knowledge and is not designed to replace human researchers.
“What we found is that we are able to incorporate the knowledge of the threat researchers into the machine by having them guide the learning of the machine. That’s a technique called guided learning,” he said.
This approach allows the system to carry out security researchers' everyday threat intelligence, leaving them to get their teeth into the really complex and obtuse threats that have yet to be defined or defeated. Lonas pointed out that this has the neat knock-on effect of keeping smart researchers engaged in their work.
“It turns the problem around and the threat researchers are completely engaged because they are solving the top problems,” he said. “They are doing the really tough things, which attracts and keeps the best researchers.”
So instead of the machines taking over form the humans, they work in tandem, putting at rest the fears that robots will render humans useless.
Learning to hunt
Machine learning on the surface is a simple case of teaching a system to react to predefined parameters and to identify and learn more classifications as it goes along.
But the sheer volume of threats aimed at machines and networks required a way to crunch data at speed and scale. Webroot has approached this problem in two ways.
The first is to use an evolution of support vector machines, a set of supervised learning methods often used in machine learning classification processes. Webroot took this basis and built on it using algorithms based on a technique called maximum entropy discrimination.
'Entropy' in this case refers to the vast amounts of dispersed data on the internet, in operating systems and systems loaded with software. 'Discrimination' and 'maximum' refer to the technique’s way of ignoring unimportant data and focusing only on elements that are relevant to threat finding.
Put simply, Webroot’s algorithms seek the best signal in a mass of noisy data and hone in on it. Filtering out these signals is a crucial part of threat intelligence, as malware never hides in plain sight.
“It turns out this malware stuff is a very noisy environment and usually the reason is the authors of the threats are trying to hide what they are doing,” said Lonas, noting how malicious software can be hidden in benign registries apps and files.
This technique allows Webroot to classify tens of thousands of threats a second, an order of magnitude more than even the best security research teams. At the same time, the human researchers keep building machine models based on new data and threats that make the system smarter as it goes along.
Crunching in the cloud
Ingesting and processing all this threat data understandably takes a solid lump of compute power, and raises the problem of carrying out threat intelligence at speed and scale. Webroot does the machine learning on the Amazon Web Services (AWS) cloud platform rather than requiring customers to have a load of spare servers to support the system.
The combination of AWS’ SQL column databases for high-speed data input and output and a global data centre footprint gives Webroot the scope to offer its services on an international level. It also takes the ‘heavy lifting’ of building and running machine learning algorithms off the shoulders of customers' infrastructures and machines.
This sets Webroot apart from other providers of threat intelligence services, but the clever part is the way that the company uses the cloud to share threat intelligence with all its customers. Lonas explained that Webroot harvests anonymous metadata on the threat parameters and pushes it to the cloud.
Around a million new files are processed by the system every day as Webroot collects data from a variety of sources, from endpoints in the Internet of Things to widespread networks and servers. The firm also absorbs threat data from its own customer-facing antivirus software that uses the threat intelligence to improve protection, which has created a form of perpetual data harvesting and consumption.
Data from multiple machines, users, companies and devices streams into one big cloud-based pot, and Webroot effectively learns from the malware its users encounter and puts that knowledge into action. So as more people use Webroot’s products and connect to its services the more effective they become.
Christina Richmond, programme director of security services at IDC, said that a growing collective mass of security endpoints and a pseudo-community of data sharing is at the heart of effective threat intelligence.
"Threat intelligence is essentially a community activity. Attack information can come from many different sources, and iterative intelligence organises this chaotic process of information sharing to help organisations make future decisions," she said.
Webroot is not alone in using machine learning to augment threat intelligence, but putting it in the cloud and effectively creating a feedback loop of data collection and consumption has put the firm in a unique position in the cyber security market.
But we can be pretty sure that other cyber security companies will dive into machine learning as the cyber attack vectors increase with the unstoppable march of internet-connected devices.
And perhaps a few decades down the line the tech will evolve into a full artificial intelligence capable of hunting threats in every dark corner of the internet and ending the battle against hackers forever. Then it rises up and enslaves us just like Steve Wozniack warned. µ
Oh and it'll also help give aural pleasure
But it might still not be enough to make virtual reality super appealing
And a ridiculous competition
Now you can talk to your silly-looking earbuds too