Back to blog
securitymachine-learningnlplinuxresearch

Surviving the Evil MSc: NLP and machine learning for anomaly detection on system call logs

15 December 20224 min read

It's done. I've completed my Master of Science in Computer Science, specialising in Information Security, at Rhodes University. Among those who have been through it, the programme is affectionately known as the Evil MSc, and having done it while working full time, I can confirm the name is earned.

My thesis is titled Natural Language Processing with Machine Learning for Anomaly Detection on System Call Logs.

The idea

Every process on a Linux machine talks to the kernel through system calls: open, read, write, execve, connect and so on. A trace of those calls is a pretty honest record of what a program is actually doing, regardless of what it claims to be.

A sequence of system calls also looks a lot like a sentence. There is a vocabulary (the set of calls), there is ordering, and there are patterns that are "normal" for a given program. That makes it a natural fit for techniques from natural language processing (NLP), such as treating a trace as a bag of words, or more specifically a bag of system calls (BoSC), and feeding those representations into machine learning models.

The research set out to:

  • apply NLP techniques to raw system call traces from 64-bit x86 GNU/Linux,
  • use both supervised and unsupervised machine learning to identify current and previously unseen threats, and
  • ask a practical question: are these tools within reach of information security professionals, or do you need a dedicated data scientist to use them?

Data

Good, modern system call datasets are hard to come by. I used the Leipzig Intrusion Detection Data Set (LID-DS) from Leipzig University, a recent academic dataset of system call traces covering normal behaviour and attack scenarios. To get closer to real-world threats, I combined it with system call traces from executing Linux malware samples from the VirusTotal Academic dataset.

Models

I compared two decision-tree based approaches:

  • Random Forest, a supervised model trained on labelled normal and malicious behaviour.
  • Isolation Forest, an unsupervised model that learns what normal looks like and flags whatever is easiest to isolate, combined with Principal Component Analysis (PCA) to reduce the dimensionality of the feature space.

Each model was run with default parameters and again after hyperparameter tuning.

What I found

  • Isolation Forest with PCA was effective at identifying anomalies in the combined LID-DS and VirusTotal malware data. This matters because unsupervised detection doesn't need labelled attacks, which is exactly what you lack when facing something new.
  • Default parameters gave sub-optimal results. Out of the box, the models were underwhelming.
  • Hyperparameter tuning made the difference, raising accuracy to levels that are promising for anomaly detection and potentially zero-day detection.

The takeaway for security teams is encouraging: you don't need exotic deep learning to get useful signal from system call data, but you do need to understand your models well enough to tune them. Treating ML as a black box with default settings won't get you far.

Thanks

A huge thank you to my supervisors, Alan Herbert and Karen Bradshaw, for their guidance and patience, and to Liam Smit for advice and for spell checking more of my writing than anyone should have to.

The full thesis is available here. If you're working on host-based intrusion detection or ML for security and want to compare notes, get in touch.