Anomaly Detection with Machine Learning Models Using API Calls

Abstract

Malware is malicious code developed to damage telecommunications and computer systems. Many malware causes anomaly events, such as occupying the systems' resources, such as CPU and memory, or preventing their use. Malware causing these events can hide their destructive activities. Therefore, monitoring their behavior to detect and block such malicious software is necessary. In other words, the anomalies they cause are detected and intervened by monitoring the behaviors exhibited by malware. Various features such as application programming interface (API) calls or system calls, registry modification, and network activities constitute malware behavior. API calls and various statistical information of these calls, extracted by dynamic analysis, are considered one of the most representative features of behavior-based detection systems. Each API call in the sequences is associated with previous or subsequent API calls. Such relationships may contain patterns of destructive functions of malware. Many intrusion/anomaly detection systems are proposed, including machine and deep learning models, in which various information about API/system calls are used as features. This paper aims to evaluate the effect of various statistical information of API calls on the models in detecting anomaly events and classification performances. The anomaly detection performances of various machine learning (ML) models with known effects in the literature are examined using a dataset containing API calls. As a result of the experiments, it is seen that the models using statistical features of API calls have reached high performance in terms of precision, recall, f1-score, and accuracy metrics.