An Approach to Rootkit Detection Based on Virtual Machine Introspection

Document Type : Original Article

Authors

iran university of science and technology

Abstract

Kernel rootkits have posed serious security threats due to their stealthy manner. To hide their presence and activities, many rootkits hijack control flows by modifying control data or hooks in the kernel space function pointers, especially those dynamically allocated from heaps and memory pools. These areas of kernel memory are currently not monitored by kernel integrity checkers. On the other hand, traditional host-based detection tools are executed inside the host they are protecting, therefore, since these tools are executed within the kernel, they could be easily detected by the rootkits. To solve this problem, current rootkit detection tools deploy virtual machine introspection technique that monitors the state of running virtual machine at hypervisor level, without rootkits interposition. The goal of this thesis is to present an approach based on virtual machine introspection, to detect rootkits which hide themselves and their associated malwares in the main memory using system control flow modification. The proposed approach monitors the integrity of windows kernel function pointers that are potentially prone to malicious exploits, based entirely on virtual machine introspection. This approach is evaluated with a set of rootkits which use advanced hooking techniques and it is shown that it detects all of the stealth techniques utilized

Keywords


C.-M. Chen, et al., “A Methodology for Hook-Based Kernel Level Rootkits,” International Conference on Information Security Practice and Experience, Springer International Publishing, 2014.##
Z. Wang, et al., “|Countering persistent kernel rootkits through systematic hook discovery,” International Workshop on Recent Advances in Intrusion Detection, Springer Berlin Heidelberg, 2008.##
H. Yin, et al., “HookScout: proactive binary-centric hook detection,” International Conference on Detection of Intrusions and Malware, and Vulnerability Assessment, Springer Berlin Heidelberg, 2010.##
G. Yan, et al., “MOSKG: countering kernel rootkits with a secure paging mechanism,” Security and Communication Networks 8.18, pp. 3580-3591, 2015.##
S. Vomel and L. Hermann, “Visualizing indicators of Rootkit infections in memory forensics,” IT Security Incident Management and IT Forensics (IMF), 2013 Seventh International Conference on. IEEE, 2013.##
M. Carbone, et al., “Mapping kernel objects to enable systematic integrity checking,” Proceedings of the 16th ACM conference on Computer and communications security, ACM, 2009.##
J. Butler and H. Greg, “VICE-Catch the hookers!(Plus new rootkit techniques),” Black Hat USA 2004 Conference, Las Vegas, USA. 2004.##
IceSword, http://www.antirootkit.com/software/IceSword.htm##
Jr. Petroni, L. Nick, and M. Hicks, “Automated detection of persistent kernel control-flow attacks,” Proceedings of the 14th ACM conference on Computer and communications security, ACM, 2007.##
A. Baliga, V. Ganapathy, and L. Iftode, “Automatic Inference and Enforcement of Kernel Data tructure Invariants,”  In Pro-ceedings of the 24th Annual Computer Security Applications Conference (ACSAC 2008), Anaheim, California, USA, pp. 77-86, 2008.##
http://www.sans.org/course/memory-forensics-in-depth.%22memory-forensics-in-depth%20%22.2014##
Z. Wang, X. Jiang, W. Cui, and P. Ning, “Countering Kernel Rootkits with Lightweight Hook Protection,” In Proceedings of the 16th ACM Conference on Computer and Communications Security (CCS 2009), Chicago, IL, USA, pp. 545-554, 2009.##
F. Yangchun, Z. Lin, and D. Brumley, “Automatically deriving pointer reference expressions from binary code for memory dump analysis,” Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering, ACM, 2015.##
C. Weng, et al., “CloudMon: Monitoring Virtual Machines in Clouds,” IEEE Transactions on Computers 65.12, pp.    3787-3793, 2016.##
A. Bianchi, et al., “Blacksheep: detecting compromised hosts in homogeneous crowds,” Proceedings of the 2012 ACM conference on Computer and communications            security, ACM, 2012.##
 H. Yin, Z. Liang, and D. Song, “|HookFinder: Identifying and understanding malware hooking behaviors,” In Proceedings of the 15th Annual Network and Distributed System Security Symposium (NDSS'08), February 2008.##  
I. Ahmed, et al., “Integrity checking of function pointers in kernel pools via virtual machine introspection,” Information Security, Springer International Publishing, pp. 3-19, 2015.##
S. Sparks, E. Shawn, and Z. Cliff, “Windows Rootkits-a Game of Hide and Seek,” Handbook of Security and Networks, vol. 345, 2011.##
Y. Liu, et al., “Concurrent and consistent virtual machine introspection with hardware transactional memory,” 2014 IEEE 20th International Symposium on High Performance Computer Architecture (HPCA), IEEE, 2014.##
A. Prakash, et al., “On the Trustworthiness of Memory Analysis—An Empirical Study from the Perspective of Binary Execution,” IEEE Transactions on Dependable and Secure Computing 12.5, pp. 557-570, 2015.##
M. H. Ligh, A. Case, J. Levy, and A. Walters, “The Art of Memory Forensics: Detecting Malware and Threats in Windows, Linux, and Mac Memory,” John Wiley and Sons, 2014.##
rootkit.com, http://www.rootkit.com/##