Improving Omniunpack Alghorithm in Generic Unpacking PE File with Page Monitoring

Document Type : Original Article

Authors

1 azad university

2 iran university of science and technology

Abstract

Analysts used file signature comparison to detect malware and analyze the behavior of the executable file in the past. To prevent signature examination, the authors of the new and advanced malware used obfuscation methods to hide information, of which packaging is the most important and the most efficient one. This method encrypts and compresses the code, without harming the behavior of the original executable file and the code is obscure until it is executed. The methods that are now used to unpack these files are often methods that are specially designed for each type of packer for that file. There are other methods, such as Renovo and OmniUnpack for reopening that are known as public reopens, and actually cover the weakness of previous approaches which is the need to know the type of packager, but their main problem is finding the original point. The main entry of the program is the end of the unpacking section. Our approach to fix this problem is a method that detects this point using tracking memory pages and monitoring pages, then executes it, and then dumps the memory for creating a new file that has been unpacked. Our method has two advantages: first, there is no need for knowledge of packaging type, and second it can also be used for packers that are created in the future. Finally, in the evaluation section, we have shown that this method has a very high performance for current packers and more than 90% of them can be unpacked with it, so it can be used on an antivirus engine.       

Keywords

References

  1. A. Sharma and S. K. Sahay, “Evolution and Detection of Polymorphic and Metamorphic Malwares,” International Journal of Computer Applications, vol. 90, no. 2, p. 7, June 2014.##
  2. W. Yang, A. Yuanyuan, Z. Juanru, L. Junliang S. Bodong, L. Wenjun, and  H. Dawu Gu,  “Bytecode Decrypting and DEX Reassembling for Packed Android Malware,” Part of the Lecture Notes in Computer Science book series (LNCS, volume 9404), Dec. 2015.##
  3. S. Jain and Y. K. Meena, “Byte Level n–Gram Analysis for Malware Detection,” In: K. R.Venugopal, L. M. Patnaik, (eds.) ICIP 2011. CCIS, vol. 157, pp. 51–59, Springer, Heidelberg, 2011.##
  4. T. Graf, “Generic unpacking: How to handle modified or unknown PE compression engines,” Virus Bulletin, 2005.##
  5. L. Martignoni, Ch. Mihai, and J. Somesh, “Omniunpack: Fast, generic, and safe unpacking of malware,” In Computer Security Applications Conference, 2007. ACSAC 2007. Twenty-Third Annual, pp. 431-441. IEEE, 2007.##
  6. N. Pors, “The Effectiveness of Self-Mutating Malware Detection Techniques,” Utica College, ProQuest Dissertations Publishing, May 2017.##
  7. X. Tan, “Anti-unpacker tricks in malicious code,” In Proceedings of 10th Annual AVAR International Conference, 2007.##
  8. M. Christodorescu, J. Somesh, K. Johannes, K. Stefan, and V. Helmut, “Software transformations to improve malware detection,” Journal in Computer Virology 3, no. 4, pp. 253-265, 2007.##
  9. S. Huda, J. Abawajy, M. Alazab, M. Abdollalihian, R. Islam, and J. Yearwood, “Hybrids of support vector machine wrapper and filter based framework for malware detection,” Future Generation Computer Systems, vol. 55, Issue C, pp. 376-390, Feb. 2016.##
  10. G. Pék, E. Author, Z. Lázár, Z. Várnagy, M. Félegyházi, and L. Buttyán, “A Posteriori Detection of Malicious Code Loading by Memory Paging Analysis,” ESORICS 2016: Computer Security – ESORICS 2016, pp.199-216, Sep. 2016.##

 11.     S. Najari and E. Terik, “Common features of destructive detection methods using learning techniques,” The 9th Symposium on Advances in Science and Technology (9thSASTech), Mashhad, Iran, Dec. 2014.##

  1. G. C. Kessler, “Anti-forensics and the digital investigator,” In Australian Digital Forensics Conference, p. 1, 2007.##
  2. Y. Park, “We can still crack you! general unpacking method for android packer (not root),” In: Black Hat Asia, 2015.##
  3. M. G. Kang, P. Pongsin, and Y. Heng, “Renovo: A hidden code extractor for packed executables,” In Proceedings of the 2007 ACM workshop on Recurring malcode, ACM, pp. 46-53, 2007.##
  4. D. Regalado, H. Shon, H. Allen, E. Chris, N. Jonathan, S. Branko, L. Ryan, and S. Stephen, “Gray Hat Hacking the Ethical Hacker's Handbook,” McGraw-Hill Education Group, 2015.##
  5. P. Battista, F. Mercaldo, and V. Nardone, A. Santone, and C. A. Visaggio, “Identification Malware Families with Model Checking,” Department of Engineering, University of Sannio, Benevento, Italy, sept. 2015.##

 17.     P. Szor, “The art of computer virus research and defense, Pearson Education, 2005.##

  1. S. Jose, “Deep Packer Inspection: A Longitudinal Study of the Complexity of Run-Time Packers,” 2015 IEEE Symposium on Security and Privacy (SP) (2015). CA, USA, May 2015.##
  2. J. Rajendran, V. Vedula, T. Labs, Austin, and R. Karri, “Detecting malicious modifications of data in third-party intellectual property cores,” DAC '15 Proceedings of the 52nd Annual Design Automation Conference Article, no. 112, June 2015.##
  3. J. Caballero, U. Zurtuza, and J. Ricardo, “Detection of Intrusions and Malware, and Vulnerability Assessment,” 13th International Conference, DIMVA 2016, San Sebastián, Spain, July 7-8, 2016.##
  4. A. Arora, T. Stallings, R. Hasan, and G. Warner, “Malware Secrets: De-Obfuscating in the Cloud,” June 2017.##

Files

History

  • Receive Date: 20 January 2018
  • Accept Date: 17 October 2018

Share

How to cite

Statistics