From DNA Sequence Variation to .NET Bits and Bobs

Editor's Notes

• #7: First we have to talk a bit about how DNA is organised. As you may already know, the DNA is made of 4 bases (A – adenine, T – thymine, C – cytosine and G – guanine). Three of these bases join together form an amino acid. There must be a precise order or sequence in the DNA, because this is used to make proteins that are responsible for most of the things that happen in our bodies. The DNA sequence that codes for protein is known as a gene.
• #8: As a whole, the human genome is comprised of about 3 billion base pairs. I’ve said base pairs because DNA has two complementary strands. Each nucleotide can pair up only with it’s complement (Adenine can be paired only with Thymine and Cytosine only with Guanine). Now, everything is nice when things are working properly, but due to various factors mutations appear and the DNA sequence changes, it’s dynamic. Today we will discuss the single nucleotide polymorphism (aka SNPs). It’s the most common type of genetic variation and is represented by a change in a single nucleotide. For example Adenine changes to Cytosine. These kind of changes may actually be involved in various diseases, because now the sequence that codes for a particular protein has changed and so the gene function is changed. Researchers are using these SNPs as biological markers to locate genes that are associated with disease.
• #9: One way the scientists are searching for bio markers is by using the genome-wide association study (aka GWAS) approach. They scan hundreds of human genomes from both healthy people (aka controls) and carriers and they are looking for SNPs that may predict the presence of a disease. The results are then displayed in a scatter plot, called a Manhattan plot because it resembles Manhattan’s skyline (kind of). The peaks indicate the SNPs that are most likely associated with a specific disease. For example, in this graph 600k SNPs were tested for more than1400 Japanese suffering of atopic dermatitis (an itchy skin disorder) and almost 8000 controls. The peaks indicate SNPs that are associated with that disease.
• #10: Now we know why the DNA sequence is important and that any change in the nucleotides can modify the instructions that code for protein. In a way it’s similar to having a sequence of opcodes where any change can modify some instruction and the result can be a different behaviour of a program. Based on this similarity we built a .NET disassembler and developed a clustering algorithm that allows us to identify malicious code.
• #12: We are going to have a quick overview of the .NET file structure and see how do we actually get to the code. From the IMAGE_DATA_DIRECTORY of a .NET file we get to the CLR_HEADER that holds a similar type of structure for the MetaData. There we can get the number of streams and immediately after we have the structure for each stream. The STREAM_HEADER structure contains the name, offset and size for each stream.
• #13: Out of all streams we are interested in the metadata stream. This stream contains the Method Definition table that can get us to the code for each method. Each Method Definition structure that follows the metadata stream contains a virtual address (RVA) which points to the first instruction for each method defined in the executable.
• #14: Using the Common Intermediate Language specs we developed the disassembler. This one starts at the beginning of each method and continues to disassemble the current method until we reach the RET instruction. Afterwards we split everything into basic blocks (a set of instructions that contains one entry point and one exit point so the instructions are executed exactly once, in order). From each instruction in a basic block we extract only the first operand and once we have all of them, we do a CRC that will be later added into a database.
• #16: There’s 2 things to consider when we want to do clustering: Features to cluster on and the distance measure to use. In our case, in line with what Andrei just explained, we used the CRCID of each FOPS in our database. Basically all the FOPS are linked to their CRC hash / length in our database and therefore all have a unique CRCID. We build a list of CRCID for each, so they can be represent as an array like we see here. This intentionally keeps the feature representation generic so that we could end up using any other kind of feature to test different clustering approach without having the change the clustering engine itself. As for the distance measure, we used a derivate of the Jaccard Index. Instead of divided the size of the intersection by the size of the union of two sets, we divide the size of the smallest set by the size of the union of both sets. This gives us a similarity value between 0 and 1, then we subtract that to 1 to get a distance measure. You can see the basic function there with a simple implementation in C#.
• #17: The speed of the getDistance operation will of course vary depending on the amount of FOPS in each file, meaning the amount of CRCID in each arrays. Based on various tests, we can assume an average speed of 0.01s per distance computation. A normal simplistic implementation of clustering where we want to calculate the distance between every pair of files before then using a different clustering algorithm on this NxN matrix of distances would give a complexity of n-squared. For example, if we have to cluster 1500 files we would get 1500 square * 0.01 = 22500 seconds .. 6.25 hours. That clearly doesn’t scale well and as a result… ..we get some very bored analysts
• #18: Luckily, we don’t need to calculate the distance for each pair of file. We implemented a few mitigation techniques that help us improve the speed considerably. First, we load all the files in memory and order them amount of Fops they contain. By loading the files, I mean loading the arrays of CRCIDs associated with each files. Then, we compute the distance of two files only if their size ratio is within the threshold value. For example, if we want clusters with a maximum distance of 30% (0.30), the biggest a the two file can be maximum 30% bigger than the smallest one for us to bother computing the distance for these two files. This is possible because the file size ratio is directly used in the distance measure, so we know in advance that a file size ratio bigger than the threshold will never return a distance measure within the threshold. Also, we support agglomerative clustering, meaning we can periodically add new files to the previously known clusters or create new ones if needed. This could be daily, hourly or every time a new file enters our system. To avoid having to reset the clusters each time, we use prototypes to represent each cluster and we will only compare the new file to the prototypes of previously created clusters. We define as prototype of a cluster the smallest (fewest amount of Fops/CRCID) file it contain. You can see it as the smallest common denominator.
• #19: Here’s a basic overview of the clustering algorithm: We first load all the Files IDs and their respective CRCIDs in a dictionary. We then sort that dictionary by amount of CRCIDs each file contain. Smallest file first. This operation is very quick and will be the stepping stone of most of the time savings done later. We can take the first file (File1) in the dictionary and put it in a new cluster. - We then loop on every subsequent files until the size ratio gets greater than the threshold value and calculate their distance to the selected file. - If we obtain a distance within the threshold value on a valid file (File2), we put it in the same cluster as File1 and remove it from the dictionary of files to cluster. Once we reach the end of the possible matches for File1, we remove it from the dictionary and go back to the third step until all files have been checked.
• #20: Now here’s this algorithm explained with an animation. Let’s say we want to cluster these 8 files with a distance threshold of 30% (or 0.30). First step is to order them by size (or amount of Fops the file contain).
• #21: Then we can start clustering..
• #39: We ran a few tests to see the impact of the threshold value on the overall speed of clustering. We can see here that with a “loose” threshold of 80%, the speed curve is still looking exponential but is leaning towards a more linear form. The same 1500 files from the first example are now taking 760 seconds to cluster. Then, with a more realistic threshold of 20% we can see massive improvement in terms of speed, which is now almost linear. The set of 1500 files are now taking 14 seconds to cluster. This slide demonstrates well the impact of the mitigation technique we implemented to speed things up.
• #40: Just to recap on the initial example of clustering on the set of 1500 files that was originally taking 22500 seconds with the simplistic approach in n-square complexity. With the mitigation techniques and a threshold value of 80% it takes 760 seconds to cluster, whereas it only take 14 seconds with a threshold of 20%.
• #42: If you remember, in the first part of the presentation we talked about how scientists are using the Manhattan plot to identify the SNPs or bio markers used to predict a disease. Based on that approach and using the data that we’ve collected through our clustering algorithm, we’ve devised a similar technique. Here is an example of how starting from a known malicious file that is part of a cluster of a few hundred files, we managed to identify only the ones that are malicious and even more, how we identified the code that is unique to the malicious files. First we’ve got the list of CRCs that are present in the target file and for each one we’ve counted the number of files that contain the CRC The second step was to calculate the median and remove everything that’s above based on the assumption that most prevalent CRCs are clean. For each remaining CRC, using a formula similar to a chi-squared distribution with k degrees of freedom, we’ve calculated the probability that the CRC is malicious.
• #43: But before we show you the results, here is an example using the same approach on a real set of genetic data, comprised of 200k SNPs. We can easily spot a few peaks that show that they are relevant to the disease.
• #44: In our set of 285 files we got a similar result. A few CRCs (which are actually basic blocks) were identified as possibly malicious. In order to verify our result, we’ve queried the database to get the files that contain those CRCs. The query returned another 9 files and after the analysis we saw that they are part of the same malware family. We haven’t stopped here though. In the next slides we will see how we developed a plugin for IDA to help and speed up our analysis.
• #46: The Idea behind the IDA Python Plugin is to use the information generated by the clustering to assist analysis. The plugin does this by calculating the basic blocks and then the fops of these basic blocks. It then takes these information and poles the database on the determinations on these basic blocks. It uses these determinations to colour the basic block and also adds a comment that gives the breakdown of the determination. Finally it appends the comment with the fop length and Fop CRC this can be used to get the md5 of files with this basic block by polling the database. You can then use these files for comparative analysis.
• #47: This are the basic steps. First Pass -> we retrieve the basic blocks, FOPs and calculate the length and CRCs on the FOPs. Poll the database for the comments and for each CRC we get the determination for all the files that contain it. Second Pass -> recalculate the basic block and this time colour and comment each basic block.
• #48: If you have looked at the dot net Intermediate Language you will constantly see pushes and pops to the stack. This is the evaluation stack and it is used to managing state information in .Net similar to how x86 uses registers. It is best to really think of CLR as a stack machine. In this vain .NET uses the maxstack function to create space on the stack for its calculations. The space created is not so relevant just that there is enough space. For that reason the maxstack instruction is not that relevant, to the programs functionality so IDA Pro does not disassemble it. Our native C disassembler did parse it, so we had discrepancies between or IDA plugin output and our native C output. After some investigation we discover that there is little value in the maxstack instructions with regard to the functionality of the basic block. So we Ignore them and we also Ignore .net “NOP”’s for similar reasons. The second major issue came with the time taken to query the database. To improve performance we added a table to give us a single query for each CRC. We believe we can improve the performance further with store procedures and an ordering of this table, as well as improvement in the python code itself.