Edge-based Discovery of Training Data for Machine Learning

Editor's Notes

• #2: Cartoon: New Yorker magazine April 20, 2018, p. 41
• #3: Deep learning has become the gold standard in many areas, especially computer vision, due to its superb accuracy. Here shows the high-level recipe when you try to apply deep learning to a problem. You collect a large amount of data and label it. Then you select a model and train a DNN. Finally you deploy the DNN for inference. Nowadays, there are many software libraries, frameworks, cloud services and web-based tools that let you do the last two steps with great convenience. Virtually all the painstaking effort is in the very first step. And it sometimes can be the showstopper of applying deep learning to your problem.
• #4: In this work, we focus on DNNs used by domain experts. Here are some examples. This animal is the transmitter of the SARS disease in China, 2003. You can imagine how valuable it would be if we had an accurate DNN detector and use it in public health effort. Likewise, this is a weapon that shot down an airplane and this is a pathological image of nuclear atypia in cancer. In all these cases, the target has low base rates – they are pretty rare in the data you are examining. And they all require expertise to correct identify. https://en.wikipedia.org/wiki/Masked_palm_civet#Connection_with_SARS
• #5: Building a training set of this kind of targets is hard. First, obviously, crowds are not experts. So crowd-sourcing methods like Amazon Mechanical Turk are not applicable to these domains. For example, only an expert can reliably and accurately distinguish between these animals. Second, there may exist access restriction of data, such as patient privacy, business policy and national security. In the worst case, a single domain expert has to generate an entire training set of thousands to tens of thousands of examples.
• #6: In this paper, we describe a system called Eureka, for efficient discovery of training examples from data sources dispersed over the Internet. The goal of Eureka is to optimally utilize an expert’s time and attention. It combines three key concepts to achieve its goal: early discard, iterative discovery workflow and edge computing, which I will describe next.
• #7: Here shows Eureka’s architecture. An expert user runs a GUI on her own computer. The GUI connects to a number of cloudlets across the Internet. These cloudlets are LAN-connected to some associated data sources. These data sources may be archival or live, depending on the specific use case. As the shape of these arrows indicates, connections between cloudlets and data sources are high-bandwidth and low-latency, while those on the Internet are the contrary. And this high-bandwidth access is used to execute early-discard code on the cloudlets to drop clearly irrelevant data. Only a tiny fraction of data long with meta-data are transmitted and shown to the user, consuming little Internet bandwidth.
• #8: Here shows an example of using the GUI to find images of deer from an unlabeled dataset. You can specify a list of early-discard filters and only images passing all of the filters are transmitted and displayed. You are seeing many false positives because the filters used in this case are very weak color and texture filters. (more time: 1. extend to general logical expression; 2. 500 more – efficient use of user attention)
• #9: To improve the efficacy of early-discard, we introduce the iterative discovery workflow. Here you see a spectrum of computer vision algorithms and machine learning models, from simple on the left, such as RGB and SIFT, to sophisticated on the right, such as deep learning. X-axis is the number of example images you have, and the Y-axis is the accuracy. While these numbers are not meant to be precise, the idea is that different models require a different amount of data to work properly, and they give you different levels of accuracy. When using Eureka, instead of creating a set of filters and searching for your target in one go, you iteratively change and improve your filters as you collect more examples, and move up the stairs when you have sufficient data to do so. When using Eureka, in the beginning, you have very few examples. So you should only use explicit features like RGB or SIFT. With these weak filters, you may be able to find a few more, which allows you to escalate to a little more advanced filter, like SVM. The SVM is considerable more accurate, making it easier to find some more positives in a reasonable amount of time. So you iterate and climb up the stairs when you have sufficient data. In this process, you are both using more and more sophisticated filters, and growing the size of the training set you collect.
• #10: Here again shows the case of finding deer, but after a few iterations of using Eureka and an SVM is being used. You see the filter has now become much more accurate.
• #11: When designing and implementing Eureka, we have two major concerns. First is software generality. We want to allow the use computer vision code written in a diversity of languages, libraries and frameworks, so that we can empower experts with the newest computer vision innovations quickly. To do so, we encapsulate filters in Docker containers. Second is runtime efficiency. Eureka needs to be able to rapidly discard large volume of data. To do so, we exploit specialized hardware such as GPUs on cloudlets when available. We also cache filter results to exploit temporal locality in typical Eureka workloads.
• #12: Another interesting problem is to match the Eureka system to the user. We propose that, ideally, the system should deliver images to user at a rate the user can inspect them. Because if the system is delivering too fast, you are pumping lots of results into the network which the user may never see. So it’s a waste of computation and precious Internet bandwidth. Our suggestion in this case is to restrict your search to fewer cloudlets ( or to bias your filters towards precision rather than recall.)
• #13: On the other hand, if the system is delivering too slowly, you are basically forcing the user to wait. And wasting an expert’s time is a really bad thing to do. An obvious solution is to scale out to more cloudlets. But there is a risk here. Showing more “junk” to user will cause user annoyance and dissatisfaction. So you really need to strike a balance between avoiding user wait time and avoiding too many false positives. Our rule of thumb in this scenario is one should focus on reducing the false positive rate before scaling out the many cloudlets.
• #14: To evaluate Eureka, we 99 million Flickr images from the YFCC100M dataset. On the edge we have 8 cloudlets with local access to data. And the client GUI connects to the cloudlets over the Internet.
• #15: We conducted three case studies using these three chosen targets – deer, Taj Mahal and fire hydrant. As you can see from the base rate, these are fairly rare objects in Flickr photos. We used Eureka to collect about 100 positive examples of each. Here you can see the number of images viewed by the user, and images discarded by Eureka in the whole process. You can see how effective Eureka is in reducing the amount of data the user needs to look at and label.
• #16: We compare Eureka with a brute-force method, where the user goes through the images one by one and label them. That’s basically how many datasets are curated today. For reference, we also compare with what we called “single-iteration Eureka”, which means using early-discard, but without iterative improvement. Y-axis shows how many images the user viewed in order to collect the same number of positives. As you can see, compared with brute-force, single-iteration Eureka gives you up to an order of magnitude of improvement, showing the efficacy of early-discard. On top of that, full Eureka gives another order of magnitude of improvement, showing the benefit of the iterative workflow.
• #17: We show how Eureka is iteratively improving user’s productivity, in the case of deer. We measure productivity in terms of new true positives found in each Eureka iteration. Over five iterations of using Eureka, the productivity increases from 0.4 to 4.7, more than 10X improvement.
• #18: Finally, we show the importance of edge computing. Specifically, we show when the data is at the edge, the compute must also be at the edge for Eureka to be efficient. To do so, we throttled the bandwidth between the cloudlet and the data source, and measure the machine processing throughput of an RGB histogram filter. The result shows it really needs LAN-connectivity at 1Gbps to deliver sufficiently high throughput. If the data is shipped over the wide area network, it slows down by about 10X.
• #20: In conclusion, … (….) Our evaluation shows ….
• #30: Why is it hard? Most importantly, crowds are not experts. So crowd-sourcing approaches like Amazon Mechanical Turk are not applicable in these domains. Only an expert can reliably classify these animals. Besides, these interesting phenomena are usually rare, making it difficult to find positive examples in unlabeled. Finally, there may exists access restriction of data, such as patient privacy, business policy and national security. In the worst case, a single expert has to generate an entire training set of thousands of to tens of thousands of examples.
• #31: Here shows the execution model. On the cloudlet, a component called itemizer reads whatever data in its raw format, and emits individual items. Items are independently unit of early-discard. Items are then feed into the item processor. Here a chain of filters evaluate the items and try to drop them. We encapsulate filters in Docker containers to achieve software generality I just mentioned. And we cache filter results to improve efficiency. Finally, the filters also attach key-value attributes to each item. These attributes both facilitates communication between filters during the run time and post-analysis after they are sent back to the user.
• #32: Finally, we study the importance of edge computing for Eureka. Specifically, how necessary is high-bandwidth access to data. Here we throttle the bandwidth between the cloudlet and the data source, and measure the machine processing throughput of three filters, including cheap ones and expensive ones. As you can see, when we decrease the bandwidth, the throughput drops significantly. Under 25 Mbps, there is basically no difference between cheap filters and expensive filters, because data access time becomes the bottleneck. So we see high-bandwidth access is crucial to the efficacy of Eureka.