Ad
More Related Content
What's hot (20)
Similar to Machine learning with Big Data power point presentation (20)
Ad
Recently uploaded (20)
Ad
Machine learning with Big Data power point presentation
- 1. MACHINE LEARNING WITH BIG DATA CHALLENGES & APPROACHES BY K.DAVID DMCA-40
- 2. Introduction The Big Data revolution promises to transform how we live, work, and think by enabling process optimization, empowering insight discovery and improving decision making. Today, the amount of data is exploding at an unprecedented rate as a result of developments in Web technologies, social media, and mobile and sensing devices. For example, Twitter processes over 70M tweets per day, thereby generating over 8TB daily. The ability to extract value from Big Data depends on data analytics which consider analytics to be the core of the Big Data revolution Data analytics involves various approaches, technologies, and tools such as those from text analytics, business intelligence, data visualization, and statistical analysis. The machine learning (ML) as a fundamental component of data analytics. The ML will be one of the main drivers of the Big Data revolution. The reason for this is its ability to learn from data and provide data driven insights, decisions, and predictions.
- 3. ML Challenges Originating From Big Data Definition Big Data are often described by its dimensions, which are referred to as its V’s.Earlier definitions of Big Data focussed on three V’s (volume, velocity, and variety); however, a more commonly accepted definition now relies upon the following four V’s : volume, velocity, variety, and veracity. It is important to note that other Vs can also be found in the literature. For example, value is often added as a 5th V. However, value is defined as the desired outcome of Big Data processing and not as defining characteristics of Big Data itself This section identifies machine learning challenges and associates each challenge with a specific dimension of Big Data.
- 4. Charecteristics associated with challengs
- 5. VOLUME: The first and the most talked about characteristic of Big Data is volume: it is the amount, size, and scale of the data. In the machine learning context, size can be defined either Vertically by the number of records or samples in a dataset or horizontally by the number of features or attributes it contains. 1) Processing Performance:- One of the main challenges encountered in computations with Big Data comes from the simple principle that scale, or volume, adds computational complexity. Consequently, as the scale becomes large, even trivial operations can become costly.
- 6. volume For example, the standard support vector machine (SVM) algorithm has a training time complexity of O(m3 ) and a space complexity of O(m2 ), where m is the number of training samples. Therefore, an increase in the size m will drastically affect the time and memory needed to train the SVM algorithm and may even become computationally infeasible on very large datasets. Many other ML algorithms also exhibit high time complexity: for example, the time complexity of principal component analysis is O(mn2 +n3 ) , that of logistic regression O(mn2 +n3 ) , that of locally weighted linear regression O(mn2+n3) , and that of Gaussian discriminative analysis O(mn2+n3), where m is the number of samples and n the number of features.
- 7. Volume Class imbalance:- As datasets grow larger, the assumption that the data are uniformly distributed across all classes is often broken. This leads to a challenge referred to as class imbalance The performance of a machine learning algorithm can be negatively affected when datasets contain data from classes with various probabilities of occurrence. This problem is especially prominent when some classes are represented by a large number of samples and some by very few.
- 8. volume Curse of Dimensionality:- It refers to difficulties encountered when working in high dimensional space. Specifically, the dimensionality describes the number of features or attributes present in the dataset. In addition, dimensionality affects processing performance: the time and space complexity of ML algorithms is closely related to data dimensionality. The time complexity of many ML algorithms is polynomial in the number of dimensions.
- 9. volume Feature Engineering:- This is the process of creating features, typically using domain knowledge, to make machine learning perform better. Indeed, the selection of the most appropriate features is one of the most time consuming pre-processing tasks in machine Learning. As the dataset grows, both vertically and horizontally, it becomes more difficult to create new, highly relevant features.
- 10. volume Bonferonni’s principle : It embodies the idea that if one is looking for a specific type of event within a certain amount of data, the likelihood of finding this event is high. However, more often than not these occurrences are bogus, meaning that they have no cause and are therefore meaningless instances within a Dataset. In statistics, the Bonferonni correction theorem provides a means of avoiding those bogus positive searches within a dataset. It suggests That :- if testing m hypotheses with a desired significance of α , each
- 11. volume Variance and Bias Machine learning relies upon the idea of generalization; through observations and manipulations of data, representations can be generalized to enable analysis and prediction. Generalization error can be broken down into two components: variance and bias Variance describes the consistency of a learner’s ability to predict random things, whereas bias describes the ability of a learner to learn the wrong thing
- 13. APPROACHES This paper reviews and organizes various proposed machine learning approaches and discusses how they address the identified challenges. The big picture of approach-challenge correlations is presented in Table. It includes a list of approaches along with the challenges that each best addresses. Symbol ‘√’ indicate high degree of remedy while ‘*’ represents partial resolution.
- 15. Approaches ... The following sub-sections introduce techniques and methodologies being developed and used to handle the challenges associated with machine learning with Big Data. First, manipulation techniques used in conjunction with existing algorithms are presented. Second, various machine learning paradigms that are especially well suited to handle Big Data challenge
- 16. A. Manipulations for Big Data Data analytics using machine learning relies on an established suite of events, also known as the data analytics pipeline. The approaches presented in this section discuss possible manipulations in various steps of the existing Pipeline. Dimensionality reduction aims to map high dimensionality space onto lower-dimensionality one without significant loss of information.
- 17. Data Analytics pipeline: These three categories, along with their corresponding sub categories and sample solutions
- 18. Manipulations
- 19. B. Machine Learning Paradigms for Big Data A variety of learning paradigms exists in the field of machine learning; however, not all types are relevant to all areas of Research 1) Deep Learning Deep learning is an approach from the representation learning family of machine learning. Representation learning is also often referred to as feature learning It transforms data into abstract representations that enable the features to be learnt. In a deep learning architecture, these representations are subsequently used to accomplish the machine learning tasks
- 21. 2) Online Learning Because it responds well to large-scale processing by nature, online learning is another machine learning paradigm that has been explored to bridge efficiency gaps created by Big Data. Online learning can be seen as an alternative to batch learning, the paradigm typically used in conventional machine learning.
- 22. Local Learning Local learning is a strategy that offers an alternative to typical global learning. Conventionally, ML algorithms make use of global learning through strategies such as generative learning The idea behind it is to separate the input space into clusters and then build a separate model for each cluster. This reduces overall cost and complexity.
- 24. Transfer Learning Transfer learning is an approach for improving learning in a particular domain, referred to as the target domain, by training the model with other datasets from multiple domains, denoted as source domains, with similar attributes or features, such as the problem and constraints. This type of learning is used when the data size within the target domain is insufficient or the learning task is different . Figure shows an abstract view of transfer learning.
- 26. Lifelong learning Lifelong learning mimics human learning; learning is continuous; knowledge is retained and used to solve different problems. It is directed to maximize overall learning, to be able to solve a new task by training either on one single domain or on heterogeneous domains collectively The learning outcomes from the training process are collected and combined together in a space called the topic model or knowledge model.
- 28. Conclusion This paper has provided a systematic review of the challenges associated with machine learning in the context Of Big Data and categorized them according to the V dimensions of Big Data. Moreover, it has presented an overview of ML approaches and discussed how these techniques overcome the various challenges identified.
- 29. Thank you