Common Machine Learning Algorithms for Beginners

Common Machine Learning Algorithms for Beginners


 

Top Machine Learning Algorithms

According to a recent study, machine learning algorithms are expected to replace 25% of the jobs across the world, in the next 10 years. With the rapid growth of big data and availability of programming tools like Python and R –machine learning is gaining mainstream presence for data scientists. Machine learning applications are highly automated and self-modifying which continue to improve over time with minimal human intervention as they learn with more data. For instance, Netflix’s recommendation algorithm learns more about the likes and dislikes of a viewer based on the shows every viewer watches. To address the complex nature of various real-world data problems, specialized machine learning algorithms have been developed that solve these problems perfectly. For beginners who are struggling to understand the basics of machine learning, here is a brief discussion on the top machine learning algorithms used by data scientists.

Access Solved Big Data and Data Projects

What are machine learning algorithms?

A machine learning algorithm can be related to any other algorithm in computer science. An ML algorithm is a procedure that runs on data and is used for building a production-ready machine learning model. If you think of machine learning as the train to accomplish a task then machine learning algorithms are the engines driving the accomplishment of the task. Which type of machine learning algorithm works best depends on the business problem you are solving, the nature of the dataset, and the resources available at hand.

Types of Machine Learning Algorithms

Machine Learning algorithms are classified as –

1) Supervised Machine Learning Algorithms

Machine learning algorithms that make predictions on a given set of samples. Supervised machine learning algorithm searches for patterns within the value labels assigned to data points. Some popular machine learning algorithms for supervised learning include SVM for classification problems, Linear Regression for regression problems, and Random forest for regression and classification problems.

2) Unsupervised Machine Learning Algorithms

There are no labels associated with data points. These machine learning algorithms organize the data into a group of clusters to describe its structure and make complex data look simple and organized for analysis.

3) Reinforcement Machine Learning Algorithms

These algorithms choose an action, based on each data point and later learn how good the decision was. Over time, the algorithm changes its strategy to learn better and achieve the best reward.

List of Common Machine Learning Algorithms Every Engineer must know

Common Machine Learning Algorithms Infographic 

 

Machine Learning Algorithms

1. Naive Bayes Classifier Algorithm

It would be difficult and practically impossible to classify a web page, a document, an email, or any other lengthy text notes manually. This is where the Naïve Bayes Classifier machine learning algorithm comes to the rescue. A classifier is a function that allocates a population’s element value from one of the available categories. For instance, Spam Filtering and weather forecast are some of the popular applications of the Naïve Bayes algorithm. Spam filter here is a classifier that assigns a label “Spam” or “Not Spam” to all the emails.

Naïve Bayes Classifier is amongst the most popular learning method grouped by similarities, that works on the popular Bayes Theorem of Probability- to build machine learning models particularly for disease prediction and document classification. It is a simple classification of words based on the Bayes Probability Theorem for subjective analysis of content. This classification algorithm uses probabilities using the Bayes theorem. The basic assumption for Naive Bayesian algorithms is that all the features are considered to be independent of each other. It is a very simple algorithm and it is easy to implement. It is particularly useful for large datasets and can be implemented for text datasets.

Bayes theorem gives a way to calculate posterior probability P(A|B) from P(A), P(B) and P(B|A).

The formula is given by:  P(A|B) = P(B|A) * P(A) / P(B)

Where P(A|B) is the posterior probability of A given B, P(A) is the prior probability, P(B|A) is the likelihood which is the probability of B given A and P(B) is the prior probability of B. 

Learn More About Classification Algorithms

1.1. When to use the Naive Bayes Classifier algorithm?

  1. If you have a moderate or large training data set.
  2. If the instances have several attributes.
  3. Given the classification parameter, attributes that describe the instances should be conditionally independent.

1.2. Applications of Naive Bayes Classifier
 

Naive Bayes Algorithm Applications
 

  1. Sentiment Analysis- It is used at Facebook to analyse status updates expressing positive or negative emotions.
  2. Document Categorization- Google uses document classification to index documents and find relevancy scores i.e. the PageRank. PageRank mechanism considers the pages marked as important in the databases that were parsed and classified using a document classification technique.
  3. Naive Bayes Algorithm is also used for classifying news articles about Technology, Entertainment, Sports, Politics, etc.
  4. Email Spam Filtering-Google Mail uses Naive Bayes algorithm to classify your emails as Spam or Not Spam

1.3. Advantages of the Naive Bayes Classifier Algorithm

  1. Naive Bayes Classifier algorithm performs well when the input variables are categorical
  2. A Naïve Bayes classifier converges faster, requiring relatively little training data than other discriminative models like logistic regression, when the Naïve Bayes conditional independence assumption holds.
  3. With Naïve Bayes Classifier algorithm, it is easier to predict class of the test data set. A good bet for multi class predictions as well.
  4. Though it requires conditional independence assumption, Naïve Bayes Classifier has presented good performance in various application domains.

Data Science Libraries in Python to implement Naïve Bayes – Sci-Kit Learn

Get Closer To Your Dream of Becoming a Data Scientist with 70+ Solved End-to-End ML Projects

2. K Means Clustering Algorithm

K-means is a popularly used unsupervised machine learning algorithm for cluster analysis. K-Means is a non-deterministic and iterative method. The algorithm operates on a given data set through a pre-defined number of clusters, k. The output of the K Means algorithm is k clusters with input data partitioned among the clusters. For instance, let’s consider K-Means Clustering for Wikipedia Search results. The search term “Jaguar” on Wikipedia will return all pages containing the word Jaguar which can refer to Jaguar as a Car, Jaguar as Mac OS version, and Jaguar as an Animal. K Means clustering algorithm can be applied to group the web pages that talk about similar concepts. So, the algorithm will group all web pages that talk about Jaguar as an Animal into one cluster, Jaguar as a Car into another cluster, and so on.

For any new incoming data point, the data point is classified according to its proximity to the nearby classes. Datapoints inside a cluster will exhibit similar characteristics while the other clusters will have different properties.  The basic example of clustering would be the grouping of the same kind of customers in a certain class for any kind of marketing campaign. It is also a useful algorithm for document clustering. 

The steps followed in the k means algorithm are as follows - 

  1. Specify the number of clusters as k
  2. Randomly select k data points and assign them to the clusters
  3. Cluster centroids will be calculated subsequently
  4. Keep iterating from 1-3 steps until you find the optimal centroid after which values won’t change. 

i) The sum of squared distance between the centroid and the data point is computed

ii) Assign each data point to the cluster that is closer to the other cluster

iii) Compute the centroid for the cluster by taking the average of all the data points in the cluster

We can find the optimal number of clusters k by plotting the value of sum squared distance which decreases gradually to come to an optimal number k. 

2.1. Advantages of using K-Means Clustering ML Algorithm

  1. In case of globular clusters, K-Means produces tighter clusters than hierarchical clustering.
  2. Given a smaller value of K, K-Means clustering computes faster than hierarchical clustering for large number of variables.

2.2. Applications of K-Means Clustering

K Means Clustering algorithm is used by most of the search engines like Yahoo, Google to cluster web pages by similarity and identify the ‘relevance rate’ of search results. This helps search engines reduce the computational time for the users.

Data Science Libraries in Python to implement K-Means Clustering – SciPy, Sci-Kit Learn, Python Wrapper

Data Science Libraries in R to implement K-Means Clustering – stats

Click here to view a list of 50+ solved, end-to-end Big Data and Machine Learning Project Solutions (reusable code + videos)

3. Support Vector Machine Learning Algorithm

Support Vector Machine is a supervised machine learning algorithm for classification or regression problems where the dataset teaches SVM about the classes so that SVM can classify any new data. It works by classifying the data into different classes by finding a line (hyperplane) which separates the training data set into classes. As there are many such linear hyperplanes, SVM algorithm tries to maximize the distance between the various classes that are involved and this is referred as margin maximization. If the line that maximizes the distance between the classes is identified, the probability to generalize well to unseen data is increased.

3.1. SVM’s are classified into two categories

  1. Linear SVM’s – In linear SVM’s the training data i.e. classifiers are separated by a hyperplane.
  2. Non-Linear SVM’s- In non-linear SVM’s it is not possible to separate the training data using a hyperplane. For example, the training data for Face detection consists of group of images that are faces and another group of images that are not faces (in other words all other images in the world except faces). Under such conditions, the training data is too complex that it is impossible to find a representation for every feature vector. Separating the set of faces linearly from the set of non-face is a complex task.

Real-time Solved Machine Learning

3.2. Advantages of Using SVM

  1. SVM offers best classification performance (accuracy) on the training data.
  2. SVM renders more efficiency for correct classification of the future data.
  3. The best thing about SVM is that it does not make any strong assumptions on data.
  4. It does not over-fit the data.

3.3. Applications of Support Vector Machine

SVM is commonly used for stock market forecasting by various financial institutions. For instance, it can be used to compare the relative performance of the stocks when compared to performance of other stocks in the same sector. The relative comparison of stocks helps manage investment making decisions based on the classifications made by the SVM learning algorithm.

Data Science Libraries in Python to implement Support Vector Machine –SciKit Learn, PyML , SVMStruct  Python , LIBSVM

Free access to solved code Python and R examples can be found here (these are ready-to-use for your Data Science and ML projects) 

4. Apriori Machine Learning Algorithm

Apriori algorithm is an unsupervised machine learning algorithm that generates association rules from a given data set. Association rule implies that if an item A occurs, then item B also occurs with a certain probability. Most of the association rules generated are in the IF_THEN format. For example, IF people buy an iPad THEN they also buy an iPad Case to protect it. For the algorithm to derive such conclusions, it first observes the number of people who bought an iPad case while purchasing an iPad. This way a ratio is derived like out of the 100 people who purchased an iPad, 85 people also purchased an iPad case.

4.1. Principle on which Apriori Machine Learning Algorithm works:

  1. If an item set occurs frequently then all the subsets of the item set, also occur frequently.
  2. If an item set occurs infrequently then all the supersets of the item set have infrequent occurrence.
     

4.2. Advantages of Apriori Algorithm

  1. It is easy to implement and can be parallelized easily.
  2. Apriori implementation makes use of large item set properties.
     

4.3. Applications of Apriori Algorithm
 

Applications of Apriori Algorithm
 

  1. Detecting Adverse Drug Reactions - Apriori algorithm is used for association analysis on healthcare data like-the drugs taken by patients, characteristics of each patient, adverse ill-effects patients experience, initial diagnosis, etc. This analysis produces association rules that help identify the combination of patient characteristics and medications that lead to adverse side effects of the drugs.
     
  2. Market Basket Analysis - Many e-commerce giants like Amazon use Apriori to draw data insights on which products are likely to be purchased together and which are most responsive to promotion. For example, a retailer might use Apriori to predict that people who buy sugar and flour are likely to buy eggs to bake a cake.
     
  3. Auto-Complete Applications - Google auto-complete is another popular application of Apriori wherein - when the user types a word, the search engine looks for other associated words that people usually type after a specific word.

Applications of Apriori Machine Learning Algorithm

Data Science Libraries in Python to implement Apriori Machine Learning Algorithm – There is a python implementation for Apriori in PyPi

Data Science Libraries in R to implement Apriori Machine Learning Algorithm – arules

Become a Machine Learning Engineer

5. Linear Regression Machine Learning Algorithm

Linear Regression algorithm shows the relationship between 2 variables and how the change in one variable impacts the other. The algorithm shows the impact on the dependent variable on changing the independent variable. The independent variables are referred as explanatory variables, as they explain the factors the impact the dependent variable. The dependent variable is often referred to as the factor of interest or predictor. Linear regression is used for estimating real continuous values. The most common examples of linear regression are housing price predictions, sales predictions, weather predictions, employee salary estimations, etc. The basic goal for linear regression is to fit the best line amongst the predictions. The equation for linear regression is Y=a*x+b where y is the dependent variable and x is the set of independent variables. a is the slope and b is the intercept. 

The best example from human lives would be the way a child would solve a simple problem like - ordering the children in class height orderwise without asking the heights of the children. The child will be able to solve this problem by visually looking at the heights of the children and subsequently arranging them height-wise. This is how you can perceive linear regression in a real-life scenario. The weights which are the heights and the build of the children have been learnt by the child gradually. Looking back at the equation, a and b are the coefficients that will be learned through the regression model by minimizing the sum of squared errors in the model values. 

The graph below shows the relation between the number of umbrellas sold and the rainfall in a particular region -

Linear Regression Machine Learning Algorithm

5.1. Advantages of Linear Regression Machine Learning Algorithm

  1. It is one of the most interpretable machine learning algorithms, making it easy to explain to others.
  2. It is easy of use as it requires minimal tuning.
  3. It is the mostly widely used machine learning technique that runs fast.

5.2. Applications of Linear Regression

  1. Estimating Sales

    Linear Regression finds great use in business, for sales forecasting based on the trends. If a company observes steady increase in sales every month - a linear regression analysis of the monthly sales data helps the company forecast sales in upcoming months.

  2. Risk Assessment

    Linear Regression helps assess risk involved in insurance or financial domain. A health insurance company can do a linear regression analysis on the number of claims per customer against age. This analysis helps insurance companies find, that older customers tend to make more insurance claims. Such analysis results play a vital role in important business decisions and are made to account for risk.

Data Science Libraries in Python to implement Linear Regression – statsmodel and SciKit

Data Science Libraries in R to implement Linear Regression – stats

Explanations about the top machine learning algorithms will continue, as it is a work in progress. Stay tuned to our blog to learn more about the popular machine learning algorithms and their applications!!!

6. Logistic Regression Machine Learning Algorithm

The name of this algorithm could be a little confusing in the sense that the Logistic Regression machine learning algorithm is for classification tasks and not regression problems. The name ‘Regression’ here implies that a linear model is fit into the feature space. This algorithm applies a logistic function to a linear combination of features to predict the outcome of a categorical dependent variable based on predictor variables.

The odds or probabilities that describe the outcome of a single trial are modeled as a function of explanatory variables. Logistic regression algorithms help estimate the probability of falling into a specific level of the categorical dependent variable based on the given predictor variables.

Just suppose that you want to predict if there will be a snowfall tomorrow in New York. Here the outcome of the prediction is not a continuous number because there will either be snowfall or no snowfall and hence linear regression cannot be applied. Here the outcome variable is one of the several categories and using logistic regression helps.

6.1. Types of Logistic Regression

  1. Binary Logistic Regression - The most commonly used logistic regression when the categorical response has 2 possible outcomes i.e. either yes or not. Example –Predicting whether a student will pass or fail an exam, predicting whether a student will have low or high blood pressure, predicting whether a tumor is cancerous or not.
  2. Multi-nominal Logistic Regression - Categorical response has 3 or more possible outcomes with no order. Example- Predicting what kind of search engine (Yahoo, Bing, Google, and MSN) is used by majority of US citizens.
  3. Ordinal Logistic Regression - Categorical response has 3 or more possible outcomes with natural ordering. Example- How a customer rates the service and quality of food at a restaurant based on a scale of 1 to 10.

Let us consider a simple example where a cake manufacturer wants to find out if baking a cake at 160°C, 180°C and 200°C will produce a ‘hard’ or ‘soft’ variety of cake ( assuming the fact that the bakery sells both the varieties of cake with different names and prices). Logistic regression is a perfect fit in this scenario instead of other statistical techniques. For example, if the manufacturers produce 2 cake batches wherein the first batch contains 20 cakes (of which 7 were hard and 13 were soft ) and the second batch of cake produced consisted of 80 cakes (of which 41 were hard and 39 were soft cakes). Here in this case if a linear regression algorithm is used it will give equal importance to both the batches of cakes regardless of the number of cakes in each batch. Applying a logistic regression algorithm will consider this factor and give the second batch of cakes more weightage than the first batch.

6.2. When to Use Logistic Regression Machine Learning Algorithm

  1. Use logistic regression algorithms when there is a requirement to model the probabilities of the response variable as a function of some other explanatory variable. For example, the probability of buying a product X as a function of gender
  2. Use logistic regression algorithms when there is a need to predict probabilities that categorical dependent variables will fall into two categories of the binary response as a function of some explanatory variables. For example, what is the probability that a customer will buy a perfume given that the customer is a female?
  3. Logistic regression algorithms is also best suited when the need is to classify elements into two categories based on the explanatory variable. For example-classify females into ‘young’ or ‘old’ group based on their age.

6.3. Advantages of Using Logistic Regression

  1. Easier to inspect and less complex.
  2. Robust algorithm as the independent variables need not have equal variance or normal distribution.
  3. These algorithms do not assume a linear relationship between the dependent and independent variables and hence can also handle non-linear effects.
  4. Controls confounding and tests interaction.

6.4. Drawbacks of Using Logistic Regression

  1. When the training data is sparse and high dimensional, in such situations a logistic model may overfit the training data.
  2. Logistic regression algorithms cannot predict continuous outcomes. For instance, logistic regression cannot be applied when the goal is to determine how heavily it will rain because the scale of measuring rainfall is continuous. Data scientists can predict heavy or low rainfall but this would make some compromises with the precision of the dataset.
  3. Logistic regression algorithms require more data to achieve stability and meaningful results. These algorithms require a minimum of 50 data points per predictor to achieve stable outcomes.
  4. It predicts outcomes depending on a group of independent variables and if a data scientist or a machine learning expert goes wrong in identifying the independent variables then the developed model will have minimal or no predictive value.
  5. It is not robust to outliers and missing values.

6.5. Applications of Logistic Regression

  1. Logistic regression algorithm is applied in the field of epidemiology to identify risk factors for diseases and plan accordingly for preventive measures.
  2. Used to predict whether a candidate will win or lose a political election or to predict whether a voter will vote for a particular candidate.
  3. Used to classify a set of words as nouns, pronouns, verbs, adjectives.
  4. Used in weather forecasting to predict the probability of rain.
  5. Used in credit scoring systems for risk management to predict the defaulting of an account.

The Data Science libraries in Python language to implement Logistic Regression Machine Learning Algorithm is Sci-Kit Learn.

The Data Science libraries in R language to implement Logistic Regression Machine Learning Algorithm is stats package (glm () function)  

Free access to solved code Python and R examples can be found here (these are ready-to-use for your Data Science and ML projects) 

Get FREE Access to Machine Learning Example Codes for Data Cleaning, Data Munging, and Data Visualization

7. Decision Tree Machine Learning Algorithm
 

Decision Tree Machine Learning Algorithm Explained

You are making a weekend plan to visit the best restaurant in town as your parents are visiting but you are hesitant in making a decision on which restaurant to choose. Whenever you want to visit a restaurant you ask your friend Tyrion if he thinks you will like a particular place. To answer your question, Tyrion first has to find out, the kind of restaurants you like. You give him a list of restaurants that you have visited and tell him whether you liked each restaurant or not (giving a labelled training dataset). When you ask Tyrion that whether you will like a particular restaurant R or not, he asks you various questions like “Is “R” a roof top restaurant?” , “Does restaurant “R” serve Italian cuisine?”, “Does R have live music?”, “Is restaurant R open till midnight?” and so on. Tyrion asks you several informative questions to maximize the information gain and gives you YES or NO answer based on your answers to the questionnaire. Here Tyrion is a decision tree for your favourite restaurant preferences.

A decision tree is a graphical representation that makes use of branching methodology to exemplify all possible outcomes of a decision, based on certain conditions. In a decision tree, the internal node represents a test on the attribute, each branch of the tree represents the outcome of the test and the leaf node represents a particular class label i.e. the decision made after computing all of the attributes. The classification rules are represented through the path from root to the leaf node.

7.1. Types of Decision Trees

Classification Trees- These are considered as the default kind of decision trees used to separate a dataset into different classes, based on the response variable. These are generally used when the response variable is categorical in nature.

Regression Trees-When the response or target variable is continuous or numerical, regression trees are used. These are generally used in predictive type of problems when compared to classification.

Decision trees can also be classified into two types, based on the type of target variable- Continuous Variable Decision Trees and Binary Variable Decision Trees. It is the target variable that helps decide what kind of decision tree would be required for a particular problem.
 

Why use decision tree machine learning algorithm

7.2. Why should you use Decision Tree ML algorithm?

  1. These machine learning algorithms help make decisions under uncertainty and help you improve communication, as they present a visual representation of a decision situation.
  2. Decision tree machine learning algorithms help a data scientist capture the idea that if a different decision was taken, then how the operational nature of a situation or model would have changed intensely.
  3. Decision tree algorithms help make optimal decisions by allowing a data scientist to traverse through forward and backward calculation paths.

7.3. When to use Decision Tree Machine Learning Algorithm>

  1. Decision trees are robust to errors and if the training data contains errors- decision tree algorithms will be best suited to address such problems.
  2. Decision trees are best suited for problems where instances are represented by attribute value pairs.
  3. If the training data has missing value then decision trees can be used, as they can handle missing values nicely by looking at the data in other columns.
  4. Decision trees are best suited when the target function has discrete output values.

7.4. Advantages of Using Decision Tree ML Algorithms

  1. Decision trees are very instinctual and can be explained to anyone with ease. People from a non-technical background, can also decipher the hypothesis drawn from a decision tree, as they are self-explanatory.
  2. When using decision tree machine learning algorithms, data type is not a constraint as they can handle both categorical and numerical variables.
  3. Decision tree machine learning algorithms do not require making any assumption on the linearity in the data and hence can be used in circumstances where the parameters are non-linearly related. These machine learning algorithms do not make any assumptions on the classifier structure and space distribution.
  4. These algorithms are useful in data exploration. Decision trees implicitly perform feature selection which is very important in predictive analytics. When a decision tree is fit to a training dataset, the nodes at the top on which the decision tree is split, are considered as important variables within a given dataset and feature selection is completed by default.
  5. Decision trees help save data preparation time, as they are not sensitive to missing values and outliers. Missing values will not stop you from splitting the data for building a decision tree. Outliers will also not affect the decision trees as data splitting happens based on some samples within the split range and not on exact absolute values.

Get More Practice, More Data Science and Machine Learning Projects, and More guidance.Fast-Track Your Career Transition with ProjectPro

7.5. Drawbacks of Using Decision Tree ML Algorithms

  1. The more the number of decisions in a tree, less is the accuracy of any expected outcome.
  2. A major drawback of decision tree machine learning algorithms, is that the outcomes may be based on expectations. When decisions are made in real-time, the payoffs and resulting outcomes might not be the same as expected or planned. There are chances that this could lead to unrealistic decision trees leading to bad decision making. Any irrational expectations could lead to major errors and flaws in decision tree analysis, as it is not always possible to plan for all eventualities that can arise from a decision.
  3. Decision Trees do not fit well for continuous variables and result in instability and classification plateaus.
  4. Decision trees are easy to use when compared to other decision making models but creating large decision trees that contain several branches is a complex and time consuming task.
  5. Decision tree machine learning algorithms consider only one attribute at a time and might not be best suited for actual data in the decision space.
  6. Large sized decision trees with multiple branches are not comprehensible and pose several presentation difficulties.

7.6. Applications of Decision Tree Machine Learning Algorithm

  1. Decision trees are among the popular machine learning algorithms that find great use in finance for option pricing.
  2. Remote sensing is an application area for pattern recognition based on decision trees.
  3. Decision tree algorithms are used by banks to classify loan applicants by their probability of defaulting payments.
  4. Gerber Products, a popular baby product company, used decision tree machine learning algorithm to decide whether they should continue using the plastic PVC (Poly Vinyl Chloride) in their products.
  5. Rush University Medical Centre has developed a tool named Guardian that uses a decision tree machine learning algorithm to identify at-risk patients and disease trends.

The Data Science libraries in Python language to implement Decision Tree Machine Learning Algorithm are – SciPy and Sci-Kit Learn.

The Data Science libraries in R language to implement Decision Tree Machine Learning Algorithm is caret.

 

Applications of Decision Tree Machine Learning algorithm

8. Random Forest Machine Learning Algorithm

Let’s continue with the same example that we used in decision trees, to explain how Random Forest Machine Learning Algorithm works. Tyrion is a decision tree for your restaurant preferences. However, Tyrion being a human being does not always generalize your restaurant preferences with accuracy. To get more accurate restaurant recommendation, you ask a couple of your friends and decide to visit the restaurant R, if most of them say that you will like it. Instead of just asking Tyrion, you would like to ask Jon Snow, Sandor, Bronn and Bran who vote on whether you will like the restaurant R or not. This implies that you have built an ensemble classifier of decision trees - also known as a forest.

You don’t want all your friends to give you the same answer - so you provide each of your friends with slightly varying data. You are also not sure of your restaurant preferences and are in a dilemma.You told Tyrion that you like Open Roof Top restaurants but maybe, just because it was summer when you visited the restaurant you could have liked it then. You may not be a fan of the restaurant during the chilly winters. Thus, all your friends should not make use of the data point that you like open roof top restaurants, to make their recommendations for your restaurant preferences.

By providing your friends with slightly different data on your restaurant preferences, you make your friends ask you different questions at different times. In this case just by slightly altering your restaurant preferences, you are injecting randomness at model level (unlike randomness at data level in case of decision trees). Your group of friends now form a random forest of your restaurant preferences.

Random Forest is the go to machine learning algorithm that uses a bagging approach to create a bunch of decision trees with random subset of the data. A model is trained several times on random sample of the dataset to achieve good prediction performance from the random forest algorithm.In this ensemble learning method, the output of all the decision trees in the random forest, is combined to make the final prediction. The final prediction of the random forest algorithm is derived by polling the results of each decision tree or just by going with a prediction that appears the most times in the decision trees.

For instance, in the above example - if 5 friends decide that you will like restaurant R but only 2 friends decide that you will not like the restaurant then the final prediction is that, you will like restaurant R as majority always wins.

8.1. Why use Random Forest Machine Learning Algorithm?

  1. There are many good open source, free implementations of the algorithm available in Python and R.
  2. It maintains accuracy when there is missing data and is also resistant to outliers.
  3. Simple to use as the basic random forest algorithm can be implemented with just a few lines of code.
  4. Random Forest machine learning algorithms help data scientists save data preparation time, as they do not require any input preparation and are capable of handling numerical, binary and categorical features, without scaling, transformation or modification.
  5. Implicit feature selection as it gives estimates on what variables are important in the classification.

8.2. Advantages of Using Random Forest ML Algorithm

  1. Overfitting is less of an issue with Random Forests, unlike decision tree machine learning algorithms. There is no need of pruning the random forest.
  2. These algorithms are fast but not in all cases. A random forest algorithm, when run on an 800 MHz machine with a dataset of 100 variables and 50,000 cases produced 100 decision trees in 11 minutes.
  3. Random Forest is one of the most effective and versatile machine learning algorithm for wide variety of classification and regression tasks, as they are more robust to noise.
  4. It is difficult to build a bad random forest. In the implementation of Random Forest Machine Learning algorithms, it is easy to determine which parameters to use because they are not sensitive to the parameters that are used to run the algorithm. One can easily build a decent model without much tuning.
  5. Random Forest machine learning algorithms can be grown in parallel.
  6. This algorithm runs efficiently on large databases.
  7. Has higher classification accuracy.

8.3. Drawbacks of Using Random Forest ML Algorithm

  1. They might be easy to use but analysing them theoretically, is difficult.
  2. Large number of decision trees in the random forest can slow down the algorithm in making real-time predictions.
  3. If the data consists of categorical variables with different number of levels, then the algorithm gets biased in favour of those attributes that have more levels. In such situations, variable importance scores do not seem to be reliable.
  4. When using RandomForest algorithm for regression tasks, it does not predict beyond the range of the response values in the training data.

8.4. Applications of Random Forest Machine Learning Algorithm

  1. Random Forest algorithms are used by banks to predict if a loan applicant is a likely high risk.
  2. They are used in the automobile industry to predict the failure or breakdown of a mechanical part.
  3. These algorithms are used in the healthcare industry to predict if a patient is likely to develop a chronic disease or not.
  4. They can also be used for regression tasks like predicting the average number of social media shares and performance scores.
  5. Recently, the algorithm has also made way into predicting patterns in speech recognition software and classifying images and texts.

Data Science libraries in Python language to implement Random Forest Machine Learning Algorithm is Sci-Kit Learn.

Data Science libraries in R language to implement Random Forest Machine Learning Algorithm is randomForest.
 

Random Forest Machine Learning Algorithm

9. Artificial Neural Networks Machine Learning Algorithm

The human brain has a highly complex and non-linear parallel computer that can organize the structural constituents i.e. the neurons interconnected in a complex manner between each other. Let us take a simple example of face recognition-whenever we meet a person, a person who is known to us can be easily recognized with his name or he works at XYZ place or based on his relationship with you. We may be knowing thousands of people, the task requires the human brain to immediately recognize the person (face recognition). Now, suppose instead of the human brain doing it, if a computer is asked to perform this task.  It is not going to be an easy computation for the machine as it does not know the person. You have to teach the computer that there are images of different people. If you know 10,000 people then you have to feed all the 10,000 photographs into the computer. Now, whenever you meet a person you capture an image of the person and feed it to the computer. The computer matches this photograph with all the 10,000 photographs that you have already fed into the database. At the end of all the computations-it gives the result with the photograph that best resembles the person. This could take several hours or more depending on the number of images present in the database. The complexity of the task will increase with the increase in the number of images in the database. However, a human brain can recognize it instantly.

Can we recognize this instantly using a computer? Is it that the computation capability that exists in humans different from the computers? If you consider the processing speed of a silicon IC it is of the order of 10-9 (order of nanoseconds) whereas the processing speed of a human neuron is 6 times slower than typical IC’s i.e. 10-3 (order of milliseconds). In that case, there is a puzzling question then how is that the processing time of the human brain faster than that of a computer. Typically, there are 10 billion neurons with approximately 60 trillion interconnections inside the human brain but still, it processes faster than the computer. This is because the network of neurons in the human brain is massively parallel.

Now the question is that is it possible to mimic the massively parallel nature of the human brain using computer software. It is not that easy as we cannot really think of putting so many processing units and realizing them in a massively parallel fashion. All that can be done within a limitation is interconnecting a network of processors. Instead of considering the structure of a human brain in totality, only a very small part of the human brain can be mimicked to do a very specific task. We can make neurons but they will be different from the biological neuron of the human brain. This can be achieved using Artificial Neural Networks. By artificial we inherently mean something that is different from the biological neurons. ANN’s are nothing but simulated brains that can be programmed the way we want. By defining rules to mimic the behavior of the human brain, data scientists can solve real-world problems that could have never been considered before.

9.1 How do Artificial Neural Network algorithms work?

It is a subfield of artificial intelligence which is modeled after the brain. It is a computational network consisting of neurons interconnected to each other. This interconnected structure is used for making various predictions for both regressions as well as classification problems. The ANN consists of various layers - the input layer, the hidden layer, and the output layers. The hidden layers could be more than 1 in number. The hidden layer is the place where all the mathematics of the neural network takes place. The basic formulas of weights and biases are added here along with the application of the activation functions. These activation functions are responsible for delivering the output in a structured and trimmed manner. It is majorly used for solving non-linear problems - handwriting recognition, traveling salesman problems, etc. ANNs involves complex mathematical calculations and are highly compute-intensive in nature.

Imagine you are walking on a walkway and you see a pillar (assume that you have never seen a pillar before). You walk into the pillar and hit it. Now, the next time whenever you see a pillar you stay a few meters away from the pillar and continue walking on the side. This time your shoulder hits the pillar and you are hurt again. Again when you see the pillar you ensure that you don’t hit it but this time on your path you hit a letter-box (assuming that you have never seen a letter-box before). You walk into it and the complete process repeats again. This is how an artificial neural network works, it is given several examples and it tries to get the same answer. Whenever it is wrong, an error is calculated. So, next time for a similar example the value at the synapse (weighted values through which neurons are connected in the network) and neuron is propagated backward i.e. backpropagation takes place. Thus, an ANN requires lots of examples and learning and they can be in millions or billions for real-world applications.

Recommended Reading: Types of Neural Networks

9.2 Why use Artificial Neural Networks?

  1. ANN’s have interconnection of non-linear neurons thus these machine learning algorithms can exploit non-linearity in a distributed manner.
  2. They can adapt free parameters to the changes in the surrounding environment.
  3. Learns from its mistakes and takes better decisions through backpropagation.

9.3 Advantages of Using Artificial Neural Networks

  • Easy to understand for professionals who do not want to dig deep into math-related complex machine learning algorithms. If you are trying to sell a model to an organization which would you rather say Artificial Neural Networks (ANN) or Support Vector Machine (SVM). We guess the answer obviously is going to be ANN because you can easily explain to them that they just work like the neurons in your brain.
  • They are easy to conceptualize.
  • They have the ability to identify all probable interactions between predictor variables.
  • They have the ability to subtly identify complex nonlinear relationships that exists between independent and dependent variables.
  • It is relatively easy to add prior knowledge to the model.

9.4 Disadvantages of Using Artificial Neural Networks

  • It is very difficult to reverse engineer artificial neural network algorithms. If your ANN machine learning algorithm learns that the image of a dog is actually a cat then it is very difficult to determine “why”. All than can be done is continuously tweak or train the ANN further.

  • Artificial Neural Network algorithms are not probabilistic meaning if the output of the algorithm is a continuous number it is difficult to translate it into a probability.
  • They are not magic wands and cannot be applied to solve any kind of machine learning algorithm.
  • Artificial Neural Networks in native implementation are not highly effective at practical problem-solving. However, this can be improved with the use of deep learning techniques.
  • Multi-layered artificial neural network algorithms are hard to train and require tuning a lot of parameters.

9.5 Applications of Artificial Neural Networks

Artificial Neural Networks are among the hottest machine learning algorithms in use today solving problems of classification to pattern recognition. They are extensively used in research and other application areas like –

  • Financial Institutions use Artificial Neural Networks machine learning algorithms to enhance the performance in evaluating loan applications, bond rating, target marketing, credit scoring. They are also used to identify instances of fraud in credit card transactions.
  • Buzzfeed uses artificial neural network algorithms for image recognition to organize and search videos or photos.
  • Many bomb detectors at US airports use artificial neural networks to analyze airborne trace elements and identify the presence of explosive chemicals.
  • Google uses Artificial Neural Networks for Speech Recognition, Image Recognition, and other pattern recognition (handwriting recognition) applications. ANN’s are used at Google to sniff out spam and for many other applications.
  • Artificial Neural Networks find great applications in robotic factories for adjusting temperature settings, controlling machinery, diagnose malfunctions.

10) K-Nearest Neighbors

KNN is the simplest classification algorithm. It is also used for the prediction of continuous values like regression. Distance-based measures are used in K Nearest Neighbors to get the closest correct prediction. The final prediction value is chosen on the basis of the k neighbors. The various distance measures used are Euclidean, Manhattan, Minkowski, and Hamming distance. The first three are continuous functions while Hamming distance is used for categorical variables. Choosing the value of K is the most important task in this algorithm. It is often referred to as the lazy learner algorithm.

 

K Nearest Neighbor ML Algorithm

Image Credit: medium.com 

As shown in the diagram above, the distances from the new point are calculated with each of the classes. Lesser the distance, accordingly the new point will be assigned to the class that is closer to the point. 

10.1 Advantages of Using K-Nearest Neighbors

  1. High accuracy but better algorithms exist
  2. It's very useful for non-linear data as there are no assumptions here

10.2 Disadvantages of Using K-Nearest Neighbors

  1. Computationally expensive requires high memory storage
  2. Sensitive to scaling of data

11) Gradient Boosting Machine Learning Algorithm

Gradient Boosting Classifier uses the boosting methodology where the trees which are created follow the decision tree method with minor changes. The weak learners from every tree are subsequently given more weightage and given to the next tree in succession so that the predictions for the trees are improved versions from the previous ones. It uses the weighted average for calculating the final predictions. Boosting is used when we have a large amount of data with high predictions.

11.1 XgBoost 

XgBoost is an advanced implementation of gradient boosting algorithms. It is different from gradient boosting in its calculations as it applies the regularization technique internally. Xgboost is referred to as regularized boosting technique 

Pros

  1. It is much faster than the gradient boosting mechanism.
  2. XGBoost allows users to define custom optimization objectives and evaluation criteria.
  3. XgBoost has techniques to handle missing values

Cons

  1. Difficult interpretation
  2. Overfitting is possible
  3. Harder to tune

11.2 CatBoost

CatBoost is an open-source gradient boosting library which is used for training large amounts of data using ML. It supports the direct usage of categorical variables. It gives a very high performance in comparison to the other boosting algorithms. It is very easy to implement and run. It is a model developed by Yandex. It provides support for out-of-the-box descriptive data formats and it does not require much training. It gives a good performance with a lesser number of training iterations.

11.3 Light GBM

LightGBM is a gradient boosting framework that uses decision tree algorithms. As the name suggests, its training speed is very fast and can be used for training large datasets. 

Pros

  1. Faster training speed and accuracy
  2. Lower memory usage
  3. Parallel GPU support
  4. Higher efficiency and performance

Cons

  1. Narrow user base

 

12) Linear Discriminant Analysis Machine Learning Algorithm

Linear Discriminant Analysis or LDA is a machine learning algorithm that provides an indirect approach to solve a classification machine learning problem. To predict the probability, Pn(X) that a given feature, X belongs to a given class Yn or not, it assumes a density function of all the features that belong to that class. It then uses this density function, fn(X) to predict the probability Pn(X) using

Where πn is the overall or prior probability that a randomly picked observation belongs to nth class.

Let the dataset have only feature variable, then the LDA assumes a Gaussian distribution function for fn(X) having a class-specific mean vector (𝜇n) and a covariance matrix that is applicable for all N classes. After that to assign a class to an observation from the test dataset, it evaluates the discriminant function


The  LDA classifier then predicts the class of the test variable for which the value of the discriminant function is the largest. We call the machine algorithm as “linear” discriminant analysis because, in the discriminant function, the component functions are all linear functions of x. 

Note: In the case of more than one variable, LDA assumes a  multivariate gaussian function and the discriminant function is scaled accordingly.

Advantages of Linear Discriminant Analysis Machine Learning Algorithm
1. It works well for machine learning problems where the classes to be assigned are well-separated.
2. If the number of feature variables in the given dataset is small and fits the normal distribution well, the LDA provides stable results.
3. It is easy to understand and simple to use.

Disadvantages of Linear Discriminant Analysis Machine Learning Algorithm
1. It requires the feature variables to follow the Gaussian distribution and thus has limited applications. 
2. It does not perform very well on datasets having a small number of target variables.


Applications of Linear Discriminant Analysis Machine Learning Algorithm
Classifying the Iris Flowers: The famous Iris Dataset contains four features (sepal length, petal length, sepal width, petal width) of three types of Iris flowers. You can use LDA to classify these flowers based on the given four features.


13. Quadratic Discriminant Analysis Machine Learning Algorithm

This machine-learning algorithm is similar to the LDA machine learning algorithm that we discussed above. Similar to LDA, the QDA algorithm assumes that feature variables belonging to a particular class obey the gaussian distribution function and utilizes Bayes’ theorem for predicting the covariance matrix.

However, in contrast to LDA, QDA presumes that each class in the target variables has its covariance matrix. And, if one implements this assumption to evaluate the expression for the discriminant function, they will obtain a few quadratic functions of x in the formula. Hence, the word linear is replaced by quadratic. Learn how to implement this algorithm

Advantages of Quadratic Discriminant Analysis Machine Learning Algorithm
1. It performs well for machine learning problems where the size of the training set is large.

2. QDA advises for machine learning problems where the feature variables in the given dataset clearly don’t seem to have a common covariance matrix for N classes.

Machine Learning Projects

PREVIOUS

NEXT

 

Copy of How to Start a Travel Blog Graphic


Tutorials