Statistics provides the tools and methods to find structure and meaningful insights into our data, and also helps us to quantify the embedded uncertainty and hence, having a good foundation of Statistics is crucial for any Data Scientist. So, for any Data Science interview, we get tested based on our knowledge of Statistics.
Here in this article, I have compiled 12 statistical concepts that I found to be very useful for cracking interviews. So, here I am going to go over those 12 concepts and explain what they’re all about.
- Conditional Probability and Bayes’ Theorem For any two events A and B, P(A|B) represents the conditional probability of occurrence of event A given that event B has already occurred. The formula for conditional probability is given by the following equation –
In continuation to the discussion of conditional probability, revising our prior probability of an event when new information becomes available is a crucial phase and that is where Bayes’ Theorem becomes useful. The following mathematical equation sums up Bayes’ Theorem.
In this equation A is an event and B is empirical evidence or information received from data. So, P(A) is the prior probability of event A and P(B) is the probability of event B based on evidence from data, and P(B|A) is known as the likelihood. So, Bayes’ Theorem gives us the probability of an event based on our prior knowledge about the event and updates that conditional probability when we get some new information about the same.
A very easy example of Bayes’ theorem can be to predict the probability of raining on a particular day given that the morning was cloudy. Suppose, the probability of raining i.e., P(Rain) is 10% on a day in June and the probability that the morning was cloudy given it rained i.e., P(Cloud|Rain) is 50%. Additionally, the probability of a cloudy morning in any day in June i.e., P(Cloud) is 40%, then applying Bayes’ theorem we can conclude that the probability that it will rain today given that it was cloudy in the morning is:
2. Sampling technique
If we call all elements in a group as the Population, then a subset of that population is called a sample. The total number of observations or individuals in that sample is known as the sample size. Any statistical constants of the population, for example, the descriptive measures like mean, the variance of any characteristic of the population are known as parameters. The same statistical measure will be known as a statistic when it is computed based on sample observations. Now the method of choosing the sample from a population can be of two types- probabilistic and non-probabilistic.
Probabilistic Sampling Methods
a. Simple random sampling
In simple random sampling, each sample is equally likely to be selected from the population. This technique reduces selection bias. However, a disadvantage of this technique is that we may not be able to select samples based on any particular characteristic, which is uncommon.
Example – An example of simple random sampling can be of choosing 10 students from a class of 100 based on a lottery.
b. Systematic sampling
In Systematic sampling, samples are collected at regular and specified intervals. The advantage of systematic sampling is that it helps to maintain an adequate sampling size.
An example of systematic sampling can be of choosing every 10th customer in a grocery store for a customer satisfaction survey.
c. Stratified sampling
In stratified sampling, the entire population is divided into various subgroups, also known as, strata which have similar characteristics. The main objective of this sampling is that to have representation from all the subgroups which show variability in the characteristics of interest. .
For example, if we are interested to know about people’s opinion on any particular topic in India, we can divide the population based on cities, gender and age group. These sub-populations based on the chosen characteristics are called strata and each of the samples in a strata are chosen based on probability sampling method e.g., simple random sampling.
Non-Probabilistic Sampling Methods
a. Convenience sampling
In Convenience sampling, participants are chosen based on their availability and willingness to take part in the study. However, this technique can suffer from volunteer bias which can be a risk of all non-probability-based sampling methods because of the samples choosing themselves instead of getting randomly picked.
b. Quota sampling
Quota sampling is quite popular in market research. In this case, an interviewer simply chooses samples based on a required quota for any research/study. For example, a smartphone company might want to investigate what age group using which brand of phone in a city. Then they apply quotas e.g., 100 people from each of these age groups e.g., 21–30, 31–40, 41–50 or they can further impose quotas based on gender e.g., 50 male and 50 female from each age group. They will choose the samples based on ease of access and their research budget. However, as mentioned earlier being a non-probabilistic method, quota sampling may still have voluntary bias.
c. Purposive Sampling
In Purposive sampling, the Researchers apply their expert knowledge of the target population in order to select the "representative" sample in a non-random manner. Purposive sampling is definitely time-and cost-effective but in addition to volunteer bias, this type of sampling can cause judgment bias/error by the researcher in choosing participants.
3. Probability Distributions
Probability distributions are crucial for understanding any data structure. There are primarily two types of distributions – Discrete and Continuous.
a. Discrete probability distributions
These distributions model the probabilities of random variables that have discrete values as outcomes. For example, if X is a discrete random variable describing the number of heads in a toss, we can repeat the toss many times and record the probability of each possible outcome. If we represent this in a form of function then it will be called a "probability mass function (PMF)" which defines the probability of the discrete random variable X taking on a particular value x. Examples of such distributions are Bernoulli, Binomial, Poisson.
b. Continuous probability distributions: On the other hand, the continuous probability distributions model the probabilities of random variables that can have any possible outcome which is essentially continuous in nature. For example, the possible values for the random variable X that represents weights of citizens in a town can have any value like 54.5, 47.2, 60.3, etc. Examples can be Normal, Student’s T, Chi-square, Exponential distribution, etc.
4. Hypothesis Testing
We use a Hypothesis to make a claim or assumption about the population parameter. Now in order to verify our claim on the basis of the sample data that we collected we create two hypotheses – null and alternative.
The null hypothesis reflects a researcher’s neutral opinion about the outcome of the hypothesis test. It basically states the default situation e.g., in a medicine efficacy experiment, the null hypothesis can be that there is no difference in cure rate between the patients who got the medicine and the group of patients who got a placebo. The exact opposite statement of the null hypothesis will be the alternative hypothesis. Alternative hypothesis always challenges the null hypothesis. The decision to accept or reject the null hypothesis is done based on the sample data.
5. Confidence level
Continuing with the hypothesis testing concepts it is very important to understand what the confidence level is. The confidence level in hypothesis testing is the probability of not rejecting the null hypothesis when the null hypothesis is actually true. It conveys how sure we are about getting the same results if we repeat the experiment again. So, we construct a confidence interval which is an interval estimation of a parameter generated from statistical inference. This interval is calculated by the following formula:
[point estimation ± critical value based on sample distribution at chosen confidence level*standard deviation of sample]
For example, if we have a sampling distribution with mean 60 and standard deviation 5, then the 95% confidence interval will range from (60–1.965) i.e., 50.2 and (60 +1.965) i.e., 69.8. Here, 1.96 is the critical value which comes from the fact that for a normal distribution, 95% area lies within a standard deviation of 1.96
We can never be 100 % confident in statistics as we will always have some uncertainties. Hence, we mostly come across 99%, 95%, or 90% confidence levels in statistical studies.
6. p values and level of significance
For any statistical inferential study we need to make a decision about rejecting/accepting the null hypothesis and that decision is made based on the observed values of the randomized sample. However, there is always a probability that the conclusion that we are drawing from the samples about the population is wrong. The error associated with rejecting a null hypothesis when it is actually true is known as Type I error and the error associated with accepting the null hypothesis when it is false is called Type II error. Now, let’s assume, the probability of committing a type I error is some number, α, which is the level of significance any researcher sets for his study. An α of 5% indicates that we are willing to accept a 5% chance that we are wrong when we reject the null hypothesis. On the other hand, the p-value is the observed significance level which gives us the probability of obtaining an effect as extreme as the one calculated from our sample data when the null hypothesis is true.
7. A/B testing
A/B testing is a randomized experiment with two groups, A and B. In a very simple language, it is a way to compare two variants of a single variable to find out which variant performs better than the other in a controlled environment. This technique is often used for marketing strategies to improve customer satisfaction.
For example, two marketing emails with different call to actions can be sent to the 30% of the total customer base of a company just to test which one has a higher clicking rate. The marketing team then wait for a few hours and they choose to send the email with more clicks to rest of the 70% customers
8. Z test and t-test
Both z test and t-test are very useful in hypothesis testing. Z tests are typically used when we want to understand if two sample means are significantly different from each other when the sample size is large i.e., n ≥ 30, and also the population standard deviation is known to us. On the other hand, t-tests can be used to compare two sample means for a small sample size n >0 and even when the population standard deviation is not known to us. t-test follows student’s t- distribution whereas the z test assumes that the sample distribution is normal. In the case of a large sample the Z test and t-Test will tend to give us the same results as for a large enough sample size, t-distribution approaches the normal distribution and the difference between the z score and t score becomes negligible. Now, t-tests can be used to perform a One-Sample t-test when we want to understand if a sample mean is significantly different than the population average. Since we don’t know the population standard deviation in the case of a t-test we use the sample standard deviation.
9. Difference between Covariance and Correlation
The covariance measure gives us the direction of the linear relationship between two variables. On the other hand, Correlation gives us both the strength and direction of the relationship between those two variables. If we think about the mathematical formula, we can calculate the correlation coefficient of two variables by dividing the covariance of these variables by the product of the standard deviations of the same variables. So, correlation values are standardized. Hence, the Correlation coefficient lies between -1 to +1 but covariance can lie between -∞ to +∞.
10. Linear regression vs logistic regression
Linear Regression is commonly used when the dependent variable is continuous. One of the key assumptions of linear regression is that there is a linear relationship between dependent and independent variables. On the other hand, Logistic Regression is used when the dependent variable is binary. It predicts the probability of occurrence of an event by fitting the data into a logit function. For Logistic regression, we do not need a linear relationship between the dependent and independent variables.
Linear Regression fits a straight line in the data while Logistic Regression fits a curve to the data. Linear regression assumes the Gaussian (or normal) distribution of the prediction error. However, the dependent variable or independent variables of the linear regression-based model need not follow the normal distribution. Sometimes when the training data has a few outliers or the dependent variable is too skewed, it is possible that the resulting error in the model also does not follow the normal distribution. Logistic regression assumes the binomial distribution of the dependent variable.
11. Central limit theorem and Law of large numbers
The central limit theorem states that regardless of a particular variable’s distribution in the population when we are collecting samples from the population the sampling distribution of the variable mean will approach the normal distribution as the sample size increases. The Central Limit Theorem is very important in statistics for two main reasons – the normality assumption is crucial for parametric hypothesis testing and the precision of the estimates. Often in real-life data, we come across the non-normal distribution. So, we might get worried that the hypothesis tests that we are conducting or the parameter estimates we are getting are not giving us accurate results. However, if we have a large sample size (>30), then CLT allows us to use the tests and the estimates even when the data is not normally distributed.
On the other hand, as per the Law of Large Numbers as we increase the number of trials in an experiment the average result of the trials will eventually approach the true population average. For example, we are more likely to see Heads coming up half of the time when we toss a fair coin 1000 times as compared to tossing the same coin only 10 times.
12. Maximum Likelihood Estimation (MLE)
Maximum Likelihood Estimation involves estimating the parameter by maximizing the likelihood function to find the parameters that best explain the observed data. MLE is such a predictive modeling framework where model parameters are found through an optimization problem. Here, the likelihood function p(y| θ ) describes the likelihood of observing data y given parameter θ. We solve the optimization problem by maximizing the set of parameters (theta) that gives us the largest likelihood value. MLEs work well with large data sizes and gives us unbiased minimum variance estimators.
EndNote:
Thanks for reading!
Having a very strong understanding of Statistics is a prerequisite for being successful as a Data Scientist. I hope this article will help you to prepare for your next Data Science interview. The 12 concepts that I have discussed here will set up the focus areas for you to learn further.
