Learn & Review: Master Regression Analysis Full Course

Regression Analysis: A Comprehensive Tutorial

This tutorial provides a comprehensive guide to regression analysis, covering its fundamentals, applications, and various types. It is structured into four main lessons: an overview, simple linear regression, multiple linear regression, and logistic regression.

1. Overview of Regression Analysis

• Definition: Regression analysis is a method for modeling relationships between variables, allowing for the inference or prediction of one variable based on one or more other variables.
• Key Terms:
  • Dependent Variable: The variable to be inferred or predicted.
  • Independent Variable(s): The variable(s) used for prediction. These may also be called predictor variables, input variables, response variables, output variables, or target variables.
• Goals of Regression Analysis:
  • Measuring Influence: To understand the impact of one or more variables on another variable. This is often research-based and applied in social and economic sciences.
    • Example: Investigating factors influencing children's concentration ability without necessarily predicting it.
    • Example: Determining if parental education level and place of residence influence a child's future educational level.
  • Prediction: To predict a variable based on other variables. This is more application-oriented and focuses on enhancing efficiency.
    • Example: Predicting a patient's hospital stay duration based on their characteristics to optimize hospital planning.
    • Example: Suggesting products to online store visitors based on their predicted purchase likelihood to increase sales.

2. Types of Regression Analysis

• Simple Linear Regression:
  • Uses one independent variable to predict the dependent variable.
  • Example: Predicting a person's salary using only their years of education.
• Multiple Linear Regression:
  • Uses several independent variables to predict or infer the dependent variable.
  • Example: Predicting a person's salary using their highest level of education, weekly working hours, and age.
• Logistic Regression:
  • Used when the dependent variable is categorical.
  • Most common form is binary logistic regression, where the outcome variable has two possible values (e.g., yes/no, success/failure).
  • Example: Predicting whether a person is at risk of burnout (yes/no).
  • Example: Predicting whether a person is deceased or not.

Variable Measurement Levels:

• Dependent Variable:
  • Metric: In linear regression (simple and multiple), the dependent variable must be metric (continuous numerical values).
    • Examples: Salary, body size, electricity consumption.
  • Categorical: In logistic regression, the dependent variable is categorical.
    • Examples: Burnout risk (yes/no), deceased status (yes/no), animal type.
• Independent Variables:
  • Can be nominal, ordinal, or metric.
  • Categorical independent variables with more than two categories require the creation of dummy variables.

Examples of Regression Questions:

• Simple Linear Regression: Does weekly working time have an impact on hourly wage?
• Multiple Linear Regression: Do weekly working hours and age influence hourly wage? (At least two independent variables).
• Logistic Regression: Do weekly working hours and age influence the probability of being at risk of burnout (yes/no)?

3. Simple Linear Regression in Detail

• Purpose: To understand the relationship between two variables and predict one based on the other.
• Example: Predicting house prices (dependent variable) based on house size in square feet (independent variable).
• Calculation:
  • Requires data for both variables.
  • Visualized using a scatterplot with the independent variable on the x-axis and the dependent variable on the y-axis.
  • The goal is to find a regression line that minimizes the errors (distances between actual data points and the line).
• Regression Equation:
  • y = a + bx
    • y: Dependent variable (e.g., house price)
    • x: Independent variable (e.g., house size)
    • b: Slope of the line. Indicates how much the dependent variable changes for a one-unit increase in the independent variable.
    • a: Y-intercept. The predicted value of the dependent variable when the independent variable is zero. (Note: May not be meaningful in all contexts, e.g., predicting price for a house of zero size).
• Calculating Coefficients (b and a):
  • Can be done manually or using statistical software.
  • Formula for slope (b): b = r * (Sy / Sx)
    • r: Correlation coefficient between x and y.
    • Sy: Standard deviation of the dependent variable (y).
    • Sx: Standard deviation of the independent variable (x).
  • Formula for intercept (a): a = ȳ - b * x̄
    • ȳ: Mean of the dependent variable (y).
    • x̄: Mean of the independent variable (x).
• Assumptions of Simple Linear Regression:
  1. Linear Relationship: The relationship between the independent and dependent variables is linear.
  2. Independence of Errors: The errors (differences between actual and predicted values) are independent of each other.
  3. Homoscedasticity (Equal Variance of Errors): The spread of errors is roughly the same across all values of the independent variable.
  4. Normally Distributed Errors: The errors are normally distributed.
• Checking Assumptions:
  • Linearity: Scatterplot of independent vs. dependent variable.
  • Normality of Errors: QQ plot or analytical tests (caution with analytical tests for small/large samples).
  • Independence of Errors: Durbin Watson test.
  • Homoscedasticity: Residual plot (errors vs. predicted values). A funnel shape indicates heteroscedasticity.
• Statistical Significance (p-value):
  • Helps determine if the relationship between variables is statistically significant or due to random chance.
  • Null Hypothesis (H0): There is no relationship between the independent and dependent variable.
  • If p-value < 0.05, reject H0, indicating a significant relationship.
  • If p-value > 0.05, fail to reject H0, indicating no strong evidence of a relationship.

4. Multiple Linear Regression in Detail

• Purpose: To model relationships between variables using two or more independent variables to predict or infer a dependent variable.
• Regression Equation: ŷ = a + b1*x1 + b2*x2 + ... + bk*xk
  • ŷ: Predicted value of the dependent variable.
  • a: Intercept.
  • b1, b2, ..., bk: Coefficients for each independent variable (x1, x2, ..., xk).
  • ŷ is used instead of y to denote predicted values, acknowledging potential errors in real-world data.
• Interpretation of Coefficients:
  • a: The predicted value of ŷ when all independent variables are zero.
  • bi: Indicates the change in ŷ for a one-unit increase in xi, assuming all other independent variables remain constant.
• Standardized Coefficients:
  • Useful for comparing the relative importance of independent variables measured in different units.
  • They are calculated after standardizing all variables to the same scale.
• Assumptions of Multiple Linear Regression:
  • The first four are similar to simple linear regression:
    1. Linear Relationship (can be checked by plotting each independent variable against the dependent variable).
    2. Independence of Errors (Durbin Watson test).
    3. Homoscedasticity (Residual plot).
    4. Normally Distributed Errors (QQ plot or analytical tests).
  • Fifth Assumption: No Multicollinearity:
    • Multicollinearity: Occurs when two or more independent variables are highly correlated with each other, making it difficult to separate their individual effects.
    • Problem: Makes it hard to determine the unique contribution of each independent variable (coefficients b1, b2, etc.). Less critical if the model is solely for prediction, but crucial for assessing individual variable influence.
    • Detection:
      • Run auxiliary regressions where each independent variable is regressed on the others.
      • Calculate R-squared for these auxiliary regressions. A high R-squared indicates multicollinearity.
      • Calculate Tolerance (1 - R-squared) and Variance Inflation Factor (VIF).
        • Tolerance < 0.1 or VIF > 10 are warning signs.
    • Addressing Multicollinearity:
      • Remove one of the correlated variables.
      • Combine correlated variables (e.g., by averaging).
• Model Summary Table:
  • Multiple Correlation Coefficient (R): Measures the correlation between the actual dependent variable and the predicted values (ŷ). A higher R indicates a better fit.
  • Coefficient of Determination (R-squared): Indicates the proportion of variance in the dependent variable explained by the independent variables.
  • Adjusted R-squared: Accounts for the number of independent variables in the model, providing a more accurate measure of explanatory power, especially when many variables are included.
  • Standard Error of the Estimate: Measures the average distance between observed data points and the regression line, indicating the typical error in predictions.
• Handling Categorical Independent Variables (Dummy Variables):
  • Nominal Variables: Variables with categories (e.g., gender, vehicle type).
  • Binary Variables (Two Categories): Code one category as 0 (reference category) and the other as 1. The coefficient (b) then represents the difference between the coded category and the reference category.
  • Multiple Categories: Create dummy variables for each category except one (the reference category).
    • Example: For "vehicle type" with categories (sedan, sports car, family van), create dummy variables for "sports car" and "family van". If a vehicle is a sedan, both dummy variables would be 0.
    • The number of dummy variables needed is (number of categories - 1).
    • Statistical software often handles this automatically.

5. Logistic Regression in Detail

• Purpose: Used when the dependent variable is binary (has two possible outcomes). It models the probability of an outcome occurring.
• Difference from Linear Regression:
  • Linear regression predicts a metric dependent variable.
  • Logistic regression predicts the probability of a binary dependent variable.
  • Linear regression can produce values outside the 0-1 range, while logistic regression is constrained to this range using the logistic function.
• Logistic Function: A mathematical function that maps any real-valued number into a value between 0 and 1.
• Equation: The logistic regression equation relates the independent variables to the probability of the dependent variable being 1: P(Y=1 | X) = 1 / (1 + e^-(a + b1*x1 + ... + bk*xk))
• Calculation: Coefficients are estimated using the Maximum Likelihood Method.
• Interpretation of Results:
  • Model Significance (Chi-Square Test): Evaluates if the model as a whole is statistically significant. A p-value < 0.05 indicates the model is significant.
  • Classification Table: Shows how many observations were correctly and incorrectly classified by the model.
  • Probability Threshold: A threshold (commonly 50%) is set to classify an outcome. If the predicted probability exceeds the threshold, the outcome is classified as 1; otherwise, it's classified as 0.
  • Model Summary (R-squared variations): Measures how well the model explains the dependent variable (e.g., Cox & Snell R-squared, Nagelkerke R-squared).
  • Model Coefficients (b):
    • The coefficients themselves are used in the logistic regression formula to calculate probabilities.
    • p-value: Indicates the statistical significance of each independent variable's coefficient. If p < 0.05, the variable has a significant influence.
    • Odds Ratio (OR):
      • Odds: The ratio of the probability of an event happening to the probability of it not happening (P / (1-P)).
      • Odds Ratio: Compares the odds of an event occurring in two different groups or for a one-unit change in an independent variable.
        • OR > 1: The event is more likely to occur in the first group or with an increase in the variable.
        • OR < 1: The event is less likely to occur.
        • OR = 1: No difference in odds.
      • In logistic regression, the odds ratio for a variable can be calculated by exponentiating its coefficient (e^b).
      • Example: An OR of 0.64 for medication means that patients taking the medication have 0.64 times the odds of being deceased compared to those not taking it.
      • Example: An OR of 1.04 for age means that for each one-year increase in age, the odds of being deceased increase by a factor of 1.04.