blog/multiple-linear-regression-made-simple/

[giscus[bot]]

blog/multiple-linear-regression-made-simple/

Learn how to run multiple and simple linear regression in R, how to interpret the results and how to verify the conditions of application

https://statsandr.com/blog/multiple-linear-regression-made-simple/

[foumine]

Merci à vous pour ce article bien détaillé. Puis-je savoir comment afficher l'équation de la droite de regression multiple sur le graphe? merci bien

[AntoineSoetewey]

Merci pour votre question.

Je suppose que vous vouliez dire régression simple au lieu de multiple (on visualise rarement une régression multiple vu qu'il y a plus de 2 dimensions).

Je viens de mettre à jour la section "Visualizations" en ajoutant du code pour afficher la droite de régression et le coefficient de détermination directement sur le scatterplot.

En espérant que cela réponde à votre question.

Bien à vous,
Antoine

[foumine]

oui effectivement, je voulais dire régression simple. La section ''Visualization'' répond à ma question. Merci à vous.

[AntoineSoetewey]

De rien !

[MKP2021]

Thanks AS for this post. it clarified a lot of issues related to regression analysis. Still I could not clarify one of my doubt. I have data imported to 'R' for a randomised trail having two groups. The dependant variable is continuous (cough score measured by VAS scale) between two groups (experimental and control) and the predictors are mixed - some are continuous (age, Duration of bronchoscopy) and some are categorical (indication - lung cancer versus others). When I will run multilinear regression in R, whether regression should be performed for each group (experimental and control) separately or for the whole data of both groups. "Study Group"(0/1) is one column heading in the excel/csv data set. Whether the "Study Group" to be added as one independent variable in to the model.
Thanks in advance

[AntoineSoetewey]

Hello,

It depends:

1. If your goal is to compare the experimental versus the control group in terms of the cough score (while controlling for the potential confounding effects of the independent variables), yes you can do the linear regression on the whole dataset and include the variable Study Group as an independent variable. If the coefficient for the variable Study Group is significantly different from 0 (see the p-value at the end of the coefficients table in R), it means the two groups are significantly different from each other in terms of cough score, all else being equal.
2. If your goal is to measure or describe the relationships between the dependent and independent variables for each study group separately, you can run a linear regression for each group separately.

Hard for me to tell which one to choose without knowing your research question. That being said, if all assumptions are met in both approaches and for a similar quality of adjustment, I would go for the first one because with this approach you'll be able to isolate the relationship between the independent and dependent variables, while controlling for effect of the study group.

Hope this helps.

Regards,
Antoine

[MKP2021]

Thanks AS.
I am much relived now. I will run the model as per your suggestion.

I faced another issue that I mentioned below.
Going through your above post on multilinear regression, I tried to install the following packages (performance & see) but faced some issues. Typing the extract from "R' console below:

> install.packages("performance")
also installing the dependencies ‘bayestestR’, ‘insight’, ‘datawizard’

The downloaded binary packages are in
	/var/folders/h3/ddtkm3j1243_hd0kp9_4y_dr0000gn/T//RtmpMn1Xge/downloaded_packages
> install.packages("see")
also installing the dependencies ‘colorspace’, ‘farver’, ‘labeling’, ‘munsell’, ‘RColorBrewer’, ‘viridisLite’, ‘digest’, ‘gtable’, ‘isoband’, ‘scales’, ‘plyr’, ‘effectsize’, ‘ggplot2’, ‘ggridges’, ‘parameters’


  There is a binary version available but the source version is later:
       binary source needs_compilation
digest 0.6.28 0.6.29              TRUE

Do you want to install from sources the package which needs compilation? (Yes/no/cancel)

installing the source package ‘digest’

trying URL 'https://cran.rstudio.com/src/contrib/digest_0.6.29.tar.gz'
Content type 'application/x-gzip' length 162775 bytes (158 KB)
==================================================
downloaded 158 KB

* installing *source* package ‘digest’ ...
** package ‘digest’ successfully unpacked and MD5 sums checked
** using staged installation
** libs
xcrun: error: invalid active developer path (/Library/Developer/CommandLineTools), missing xcrun at: /Library/Developer/CommandLineTools/usr/bin/xcrun
ERROR: compilation failed for package ‘digest’
* removing ‘/Library/Frameworks/R.framework/Versions/4.1-arm64/Resources/library/digest’
Warning in install.packages :
  installation of package ‘digest’ had non-zero exit status

library(performance)

check_model(model2)
Error: Package 'patchwork' is required for this function to work.
  Please install it by running install.packages('patchwork').
> install.packages("patchwork")
trying URL 'https://cran.rstudio.com/bin/macosx/big-sur-arm64/contrib/4.1/patchwork_1.1.1.tgz'
Content type 'application/x-gzip' length 3240613 bytes (3.1 MB)
==================================================
downloaded 3.1 MB


The downloaded binary packages are in
	/var/folders/h3/ddtkm3j1243_hd0kp9_4y_dr0000gn/T//RtmpMn1Xge/downloaded_packages
> library(patchwork)
> check_model(model2)
Error in loadNamespace(x) : there is no package called ‘digest’
> check_model(model2)
Error in loadNamespace(x) : there is no package called ‘digest’
> install.packages("digest")

  There is a binary version available but the source version is later:
       binary source needs_compilation
digest 0.6.28 0.6.29              TRUE

Do you want to install from sources the package which needs compilation? (Yes/no/cancel) 
installing the source package ‘digest’

trying URL 'https://cran.rstudio.com/src/contrib/digest_0.6.29.tar.gz'
Content type 'application/x-gzip' length 162775 bytes (158 KB)
==================================================
downloaded 158 KB

* installing *source* package ‘digest’ ...
** package ‘digest’ successfully unpacked and MD5 sums checked
** using staged installation
** libs
xcrun: error: invalid active developer path (/Library/Developer/CommandLineTools), missing xcrun at: /Library/Developer/CommandLineTools/usr/bin/xcrun
ERROR: compilation failed for package ‘digest’
* removing ‘/Library/Frameworks/R.framework/Versions/4.1-arm64/Resources/library/digest’
Warning in install.packages :
  installation of package ‘digest’ had non-zero exit status
library(digest)
Error in library(digest) : there is no package called ‘digest’
> check_model(model2)
Error in loadNamespace(x) : there is no package called ‘digest’

this suggest that the dependencies "digest" under the "see" package is not installed. Rest all dependencies are installed along with the "install.packages(see) command.

Because of this - I am unable to run the check_model() command.

Any solution for this.

Troubling you again.

Thanks in advance.

[AntoineSoetewey]

Try:

install.packages("digest", dependencies = TRUE)

and then click no for the dialog box that opens with it.

[MKP2021]

Dear AS Thank you so much for the guidance I could run the check_model () command successfully. Just attaching the plot I got. I think, except for the normality of residuals & homogeneity of variance (bottom left plot and top right), rest are okay to proceed with multilinear regression.  Regards Manoj

[AntoineSoetewey]

You can try to apply the logarithmic transformation to see if it helps.

[MKP2021]

thanks

[Hesham999666]

I am very thankful to the article, codes and the simple explanation it is great work. Looking forward to Quantile regression if it is possible.

I do have a big epidemiological with many independent variables, we are studying effect of occupants and home characteristic on indoor air bacteria in five cities in North Europe. data collected in different seasons.

However, some of this independent variables have missing data as people did not answer all the questions
I am doing multiple linear regression using three models.

Model 1 = key factors (city+season)
Model 2 = Key factors + home characteristic factors.
Model 3 = Key factors + home characteristic factors + Occupant factors .

The Dependent variables are bacterial diversity (Shannon index) (contentious) and Bacterial gene copy number (16SrRNA)

In Model 1, I don't have missing data of my as independent variable (city or season).
the other independent variables home characteristic factors and Occupant factors there are missing some data.

Is there right way to deal with missing data in the independent variables ?

Another thing one of my dependent variable was not normal distributed ( Bacterial gene copy number (16SrRNA) ). So I have log transformed it. Is log-transformation are often challenging to interpret for journal readers and not relevant for the original, non-transformed data. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4120293/
I was advised not to use ( log-transformation + multiple linear regression ) and use the Quantile regression instead.
what is your opinion in that ?

Thanks for your time.

[AntoineSoetewey]

Dear Hesham,

Regarding the missing values first. By default, linear regression models in R work with the complete cases of the data, that is, they exclude the cases in which there is at least one NA.

Here you will see 3 concrete approaches to handle missing data in practice: https://bookdown.org/egarpor/PM-UC3M/app-nas.html (approximately in the middle of the page).

So to recap, if you have a large sample and a high proportion of fully-observed cases, missing values should not be an issue. If you lose a substantial amount of data when restricting to fully-observed cases, you could remove predictors with many missing data or use imputation for the missing values (the simplest approach is to replace the missing data with the sample mean or median for quantitative variables, and mode for qualitative variables).

It is true that log-transformation are more challenging to interpret (here is an easy guide though). That being said, some authors argue that the normality assumptions may be relaxed for large samples. See the section about normality in these two sections: 1 & 2. So depending on what you want to do afterwards and your sample size, it is also possible to relax this normality assumption.

Regarding quantile regression, I prefer to refrain from discussing about it because I am not familiar enough with this approach.

Hope this helps.

Regards,
Antoine

[Hesham999666]

Thanks Antoine for the information.

[Hesham999666]

In multiple regression with categorical variables do I need to change data type from character to factor type data or I keep them as character when I tested both I got same results.

Hesham

[AntoineSoetewey]

I advise to change to factor if it is a categorical variable.

Even if you obtain the same results for a linear regression, some functions (not necessarily related to linear regression) requires factor variables and not character.

Regards,
Antoine

[fabric74]

Hi Antoine,

Thanks a lot for this blog on R, I find it great! Actually, I am new on R and I am trying to move my first steps (after graduating in a MSc in Epidemiology&Biostat) and most of your articles were of great help.

• I would firstly ask you if there could be a future article regarding logistic regression.

• My second question is related to data manipulation: specifically, I would recode a continuous variable (e.g. height) into a new categoric one (e.g. height classes: <160 cm, 160-180 cm, >180 cm...): how do I do it on R?

Many thanks in advance for your kind support.

Best regards,
Fabrizio

[AntoineSoetewey]

Hello Fabrizio,

Thanks for your feedback!

I would firstly ask you if there could be a future article regarding logistic regression.

It is on my to do list, but don't expect it soon. If by any chance you have some knowledge of logistic regression in R, feel free to let me know so we could write something together.

My second question is related to data manipulation: specifically, I would recode a continuous variable (e.g. height) into a new categoric one (e.g. height classes: <160 cm, 160-180 cm, >180 cm...): how do I do it on R?

You can easily do so with the cut() function, see more in this section.

For example:

dat <- data.frame(height = rnorm(100, mean = 170, sd = 10))

dat$height_cat <- cut(dat$height,
  breaks = c(0, 160, 180, 999), # cut points
  right = FALSE # closed on the left, open on the right
)

You can then easily edit the labels for each level with fct_recode() from the {forcats} package (see also this section).

## Recoding dat$height_cat
library(forcats)
dat$height_cat <- fct_recode(dat$height_cat,
  "<160 cm" = "[0,160)",
  "160-180 cm" = "[160,180)",
  ">=180 cm" = "[180,999)"
)

Hope this helps.

Regards,
Antoine

[LucyMRAc]

Hi, I am conducting a study. I am looking at the effect of self-esteem on the interpretation of ambiguous messages and emojis. Participants took part in both conditions. The first they had to rate emojis on a 5-point likert scale, the second they had to rate messages on the same 5-point likert scale. Then their self-esteem was measured by the RSES. Is this a multiple linear regression? Thanks!

[AntoineSoetewey]

Hello @LucyMRAc

Not sure I fully understand your design experiment. So what is your dependent variable (the outcome)? Is that a quantitative or qualitative?

Then what is(are) the independent variable(s)? What are their types (qualitative or quantitative)?

[LucyMRAc]

Hi! There are two independent variables:

1. self esteem (high and low) - measured via the RSES (a 5-point likert scale)
2. task (emoji and sentence) - participants had to rate emojis and sentences via 5 point likert scales ranging from extremely positive to extremely negative.

My dependent variable will be the scores on these

[AntoineSoetewey]

As the same participants are measured twice on the same outcome (emoji and sentence), observations are not independent so the independence assumption required in linear regression is violated.

You could:

1. run 2 linear regressions, one for each task
2. compute the difference of score between the two tasks and take this difference as the dependent variable
3. use a linear mixed model

Hope this helps.

Regards,
Antoine

[Sowmya-R-Krishnan]

Thank you for this detailed article clarifying all possible aspects of regression! I have some questions regarding correcting the linearity defect in the model. You have mentioned that the relationship between mpg and horsepower is not linear, which can be corrected with log-transformation or something else to correct the model linearity.
a) Doesn't it conflict with the multi-collinearity condition required? What if making a transformation induces the variables to be "too" linear? In that case is it good to omit the variable rather than admitting a slight linearity defect in the model? What is advisable in such a situation?
b) Can a mix of log-transformed independent variables and non-transformed independent variables be used to build a MLR model?

Thank you once again for the wonderful article!

[AntoineSoetewey]

Dear @Sowmya-R-Krishnan,

Thanks for your questions.

a) If you apply a transformation to one or several variables (because it helps for linearity for instance), it is true that multi-collinearity must be checked again with these transformed variables. That being said, I have rarely seen that a (log) transformation induced the variables to be so linear that it becomes "too" linear and multi-collinearity arises. Are you asking from a theoretical point of view, or because it happened to you with your data? If that is the case with your data, I cannot tell whether it is best to omit the variable or accept a small linearity defect without seeing the magnitude of the linearity defect and the VIF, but bear in mind that:

• linearity is very important for a linear regression, and
• the VIF is acceptable if below 10.

b) Of course. You can have a mix of transformed and non-transformed variables. The interpretations should be done accordingly though.

Hope this helps.

Regards,
Antoine

[Sowmya-R-Krishnan]

Thanks a lot for your detailed answers! I was asking from a purely theoretical point of view - I have not encountered this so far with any of my datasets.

[riscas14]

Hello Antoine,

Thanks for this amazing article.

I am conducting an analysis on air routes, where the main goal is to determine which variables affect the efficiency of these routes. One of my independent variables is the time that it takes to go from one city to another, by public transports. With that being said, in some cases there are no public transports that connect the two cities. Since this is not exactly "missing data", I was trying to come up with a way to model the existence of public transports combined with the time. Mathematically, I want to do something like this:

efficiency = intercept + b1 * if_public_transports + b2 * if_public_transports * time_public_transports,

where "if_public_transports" is a dummy variable: value 1 if public transports exist; 0 otherwise. The reason behind this is because I expect air routes without competition, from other public transports, to be more efficient, since they are expected to attract more passengers. Thus, is very important to keep these two variables somehow.

The problem is: I can't run this linear regression because there is a high correlation between the two variables, so I get an error from RStudio (coefficients not defined due to singularities). Moreover, I don't know if this is a valid formula or a valid solution for what I am trying to model. What do you think I should do?

[AntoineSoetewey]

Hello,

Good question.
Can you contact me via www.statsandr.com/contact ? This way you’ll be able to send me the data and I’ll try on my side a few things.

Regards,
Antoine

[AntoineSoetewey]

The email address you specified in the form seems to be invalid (emails cannot be delivered). Please fill the form again with a valid email address. Thanks !

[andre-luchetti]

How to do a linear regression with n different errors in my n x variables and/or in my n y variables?
Even a simple, teaching site like PhET does it (allowing different error values for the y values: https://phet.colorado.edu/sims/html/curve-fitting/latest/curve-fitting_all.html) but I can't find how! People generally use only 1 value for the error (generally, the standard deviation).

[AntoineSoetewey]

Dear @andre-luchetti,

I'm sorry, I don't understand your question.

Could you please rephrase it?

Regards,
Antoine

[andre-luchetti]

I'm sorry, @AntoineSoetewey !
I'm looking for a method of linear regression that considers the errors for my x and y values.
So, in addition to the 2n values of my n points (x,y), I'll have 2n values of errors (each x and each y has its own error).
I've been searching for such a method for a while, and I've just found out this video (https://www.youtube.com/watch?v=FafES7bB-XY) proposing the "Williamson-York Iterative Bivariate Fit Shell", referencing Chris A. Cantrell (30 June 2008).
It seems to accomplish what I was looking for.
But I'm really surprised it is such a rare approach.
I'm a teacher in Experimental Physics for Chemistry courses, and every material suggests only the Least Square Method for the analysis we need to do. The individual errors of each x-value and y-value, however, were relevant to some discussions, so I would like to see their influence on the final values of m and b (in our y=mx+b tendency line) and on their errors.

Thank you so much for your attention.
André S. Luchetti
Universidade do Estado de São Paulo (Unesp)
Araraquara-SP, Brazil

[AntoineSoetewey]

Thanks for the clarification @andre-luchetti! Now I understand your question.

It is true that ordinary least square is so common and present in so many fields, up to the point that other methods are often not covered. And unfortunately I’m not familiar with the method you mention, so I cannot help you further on that matter.

If you are interested, I can give you the contact details of a colleague who may be able to help you. I cannot guarantee that he’ll be able to help you because your method is not so common, but it doesn’t cost anything to ask.

Let me know if you’re interested.

Regards,
Antoine

[andre-luchetti]

Thank you so much, @AntoineSoetewey !
And by the way, congratulations for your website, and your work towards the popularization of statiscal tools!

I'm interested in contacting your friend, if you would be so kind.
I'm also interested in a mid-term approach: accounting for individual errors in the y variables, still neglecting them for the x variables (that would be the approach behind the PhET simulation I shared above).

Best regards!
André

[AntoineSoetewey]

You're welcome!

Please send me an email (I believe my email address is visible in my GitHub profile and I'll send you his contact details.

This is to avoid writing his email publicly.

Regards,
Antoine