blog/correlogram-in-r-how-to-highlight-the-most-correlated-variables-in-a-dataset/

[utterances-bot]

Correlogram in R: how to highlight the most correlated variables in a dataset - Stats and R

Make the the most correlated variables stand out via a correlogram. See also how to enhance a correlation plot to show significant correlations among variables

https://statsandr.com/blog/correlogram-in-r-how-to-highlight-the-most-correlated-variables-in-a-dataset/

[AntoineSoetewey]

Comment written by Rick on February 24, 2020 02:47:58:

please look at my generalCorr package on CRAN which computes generalized correlations
it allows for nonlinear relations, generally better fits as u can see
easy to use single command
generalized matrix is nonsymmetric
cause is in the column and effect along the rows
relative size of off-diagonals help identify causal paths

> options(np.messages=FALSE)  
> gmcmtx0(mtcars)  
            mpg        cyl       disp         hp        drat         wt  
mpg   1.0000000 -0.8557900 -0.9508994 -0.9379374  0.68455456 -0.9162983  
cyl  -0.9433125  1.0000000  0.9759183  0.9583212 -0.75124945  0.8717790  
disp -0.8941676  0.9151419  1.0000000  0.9306311 -0.76973716  0.9014653  
hp   -0.8530474  0.8446589  0.8170031  1.0000000 -0.55427987  0.6931286  
drat  0.6878267 -0.7015970 -0.9458881 -0.7434288  1.00000000 -0.7502746  
wt   -0.9174405  0.7823903  0.9678176  0.9187065 -0.73035365  1.0000000  
qsec  0.7512435 -0.5908602 -0.6090980 -0.7539446  0.02921025 -0.1878419  
vs    0.6942734 -0.8141714 -0.7820383 -0.8418120  0.51150265 -0.6366761  
am    0.6031004 -0.5224062 -0.6715676 -0.5603310  0.98069420 -0.8383474  
gear  0.4920451 -0.4957241 -0.6123745 -0.5830729  0.87635616 -0.6124648  
carb -0.5572555  0.5681535  0.6718953  0.8553318 -0.84876146  0.5966302  
           qsec         vs         am       gear        carb  
mpg   0.7383513  0.6640279  0.5997807  0.6529907 -0.65305643  
cyl  -0.8496465 -0.8108103 -0.5225462 -0.7274627  0.67555239  
disp -0.7613347 -0.7104124 -0.5912007 -0.7666091  0.50701342  
hp   -0.9267578 -0.7230940 -0.2416089 -0.6629877  0.78661538  
drat  0.5493953  0.4402134  0.7127009  0.8316282 -0.02225962  
wt   -0.7718667 -0.5548968 -0.6924849 -0.6575598  0.59412177  
qsec  1.0000000  0.7445309 -0.2284076 -0.6333628 -0.66101228  
vs    0.9505094  1.0000000  0.1649016  0.6176113 -0.69219670  
am   -0.6116860  0.1665178  1.0000000  0.8089357  0.34500941  
gear -0.6150966  0.2052627  0.7940532  1.0000000  0.55254965  
carb -0.7746992 -0.5695923  0.0169436  0.4324314  1.00000000  

[AntoineSoetewey]

Comment written by Rick on February 24, 2020 02:47:58:

please look at my generalCorr package on CRAN which computes generalized correlations
it allows for nonlinear relations, generally better fits as u can see
easy to use single command
generalized matrix is nonsymmetric
cause is in the column and effect along the rows
relative size of off-diagonals help identify causal paths

> options(np.messages=FALSE)
> gmcmtx0(mtcars)
mpg cyl disp hp drat wt
mpg 1.0000000 -0.8557900 -0.9508994 -0.9379374 0.68455456 -0.9162983
cyl -0.9433125 1.0000000 0.9759183 0.9583212 -0.75124945 0.8717790
disp -0.8941676 0.9151419 1.0000000 0.9306311 -0.76973716 0.9014653
hp -0.8530474 0.8446589 0.8170031 1.0000000 -0.55427987 0.6931286
drat 0.6878267 -0.7015970 -0.9458881 -0.7434288 1.00000000 -0.7502746
wt -0.9174405 0.7823903 0.9678176 0.9187065 -0.73035365 1.0000000
qsec 0.7512435 -0.5908602 -0.6090980 -0.7539446 0.02921025 -0.1878419
vs 0.6942734 -0.8141714 -0.7820383 -0.8418120 0.51150265 -0.6366761
am 0.6031004 -0.5224062 -0.6715676 -0.5603310 0.98069420 -0.8383474
gear 0.4920451 -0.4957241 -0.6123745 -0.5830729 0.87635616 -0.6124648
carb -0.5572555 0.5681535 0.6718953 0.8553318 -0.84876146 0.5966302
qsec vs am gear carb
mpg 0.7383513 0.6640279 0.5997807 0.6529907 -0.65305643
cyl -0.8496465 -0.8108103 -0.5225462 -0.7274627 0.67555239
disp -0.7613347 -0.7104124 -0.5912007 -0.7666091 0.50701342
hp -0.9267578 -0.7230940 -0.2416089 -0.6629877 0.78661538
drat 0.5493953 0.4402134 0.7127009 0.8316282 -0.02225962
wt -0.7718667 -0.5548968 -0.6924849 -0.6575598 0.59412177
qsec 1.0000000 0.7445309 -0.2284076 -0.6333628 -0.66101228
vs 0.9505094 1.0000000 0.1649016 0.6176113 -0.69219670
am -0.6116860 0.1665178 1.0000000 0.8089357 0.34500941
gear -0.6150966 0.2052627 0.7940532 1.0000000 0.55254965
carb -0.7746992 -0.5695923 0.0169436 0.4324314 1.00000000

Comment written by Antoine Soetewey on February 25, 2020 15:34:34:

Thanks for pointing this out Rick, I'll definitely have a look at your package !

[AntoineSoetewey]

Comment written by Bernardo Lares on March 04, 2020 19:57:37:

Hi!

With lares::corr_var() and lares::corr_cross() you can actually run correlations for a whole data.frame with numerical, logical, categorical, date variables... Might complement your function. Check it out on Github: laresbernardo/lares

Cheers!

[AntoineSoetewey]

Comment written by Bernardo Lares on March 04, 2020 19:57:37:

Hi!

With lares::corr_var() and lares::corr_cross() you can actually run correlations for a whole data.frame with numerical, logical, categorical, date variables... Might complement your function. Check it out on Github: laresbernardo/lares

Cheers!

Comment written by Antoine Soetewey on March 06, 2020 10:04:46:

Hi Bernardo,

Thanks a lot for your input and for this nice package. I have added a section explaining the corr_cross() and corr_var() functions. Feel free to let me know if there is any inconsistency.

Regards,
Antoine

[AntoineSoetewey]

Comment written by Antoine Soetewey on March 06, 2020 10:04:46:

Hi Bernardo,

Thanks a lot for your input and for this nice package. I have added a section explaining the corr_cross() and corr_var() functions. Feel free to let me know if there is any inconsistency.

Regards,
Antoine

Comment written by Bernardo Lares on March 06, 2020 13:08:35:

Awesome, thanks for sharing. I'd replicate that example but with a dataset that contains categorical values so the value of the function can be fully understood. You can use data(dft) when lares is loaded to get the Titanic dataset.

Here is another post that will complement your information:

• https://datascienceplus.com/find-insights-with-ranked-cross-correlations/

Cheers and glad to know lares is already in two of your posts! :)

[AntoineSoetewey]

Comment written by Bernardo Lares on March 06, 2020 13:08:35:

Awesome, thanks for sharing. I'd replicate that example but with a dataset that contains categorical values so the value of the function can be fully understood. You can use data(dft) when lares is loaded to get the Titanic dataset.

Here is another post that will complement your information:

• https://datascienceplus.com/find-insights-with-ranked-cross-correlations/

Cheers and glad to know lares is already in two of your posts! :)

Comment written by Antoine Soetewey on April 12, 2020 07:25:39:

Sorry for my late reply. Like your previous comment, it was categorised as spam due to the links, I'll make sure to check my spam more often.

I tend to use the same dataset for the entire article to make it simple, and as the first part is only possible with quantitative variables, I prefer to use only quantitative variables in the illustration of your functions.

I'll make sure to include categorical variables if I use this code again in a future article. I have included a link to your article if readers want more information about your package.

Regards,
Antoine

[AntoineSoetewey]

Comment written by Robert C Cline, Sr. on August 16, 2020 16:02:08:

Your function corrplot2 runs for me in RMarkdown.  I tried to save the corrplot2 function in a snippet.  There are no variables to add in the snippet, so it is your function verbatim.  When I call the snippet to R script, it first requires the psych package.  After loading the psych package, when I run the function, it loads with the following comment:

" bytecode: 0x000002974bb3dc48"

I then run the function after assigning

dat <- mtcars     corrplot2(       data = dat,       method = "pearson",       sig.level = 0.05,       order = "original",       diag = FALSE,       type = "upper",       tl.srt = 75     )

The function fails with the following message:  
Error in cor.mtest(data, method = method) : object 'tmp.value' not found

Do you know how to resolve this? 

Thank you.
Robert

[AntoineSoetewey]

Comment written by Robert C Cline, Sr. on August 16, 2020 16:02:08:

Your function corrplot2 runs for me in RMarkdown.  I tried to save the corrplot2 function in a snippet.  There are no variables to add in the snippet, so it is your function verbatim.  When I call the snippet to R script, it first requires the psych package.  After loading the psych package, when I run the function, it loads with the following comment:

" bytecode: 0x000002974bb3dc48"

I then run the function after assigning

dat <- mtcars corrplot2(   data = dat,   method = "pearson",   sig.level = 0.05,   order = "original",   diag = FALSE,   type = "upper",   tl.srt = 75 )

The function fails with the following message:  
Error in cor.mtest(data, method = method) : object 'tmp.value' not found

Do you know how to resolve this? 

Thank you.
Robert

Comment written by Antoine Soetewey on August 17, 2020 06:44:23:

Dear Robert,

I tried your code on my machine, and it works perfectly. See the image below for the code and the result:

Hope this helps.

Regards,
Antoine

[Piresmoo]

Dear Antoine
Thank you for your page on Correlogram in R. I intend to do a plot using method = "spearman", use = "pairwise.complete.obs". I the scrip seems to do use = "complete.obs", and I don’t find a way to change it. I have empty cells here and there, and I am looking for a good graphical representation that produces the same corr coefficient as “proc corr spearman” of SAS (which is the case with method = "spearman", use = "pairwise.complete.obs").

I also have been looking for ways to produce a plot with an output similar to that obtained with VAR and WITH commands in SAS PROC CORR . I found this website below but I can’t produce a matrix with P-values that match the correlation values. This would allow to omit the nonsignificant correlations
(https://www.statmethods.net/stats/correlations.html)

# Correlation matrix from mtcars
# with mpg, cyl, and disp as rows
# and hp, drat, and wt as columns
x <- mtcars[1:3]
y <- mtcars[4:6]
cor(x, y)

Thanks for your help.
Sincerely
Piresmoo

[AntoineSoetewey]

Dear Antoine
Thank you for your page on Correlogram in R. I intend to do a plot using method = "spearman", use = "pairwise.complete.obs". I the scrip seems to do use = "complete.obs", and I don’t find a way to change it. I have empty cells here and there, and I am looking for a good graphical representation that produces the same corr coefficient as “proc corr spearman” of SAS (which is the case with method = "spearman", use = "pairwise.complete.obs").

I also have been looking for ways to produce a plot with an output similar to that obtained with VAR and WITH commands in SAS PROC CORR . I found this website below but I can’t produce a matrix with P-values that match the correlation values. This would allow to omit the nonsignificant correlations
(https://www.statmethods.net/stats/correlations.html)

# Correlation matrix from mtcars
# with mpg, cyl, and disp as rows
# and hp, drat, and wt as columns
x <- mtcars[1:3]
y <- mtcars[4:6]
cor(x, y)

Thanks for your help.
Sincerely
Piresmoo

Thanks for your question.

I don't use SAS so I cannot help you on this matter, but in lines 15-16 of the code, you should be able to change the use from "everything" (which is the default) to "pairwise.complete.obs".

Let me know if this fixes your issue.

Regards,
Antoine

[Roicarba]

Hello,
I am found very interesting this post. Now, I am working with a dataframe with 40 variables but I want to check the correlation of five of them (columns 22, 25, 26, 28) with only the first 14 variables. But I don´´t get exclude columns without interest on my dataframe.
I have tried things like this
corr_var(correlation[, c(1:14,22)],# group of columns, but the vector don´t work
bio1 # name of variable to focus on
top = 10 # display top 10 correlations
)
Can you help me. I know that i could create a dataframe for anyone of the columns wich I am interesting, but to check more variables, will be faster to get a code to exclude or include some of them
Thanks, so much.

[AntoineSoetewey]

Hello,

Just to make sure I understand correctly:

• you are interested in the correlation between variables:
  • 22 and 1:14
  • 25 and 1:14
  • 26 and 1:14
  • 28 and 1:14
• or you are interested in all possible correlations between variables 1:14, 22, 25, 26 and 28?

Regards,
Antoine

[Roicarba]

Hello Antoine, I am interested on correlation. - 22 and 1:14 - 25 and 1:14 - 26 and 1:14 - 28 and 1:14 Thanks, Roi.

[AntoineSoetewey]

Ok I see.

Have you tried the following?

# variable22 against all others:
corr_var(dat[, c(1:14, 22)],
 variable22,
  top = 14
)

# variable25 against all others:
corr_var(dat[, c(1:14, 25)],
 variable25,
  top = 14
)

# variable26 against all others:
corr_var(dat[, c(1:14, 26)],
 variable26,
  top = 14
)

# variable28 against all others:
corr_var(dat[, c(1:14, 28)],
 variable28,
  top = 14
)

You will need to change:

• dat by the name of your dataset and
• variable22, variable25, variable26 and variable28 by the name of the corresponding variables.

See also this section in case you need to compute correlation coefficients and perform correlation tests for all possible pairs of variables.

Hope this helps.

Regards,
Antoine

[Roicarba]

Many thanks Antoine. It is working now, I think that I have a problem or conflict with libraries, because I had tried to order the data in the same way, but the answer was that it could not read the vector.
Thanks again for your help, Roi.

[AntoineSoetewey]

Glad you fixed it Roi!

[aarontimo]

Hi @antoine, thank you (again!) for your very practical blog posts and clear descriptions. The corr_cross and corr_var are exactly what I've been looking for.

One question, when I run corr_var to focus on a variable like age group (25-34, 45-54, 55+) or income bracket ($18,201-$45,000, $45,001-$60,000 etc) the package provides the following warning message (an example of income bracket)

Warning message:
In corr_var(invest_df, G1Q00003, top = 5) :
  Not a valid input: G1Q00003 was transformed or does not exist.
  >> Automatically using 'G1Q00003_$0 - $18,200'

My question is how do I get the results for $45,001-$60,000? I have tried

corr_var(invest_df, # name of dataset
         'G1Q00003_$45,001 - $60,000', # name of variable to focus on == age
         top = 5 # display top 5 correlations
)

but I get the following error message

Error in corr_var(invest_df, "G1Q00003_$45,001 - $60,000", top = 5) : 
  Not a valid input: "G1Q00003_$45,001 - $60,000" was transformed or does not exist.

I've also tried with double quote marks like this but no success

corr_var(invest_df, # name of dataset
         "G1Q00003_$45,001 - $60,000", # name of variable to focus on == age
         top = 5 # display top 5 correlations
)

Any help or advice would be much appreciated.
Cheers

[AntoineSoetewey]

Dear @aarontimo,

Is your dataset confidential? If not, can you send it to me so I can have a look? Because it’s hard to fully understand the issue without seeing the data. You can also send me your code.

If it’s not confidential, you’ll see my email at www.antoinesoetewey.com/contact

Regards,
Antoine

[aarontimo]

Thank you @antoine - that is very generous of you! Email sent :)

[AntoineSoetewey]

Dear @aarontimo,

I just had a look at your data and code.

The issue comes from the way you labeled your variable G1Q00003 (it includes spaces, $ and -).

Try with the following labels=c("0_18200","18201_45000","45001_60000","60001_75000","75001_90000","90001_105000","105001_130000","130001_155000","155001_180000","180001_205000","205001") and then

corr_var(invest_df,
         G1Q00003_45001_60000,
         top = 5)

for the income bracket $45,0001 - $60,000.

You can apply the same principle for other income brackets.

Hope this helps.

Regards,
Antoine

[aarontimo]

Thank you @antoine - that's fixed it! Much appreciated :)

[jorlandoDPT]

Hi, this is great. How can I modify the size of the text in the code you wrote? Modifying the number size was clear but I want to make the data names smaller. Thanks!

[AntoineSoetewey]

Dear @jorlandoDPT,

I updated the code (see the gist or directly in the post) to give the possibility to choose the size of the text. This can be specified with the additional argument tl.cex in the function.

The default text size is 1, to decrease the size of the text use a value for tl.cex < 1, here for instance with 0.7:

corrplot2(
  data = dat,
  method = "pearson",
  sig.level = 0.05,
  order = "original",
  diag = FALSE,
  type = "upper",
  tl.srt = 75,
  tl.cex = 0.7 # set text size
)

Hope this helps.

Regards,
Antoine

[jorlandoDPT]

Thank you so much! I really appreciate your quick response and this post in general!

[AntoineSoetewey]

You’re welcome!