Linear models failing to display with ambiguous datatypes with unhelpful error messages

[jakewilliami]

The linear model is failing somewhere in show (I guess) if you give it DataFrames with type Any. MWE:

julia> df_test = DataFrame(date_numeric = [737810, 737841, 737869, 737900, 737930, 737961, 737991, 738022, 738053, 738083, 738114], num = Any[-74587.93, -74550.49, -74482.09, -74441.45, -74316.17, -74252.81, -73976.21, -73587.65, -73170.53, -72753.41, -72304.01])
11×2 DataFrame
 Row │ date_numeric  num
     │ Int64         Any
─────┼────────────────────────────
   1 │       737810  -74587.9
   2 │       737841  -74550.5
   3 │       737869  -74482.1
   4 │       737900  -74441.4
   5 │       737930  -74316.2
   6 │       737961  -74252.8
   7 │       737991  -73976.2
   8 │       738022  -73587.6
   9 │       738053  -73170.5
  10 │       738083  -72753.4
  11 │       738114  -72304.0

julia> model = fit(LinearModel, @formula(date_numeric ~ num), df_test)
ERROR: ArgumentError: FDist: the condition ν1 > zero(ν1) && ν2 > zero(ν2) is not satisfied.
Stacktrace:
  # ...

julia> df_test = DataFrame(date_numeric = [737810, 737841, 737869, 737900, 737930, 737961, 737991, 738022, 738053, 738083, 738114], num = Float64[-74587.93, -74550.49, -74482.09, -74441.45, -74316.17, -74252.81, -73976.21, -73587.65, -73170.53, -72753.41, -72304.01]);

julia> model = fit(LinearModel, @formula(date_numeric ~ num), df_test)
StatsModels.TableRegressionModel{LinearModel{GLM.LmResp{Vector{Float64}}, GLM.DensePredChol{Float64, LinearAlgebra.CholeskyPivoted
{Float64, Matrix{Float64}}}}, Matrix{Float64}}

date_numeric ~ 1 + num

Coefficients:
─────────────────────────────────────────────────────────────────────────────────────
                      Coef.    Std. Error       t  Pr(>|t|)  Lower 95%      Upper 95%
─────────────────────────────────────────────────────────────────────────────────────
(Intercept)   746741.0       1103.35       676.79    <1e-21  7.44245e5  749237.0
num                0.118876     0.0149383    7.96    <1e-04  0.0850827       0.152668
─────────────────────────────────────────────────────────────────────────────────────

Took me quite a while to figure out what exactly was happening here (realised something weird was happening when running model = fit(LinearModel, @formula(date_numeric ~ num), df_test); in the REPL worked, but didn't work when I didn't suppress the display (i.e., when I removed the semicolon).

[nalimilan]

The problem is that variables with eltype Any are treated as categorical. You get the same error by wrapping num in a CategoricalArray. So the underlying problem is that coeftable fails for a model with zero residual degrees of freedom. See #458.

[floswald]

I agree with the OP that the error message could be improved here. Here is another case, which I just found when reading data from an Excel sheet with XLSX.readtable (it parses everything as Any by default).

julia> d = DataFrame(Y = Any[1,2,3], X = Any[3,2,1])
3×2 DataFrame
 Row │ Y    X   
     │ Any  Any 
─────┼──────────
   1 │ 1    3
   2 │ 2    2
   3 │ 3    1

julia> lm(@formula(Y ~ X), d)
ERROR: MethodError: no method matching fit(::Type{LinearModel}, ::Matrix{Float64}, ::Matrix{Float64}, ::Nothing)
Closest candidates are:
  fit(::StatisticalModel, ::Any...) at /Users/74097/.julia/packages/StatsBase/IPydo/src/statmodels.jl:178
  fit(::Type{StatsBase.Histogram}, ::Any...; kwargs...) at /Users/74097/.julia/packages/StatsBase/IPydo/src/hist.jl:383
  fit(::Type{LinearModel}, ::AbstractMatrix{var"#s49"} where var"#s49"<:Real, ::AbstractVector{var"#s50"} where var"#s50"<:Real, ::Union{Nothing, Bool}; wts, dropcollinear) at /Users/74097/.julia/packages/GLM/hDWc9/src/lm.jl:156

can we not check for the element type of each column and show a warning if we find Any?