Unable to pip install 3.2.4 on aarch64

[xconverge]

Specifications

• OS: aarch64
• SCS Version: 3.2.4
• Compiler:

Description

I know the issue template above says A common cause of issues is not linking BLAS/LAPACK libraries correctly. If you are having this issue please search for resources on installing and linking these libraries first. You can try openblas if you need a BLAS library.

but did something change between 3.2.3 and 3.2.4 that would relate to this?

I am now getting the following error and have reverted back to 3.2.3 with no such issues

 Collecting scs==3.2.4 (from -r requirements.txt (line 72))
   Downloading scs-3.2.4.tar.gz (1.5 MB)
      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.5/1.5 MB 32.1 MB/s eta 0:00:00
   Installing build dependencies: started
   Installing build dependencies: finished with status 'done'
   Getting requirements to build wheel: started
   Getting requirements to build wheel: finished with status 'done'
   Installing backend dependencies: started
   Installing backend dependencies: finished with status 'done'
   Preparing metadata (pyproject.toml): started
   Preparing metadata (pyproject.toml): finished with status 'error'
   error: subprocess-exited-with-error

   × Preparing metadata (pyproject.toml) did not run successfully.
   │ exit code: 1
   ╰─> [20 lines of output]
       + meson setup /tmp/pip-install-pmn6aiyt/scs_a40b4b98120348a6b05b5b26238178d0 /tmp/pip-inst>
       The Meson build system
       Version: 1.2.3
       Source dir: /tmp/pip-install-pmn6aiyt/scs_a40b4b98120348a6b05b5b26238178d0
       Build dir: /tmp/pip-install-pmn6aiyt/scs_a40b4b98120348a6b05b5b26238178d0/.mesonpy-g_hrwcg1
       Build type: native build
       Project name: scs
       Project version: undefined
       C compiler for the host machine: cc (gcc 11.4.0 "cc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0")
       C linker for the host machine: cc ld.bfd 2.38
       Host machine cpu family: aarch64
       Host machine cpu: aarch64
       Program python found: YES (/home/ubuntu/src/algo-api/venv/bin/python3)
       Library openblas found: NO
       Library blas found: NO
       Library cblas found: NO

       ../meson.build:33:4: ERROR: Problem encountered: OpenBLAS or Netlib BLAS/CBLAS is required>

       A full log can be found at /tmp/pip-install-pmn6aiyt/scs_a40b4b98120348a6b05b5b26238178d0/>
       [end of output]

   note: This error originates from a subprocess, and is likely not a problem with pip.
 error: metadata-generation-failed
 × Encountered error while generating package metadata.
 ╰─> See above for output.
 note: This is an issue with the package mentioned above, not pip.
 hint: See above for details.

I only have the problem on aarch64 and not my amd64 system. Both are running linux.

If I do:

sudo apt-get install libopenblas-dev

it resolves it, but this was not necessary for me before, so I am trying to understand what changed? Manually installing this dependency is not an option for me long term at the moment so I will be staying on 3.2.3 which works fine

[bodono]

Hi @xconverge, yes SCS 3.2.4 migrated (mostly) to the meson build system, since python 3.12 no longer supports the old setuptools way to install that SCS was using. For aarch64 I think SCS was linking against a very slow blas, so if you link against openblas it might actually be faster.

Is this using pip? I don't think we can upload wheels for aarch64 from github.

Previously were you running without blas? Or was it able to find a different blas library on your system somehow?

Paging @enzbus who might have a better understanding.

[enzbus]

Hello, everything is correct there. You need to have blas installed to compile scs from source. We are distributing compiled wheels (in which we link blas statically) but not for aarch64 (if you know how to set a github CI runner for that architecture, please submit a PR in bodono/scs-python). In any case if you compile from source locally you need to have blas installed on your machine.

For the last release we rewrote the python compilation code, because 3.12 removed the old distutils library and there was no way to make the old code work. What happened before was that pip install scs attempted to link against the blas from numpy, so maybe it was possible that you could install it even if you didn't have blas available system wide. (Just my guess, it was very hard to debug.)

[enzbus]

Actually, @xconverge, could you post the output of numpy.show_config() from your aarch64 linux box when you don't have blas installed system-wide? Maybe we can add a clause to the meson.build to make it work. Also please if you want to contribute on this head over to scs-python.

[xconverge]

Here is the output from my working system on 3.2.3

>>> numpy.show_config()
openblas64__info:
    libraries = ['openblas64_', 'openblas64_']
    library_dirs = ['/usr/local/lib']
    language = c
    define_macros = [('HAVE_CBLAS', None), ('BLAS_SYMBOL_SUFFIX', '64_'), ('HAVE_BLAS_ILP64', None)]
    runtime_library_dirs = ['/usr/local/lib']
blas_ilp64_opt_info:
    libraries = ['openblas64_', 'openblas64_']
    library_dirs = ['/usr/local/lib']
    language = c
    define_macros = [('HAVE_CBLAS', None), ('BLAS_SYMBOL_SUFFIX', '64_'), ('HAVE_BLAS_ILP64', None)]
    runtime_library_dirs = ['/usr/local/lib']
openblas64__lapack_info:
    libraries = ['openblas64_', 'openblas64_']
    library_dirs = ['/usr/local/lib']
    language = c
    define_macros = [('HAVE_CBLAS', None), ('BLAS_SYMBOL_SUFFIX', '64_'), ('HAVE_BLAS_ILP64', None), ('HAVE_LAPACKE', None)]
    runtime_library_dirs = ['/usr/local/lib']
lapack_ilp64_opt_info:
    libraries = ['openblas64_', 'openblas64_']
    library_dirs = ['/usr/local/lib']
    language = c
    define_macros = [('HAVE_CBLAS', None), ('BLAS_SYMBOL_SUFFIX', '64_'), ('HAVE_BLAS_ILP64', None), ('HAVE_LAPACKE', None)]
    runtime_library_dirs = ['/usr/local/lib']
Supported SIMD extensions in this NumPy install:
    baseline = NEON,NEON_FP16,NEON_VFPV4,ASIMD
    found = ASIMDHP,ASIMDDP
    not found = ASIMDFHM

And here is me running an example from the docs with this version/system

# cat test.py
import scipy
import scs
import numpy as np

# Set up the problem data
P = scipy.sparse.csc_matrix([[3.0, -1.0], [-1.0, 2.0]])
A = scipy.sparse.csc_matrix([[-1.0, 1.0], [1.0, 0.0], [0.0, 1.0]])
b = np.array([-1, 0.3, -0.5])
c = np.array([-1.0, -1.0])

# Populate dicts with data to pass into SCS
data = dict(P=P, A=A, b=b, c=c)
cone = dict(z=1, l=2)

# Initialize solver
solver = scs.SCS(data, cone, eps_abs=1e-9, eps_rel=1e-9)
# Solve!
sol = solver.solve()

print(f"SCS took {sol['info']['iter']} iters")
print("Optimal solution vector x*:")
print(sol["x"])

print("Optimal dual vector y*:")
print(sol["y"])


# python test.py
------------------------------------------------------------------
               SCS v3.2.3 - Splitting Conic Solver
        (c) Brendan O'Donoghue, Stanford University, 2012
------------------------------------------------------------------
problem:  variables n: 2, constraints m: 3
cones:    z: primal zero / dual free vars: 1
          l: linear vars: 2
settings: eps_abs: 1.0e-09, eps_rel: 1.0e-09, eps_infeas: 1.0e-07
          alpha: 1.50, scale: 1.00e-01, adaptive_scale: 1
          max_iters: 100000, normalize: 1, rho_x: 1.00e-06
          acceleration_lookback: 0, acceleration_interval: 0
lin-sys:  sparse-direct-amd-qdldl
          nnz(A): 4, nnz(P): 3
WARN: aa_init returned NULL, no acceleration applied.
------------------------------------------------------------------
 iter | pri res | dua res |   gap   |   obj   |  scale  | time (s)
------------------------------------------------------------------
     0| 1.51e+00  1.00e+00  4.91e+00  1.14e+00  1.00e-01  2.25e-04
   100| 4.39e-12  1.50e-13  9.31e-12  1.23e+00  1.00e-01  2.73e-04
------------------------------------------------------------------
status:  solved
timings: total: 2.83e-04s = setup: 1.56e-04s + solve: 1.27e-04s
         lin-sys: 1.33e-05s, cones: 7.24e-06s, accel: 0.00e+00s
------------------------------------------------------------------
objective = 1.235000
------------------------------------------------------------------
SCS took 100 iters
Optimal solution vector x*:
[ 0.3 -0.7]
Optimal dual vector y*:
[2.7 2.1 0. ]

Sorry I didn't see that other repo for the python interface, oops!

[enzbus]

It doesn't look good, numpy is pointing to the system directory for blas, I know they ship an openblas they compile independently but it doesn't show there. I don't think we'll be able to reproduce the way scs was built with the old system (which in any case won't work on Python >= 3.12) for your particular usecase, sorry. Just out of curiosity, what's the output of ldd on the _scs*.so in the site-packages folder of your Python environment?

[xconverge]

ldd venv/lib/python3.10/site-packages/_scs*.so
venv/lib/python3.10/site-packages/_scs_direct.cpython-310-aarch64-linux-gnu.so:
        linux-vdso.so.1 (0x0000ffffb2fe7000)
        libc.so.6 => /lib/aarch64-linux-gnu/libc.so.6 (0x0000ffffb2dd0000)
        /lib/ld-linux-aarch64.so.1 (0x0000ffffb2fae000)
venv/lib/python3.10/site-packages/_scs_indirect.cpython-310-aarch64-linux-gnu.so:
        linux-vdso.so.1 (0x0000ffff855d3000)
        libc.so.6 => /lib/aarch64-linux-gnu/libc.so.6 (0x0000ffff853c0000)
        /lib/ld-linux-aarch64.so.1 (0x0000ffff8559a000)

[enzbus]

It's a mystery, I don't see any link to blas there. @bodono, any ideas?

[enzbus]

Have you tried to solve a problem with PSD cones? scs will give you an error if it's not linked to blas. Not sure if it's a good idea, but we could allow builds without blas, which give you only a partially functional scs (which now I suspect is what is going on here).

[enzbus]

I made a commit in my branch that turns the error you got into a warning. I'm not able to test it because if I remove blas from my linux box the whole graphics environment gets removed. Can you test it @xconverge by

pip install git+https://github.com/enzbus/scs-python

? You are right, with the old setup.py-based compilation scs got compiled without blas by giving a warning to the user. @bodono if that's good you can merge it in 3.2.5.

[xconverge]

I assume the solver will give an exception or fail if it tries to do something utilizing blas, so I assume what I am running works for my problem and use case

I will give it a try in a bit, and also am considering other solvers for my use case

[xconverge]

With git+https://github.com/enzbus/scs-python I get this, reverting back to 3.2.3 again after runs fine:

(CVXPY) Nov 09 08:26:10 PM: Encountered unexpected exception importing solver SCS:
ImportError('/xxxxxx/venv/lib/python3.10/site-packages/_scs_direct.cpython-310-aarch64-linux-gnu.so: undefined symbol: ddot_')

[enzbus]

Ok I think I know what's going on, there's a compilation flag that probably needs to be unset if blas is not there. Try now:
pip install git+https://github.com/enzbus/scs-python

[bodono]

Ok I think I know what's going on, there's a compilation flag that probably needs to be unset if blas is not there. Try now: pip install git+https://github.com/enzbus/scs-python

Yes, it's USE_LAPACK (see here).

[bodono]

It looks to me like you were previously running without blas/lapack installed, which means you can't solve SDPs or make use of Anderson acceleration (hence the message WARN: aa_init returned NULL, no acceleration applied.). That is a valid use case, so I think we should support being able to not link against blas/lapack if desired, although in practice I think most users will have a blas install and acceleration does tend to improve the solver performance, sometimes quite substantially. Is linking against a blas install a major blocker for you @xconverge ?

[enzbus]

@bodono I managed to run the code to compile without blas and it breaks, I need to go back in our commit history to find how it worked before (at some point we had a meson-compiled scs without blas that was still able to solve SOCPs, which I think is what @xconverge wants).

[xconverge]

Ok I think I know what's going on, there's a compilation flag that probably needs to be unset if blas is not there. Try now: pip install git+https://github.com/enzbus/scs-python

I tried again with the latest commit (aee3f12b0dea2dd6478beaa4f626878b93f60ee8) with the same issue

Regarding your questions @bodono, I am exploring my options (use a different solver that doesn't need blas, installing blas to link against, a resolution here where no blas is a warning but it works fine for my use case still, etc)

I already run cvxpy-base and added scs solver (primarily due to it was what I was already using in cvxpy) independently because of licensing issues with some of the other solvers (ecos, they seem to be changing their licensing but it isn't active yet) so I am pretty adaptable to whatever changes make sense for the project as a whole 🤷

[enzbus]

Another attempt you can make, @xconverge, is to clone the scs-python repository, git submodule --init and then python legacy_setup.py install, from the environment in which you want scs installed. You need setuptools, numpy, and scipy. This is broken on 3.12 but if you're lucky it will work on earlier versions. (It is not supported any more.)

[xconverge]

I think I will go the route of installing libopenblas-dev as part of my deployment/usage

I would like to stay in the realm of what works best for you all too and isnt prone to breakage

I also want to be sure I am running things the way they should for performance reasons/etc.

[bodono]

Ok great, let us know if you have any problems with it. It should be a little faster too.