Run memray on MacOS

[KnightArthurRen]

Hi guys, I'm just wondering does memray support MacOS at the moment? The git readme says it runs on MacOS but the documentation says it doesn't, I tried to install it on MacOS Monterey and pip errored out saying memray only works on linux environment.

Thanks!

[pablogsal]

We just finished adding MacOS support (in the main branch) in these PRs:

#174
#179

but have not yet released the version that does it, so the version you are getting from PyPi still doesn't have support. The reason the Readme says that is because we updated it in the PR where we added MacOS support instead of waiting for the release to do it. Apologies for the confusion :)

Stay tuned for the next release! 👍

[KnightArthurRen]

oh awesome thank you!

[KnightArthurRen]

Hey @pablogsal sorry just a quick follow up, do you know when the next release is going to be? Thank!

[pablogsal]

Probably this week but I am not sure still. Are you in a hurry?

Meanwhile, you could use it in a Linux docker container if you want (native mode is going to be more reliable there in any case).

[KnightArthurRen]

oh no not a hard line, I just want to get an understanding of how much time I want spend to make the project I'm working on cross platform, thanks for the update again!

[kapilt]

I tried compiling from trunk with an m1 Mac, I seem to be seeing issues getting the compilation to work correctly.

using the readme compilation step

 python3 -m pip install -e . -r requirements-test.txt -r requirements-extra.txt

results in

                                                          _Nullable                                       
      /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/signal.h:106:48: note: insert '_Nonnull' if the pointer should never be null
      int     sigvec(int, struct sigvec *, struct sigvec *);                                              
                                                         ^                                                
                                                          _Nonnull                                        
      In file included from src/memray/_memray.cpp:845:                                                   
      In file included from src/memray/_memray/source.h:9:                                                
      src/memray/_memray/lz4_stream.h:35:10: fatal error: 'lz4frame.h' file not found                     
      #include <lz4frame.h>                                                                               
               ^~~~~~~~~~~~                                                                               
      219 warnings and 1 error generated.                                                                                                                                                                             
      Traceback (most recent call last):                                                                                                                                                                              
        File "/private/var/folders/z3/8p9wm6ds2815k1mvx_k3f7740000gn/T/pip-build-env-6js_m_a3/overlay/lib/python3.10/site-packages/setuptools/_distutils/unixccompiler.py", line 186, in _compile
          self.spawn(compiler_so + cc_args + [src, '-o', obj] + extra_postargs)                                                                                                                                       
        File "/private/var/folders/z3/8p9wm6ds2815k1mvx_k3f7740000gn/T/pip-build-env-6js_m_a3/overlay/lib/python3.10/site-packages/setuptools/_distutils/ccompiler.py", line 1006, in spawn
          spawn(cmd, dry_run=self.dry_run, **kwargs)                                                      
        File "/private/var/folders/z3/8p9wm6ds2815k1mvx_k3f7740000gn/T/pip-build-env-6js_m_a3/overlay/lib/python3.10/site-packages/setuptools/_distutils/spawn.py", line 70, in spawn
          raise DistutilsExecError(                                                                       
      distutils.errors.DistutilsExecError: command '/usr/bin/clang' failed with exit code 1               

it seems to be complaining about lz4 not being present but I do have it installed via brew

❯ brew search lz4
==> Formulae
lz4 ✔    

python info (via pyenv)

>>> sys.version
'3.10.0 (default, Nov 23 2021, 19:58:18) [Clang 13.0.0 (clang-1300.0.29.3)]'

[kapilt]

actually the cflags export isn't in the readme and is required for osx if using brew, so should be included in the building from source section imo.

[pablogsal]

actually the cflags export isn't in the readme and is required for osx if using brew, so should be included in the building from source section imo.

This is what I referred with:

We don't document how to do this because this is not a memray problem, but a package manager problem. You need to check with your package manager (brew in this case) how to make the libraries available at build time.

There are ways other than brew that correctly place the libraries in places that can be seen by default, so this is not a memray problem but a package manager problem.

In any case, once we get wheels, you don;t need to compile it or install anything with brew if you just pip install it. You just need to wait for the release 😉

[kapilt]

noted re packages, but its the build from source documentation.. there's already a reference to brew in the readme on installing lz4, so giving incomplete instructions for it feels odd.

[godlygeek]

Seems like it might be worth another sentence in the docs, since we already mention Brew. Something like:

Note that the compiler and linker won't know where to find Brew libraries by default, so you'll need to export CFLAGS="-I$(brew --prefix lz4)/include" LDFLAGS="-L$(brew --prefix lz4)/lib -Wl,-rpath,$(brew --prefix lz4)/lib" before building.

I'll make that change if someone with a Mac can confirm that works for me 😄

[godlygeek]

We've updated the README to include that incantation.

[pablogsal]

Btw @kapilt I saw you opened #187, bear in mind that in MacOS the default multiprocessing method is not fork but spawn so memray will not be able to follow processes unless you call:

multiprocessing.set_start_method('fork')

at the beginning of your code. Notice also that this can have some problems because is considered unsafe:

python/cpython#77906

[kapilt]

i was on linux for the issue filed re multiprocessing, but good to know. ie. the individual recordings are fine, but aggregating reports across multiprocessing requires a machine readable structured output.