any example showing how to use the wrapper?

[ttsesm]

@sampotter I would like to use the embree lib for raycasting since it provides a nice accelerated solution. However, I am not quite sure how to use your wrapper. Would be possible to write a small example showing some use case. For example considering that someone has the indices and faces of a mesh object in numpy arrays how to create the corresponding mesh in embree by using this wrapper?

[sampotter]

This wrapper attempts to mimic the Embree C API closely. So, if you want to figure out how to do something, my goal is for a user to be able to consult the Embree API first.

That said, this wrapper is in a prototype stage and hasn't been used or tested thoroughly yet.

Here's some code that I'm using currently for raytracing, somewhat out of context:

device = embree.Device()
geometry = device.make_geometry(embree.GeometryType.Triangle)
scene = device.make_scene()

vertex_buffer = geometry.set_new_buffer(
    embree.BufferType.Vertex, # buf_type
    0, # slot
    embree.Format.Float3, # fmt
    3*np.dtype('float32').itemsize, # byte_stride
    V.shape[0], # item_count
)
vertex_buffer[:] = V[:]

index_buffer = geometry.set_new_buffer(
    embree.BufferType.Index, # buf_type
    0, # slot
    embree.Format.Uint3, # fmt
    3*np.dtype('uint32').itemsize, # byte_stride,
    F.shape[0]
)
index_buffer[:] = F[:]

geometry.commit()
geom_id = scene.attach_geometry(geometry)
geometry.release()

scene.commit()

n0, p0 = N[i0], P[i0]
dP = P - p0

mask = np.logical_and(dP@n0 >= 0, np.sum(dP*N, axis=1) <= 0)
P_mask = P[mask]

M = sum(mask)

rayhit = embree.RayHit1M(M)
rayhit.tnear[:] = 0
rayhit.tfar[:] = np.inf
rayhit.prim_id[:] = embree.INVALID_GEOMETRY_ID
rayhit.geom_id[:] = embree.INVALID_GEOMETRY_ID
rayhit.org[:] = p0 + eps*n0
rayhit.dir[:] = P_mask - p0

context = embree.IntersectContext()

scene.intersect1M(context, rayhit)

H = np.zeros((P.shape[0],), dtype=bool)
I_mask = np.where(mask)[0]
H[I_mask] = rayhit.prim_id == I_mask

In this example, I'm shooting some rays from perturbed triangle centroids to other triangles and seeing which rays actually hit their target triangles. In this case, I've loaded an OBJ using another Python library (doesn't really matter which), and then used Embree's API to tell it about the mesh's vertices and faces.

If you aren't familiar with Embree, if you consult the Embree API, you'll find that the code above matches what is done in Embree's provided examples.

Please let me know if you run into any trouble, and feel free to post code and mesh files here.

Please note: the entire API isn't wrapped yet, but should be very easy to wrap more functions. If a function from the Embree API that you need is missing, please say so. You can check embree.pyx to see if a function has been wrapped.

[ttsesm]

@sampotter thanks for the prompt response and the example. However, just before we go to code examples I would like to clarify the installation procedure. How someone should install the wrapper?

I tried to run pip install --global-option=build_ext --global-option="-IC:\Program Files\Intel\Embree3\include" --global-option="-LC:\Program Files\Intel\Embree3\lib" . from my virtual environment where C:\Program Files\Intel\Embree3 is the path to my Embree3 installation but it failed giving me some errors (I can post the terminal output if needed), thus I guess there is something I am missing.

[sampotter]

No problem.

Can you tell me a little more about your setup? I see that you're using Windows, which should work in principle, but is less familiar to me. Are you using Anaconda?

At this point, I haven't tested this wrapper on Windows, so we'll need to figure out an appropriate workflow for getting it installed.

And yes, please post the build output.

[ttsesm]

Yes, currently I am on a windows machine. I know that usually with this kind of things in linux is more straight forward. However, in principle someone should be able to install the wrapper on both systems.

Anyways back to the point, in my project I am using Pycharm where I have set both a conda and a virtualenv environment, the pip command above I've run it from my virtualenv environment.

As you correctly figured out I am on a windows 10 machine, with Python 3.8, and I have installed Embree v3 by using the windows installer.

This is the output from the build:

>>pip install --global-option=build_ext --global-option="-IC:\Program Files\Intel\Embree3\include" --global-option="-LC:\Program Files\Intel\Embree3\lib" .
c:\users\ttsesm\desktop\dev\radiosity\r_venv\lib\site-packages\pip\_internal\commands\install.py:243: UserWarning: Disabling all use of wheels due to the use of --build-option / --global-option / --install-option.
  cmdoptions.check_install_build_global(options)
Processing c:\users\ttsesm\desktop\dev\python-embree
Skipping wheel build for embree, due to binaries being disabled for it.
Installing collected packages: embree
    Running setup.py install for embree ... error
    ERROR: Command errored out with exit status 1:
     command: 'c:\users\ttsesm\desktop\dev\radiosity\r_venv\scripts\python.exe' -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\TTsesm\\AppData\\Local\\Temp\\pip-req-build-dtrlbuol\\setup.py'"'"'; __file_
_='"'"'C:\\Users\\TTsesm\\AppData\\Local\\Temp\\pip-req-build-dtrlbuol\\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"
'"'exec'"'"'))' build_ext '-IC:\Program Files\Intel\Embree3\include' '-LC:\Program Files\Intel\Embree3\lib' install --record 'C:\Users\TTsesm\AppData\Local\Temp\pip-record-rqreguy3\install-record.txt' --single-version-externally-ma
naged --compile --install-headers 'c:\users\ttsesm\desktop\dev\radiosity\r_venv\include\site\python3.8\embree'
         cwd: C:\Users\TTsesm\AppData\Local\Temp\pip-req-build-dtrlbuol\
    Complete output (14 lines):
    running build_ext
    building 'embree' extension
    creating build
    creating build\temp.win-amd64-3.8
    creating build\temp.win-amd64-3.8\Release
    C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.26.28801\bin\HostX86\x64\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MD -I/opt/local/include "-IC:\Program Files\Intel\Embree3\include" -Ic:\users\ttsesm\desktop\dev\radiosity\r_venv\include -IC:\Users\TTsesm\AppData\Local\Programs\Python\Python38\include -IC:\Users\TTsesm\AppData\Local\Programs\Python\Python38\include "-IC:\Program Files (x86)\Microsoft Visual Studio\2019\Com
munity\VC\Tools\MSVC\14.26.28801\ATLMFC\include" "-IC:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.26.28801\include" "-IC:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\include\um" "-IC:\Program Files (x
86)\Windows Kits\10\include\10.0.18362.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\shared" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\um" "-IC:\Program Files (x86)\Windows Kits\10\include\
10.0.18362.0\winrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\cppwinrt" /Tcembree.c /Fobuild\temp.win-amd64-3.8\Release\embree.obj
    embree.c
    embree.c(9654): warning C4013: 'posix_memalign' undefined; assuming extern returning int
    creating C:\Users\TTsesm\AppData\Local\Temp\pip-req-build-dtrlbuol\build\lib.win-amd64-3.8
    C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.26.28801\bin\HostX86\x64\link.exe /nologo /INCREMENTAL:NO /LTCG /DLL /MANIFEST:EMBED,ID=2 /MANIFESTUAC:NO /LIBPATH:/opt/local/lib "/LIBPATH:C:\Program F
iles\Intel\Embree3\lib" /LIBPATH:c:\users\ttsesm\desktop\dev\radiosity\r_venv\libs /LIBPATH:C:\Users\TTsesm\AppData\Local\Programs\Python\Python38\libs /LIBPATH:C:\Users\TTsesm\AppData\Local\Programs\Python\Python38 /LIBPAT
H:c:\users\ttsesm\desktop\dev\radiosity\r_venv\PCbuild\amd64 "/LIBPATH:C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.26.28801\ATLMFC\lib\x64" "/LIBPATH:C:\Program Files (x86)\Microsoft Visual Studio
\2019\Community\VC\Tools\MSVC\14.26.28801\lib\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\lib\um\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.18362.0\ucrt\x64" "/LIBPATH:C:\Program Files (x86)\Wind
ows Kits\10\lib\10.0.18362.0\um\x64" embree3.lib /EXPORT:PyInit_embree build\temp.win-amd64-3.8\Release\embree.obj /OUT:build\lib.win-amd64-3.8\embree.cp38-win_amd64.pyd /IMPLIB:build\temp.win-amd64-3.8\Release\embree.cp38-win_amd64.li
b
       Creating library build\temp.win-amd64-3.8\Release\embree.cp38-win_amd64.lib and object build\temp.win-amd64-3.8\Release\embree.cp38-win_amd64.exp
    embree.obj : error LNK2001: unresolved external symbol posix_memalign
    build\lib.win-amd64-3.8\embree.cp38-win_amd64.pyd : fatal error LNK1120: 1 unresolved externals
    error: command 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Community\\VC\\Tools\\MSVC\\14.26.28801\\bin\\HostX86\\x64\\link.exe' failed with exit status 1120
    ----------------------------------------
ERROR: Command errored out with exit status 1: 'c:\users\ttsesm\desktop\dev\radiosity\r_venv\scripts\python.exe' -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\TTsesm\\AppData\\Local\\Temp\\pip-req-build
-dtrlbuol\\setup.py'"'"'; __file__='"'"'C:\\Users\\TTsesm\\AppData\\Local\\Temp\\pip-req-build-dtrlbuol\\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close(
);exec(compile(code, __file__, '"'"'exec'"'"'))' build_ext '-IC:\Program Files\Intel\Embree3\include' '-LC:\Program Files\Intel\Embree3\lib' install --record 'C:\Users\TTsesm\AppData\Local\Temp\pip-record-rqreguy3\install-record.tx
t' --single-version-externally-managed --compile --install-headers 'c:\users\ttsesm\desktop\dev\radiosity\r_venv\include\site\python3.8\embree' Check the logs for full command output.

[sampotter]

How did you install Embree?

[ttsesm]

Do you think that it might have to do with the Embree installation which is linked against the Visual Studio 2013 (VC12) and Visual Studio 2015/2017 (VC14) runtime?

I installed it by using the .msi windows installer from here https://www.embree.org/downloads.html

[sampotter]

No, your installation doesn't appear to be a problem.

It can't find posix_memalign, which makes sense. It looks like I can use _aligned_malloc instead on Windows. I'll need to try building this on Windows, which should be pretty easy for me right now.

If you want to try debugging this yourself, look for instances of posix_memalign in embree.pyx and see if you can replace them with calls to _aligned_malloc. It will be necessary to add an aligned malloc that's platform independent, where the appropriate function calls are selected using preprocessor macros.

[ttsesm]

Actually it seems that it is not a direct replacement, since the set of input parameters are different. I found this stackoverflow thread https://stackoverflow.com/questions/33696092/whats-the-correct-replacement-for-posix-memalign-in-windows where they discussing the same issue and the provided solution seems to do the trick.

[sampotter]

That's what I was looking at, as well. :-)

To provide some context, when using Embree, things that are allocated on the heap need to be aligned appropriately for SIMD. This is explained pretty clearly in the Embree API, which I linked to in my first reply.

Regardless of the exact API of the function, what's needed is a malloc that can align allocated memory to the right byte boundary for use with SIMD.

[ttsesm]

@sampotter do you think that you could use the C's malloc, realloc and free or the Python's C-API memory allocation functions instead of the posix one.

Based on this https://tianrunhe.wordpress.com/2012/04/23/aligned-malloc-in-c/ it seems that aligned_malloc can be done with calls to malloc.

Here are some other reference links:

https://cython.readthedocs.io/en/latest/src/tutorial/memory_allocation.html

https://github.com/EveryTimeIWill18/Cython_Repo/blob/master/Cython_Tutorial_Three.ipynb?utm_campaign=News&utm_medium=Community&utm_source=DataCamp.com

https://github.com/linux-rdma/rdma-core/blob/master/pyverbs/mem_alloc.pyx

I couldn't find any way to use calls to _aligned_malloc, which is valid for windows.

I also found the following link which could be usefull:

numpy/numpy#5312

[sampotter]

Hey @ttsesm, I just pushed a commit that should fix this. I switched from posix_memalign to aligned_alloc on POSIX, and _aligned_malloc on Windows, which has the same API. I was able to compile without any trouble on Windows but haven't tested this. Let me know how this works.

[sampotter]

Hmm, thought I had fixed this, but my changed cause some problems on macOS, which I wasn't able to figure out (in the small amount of time I budgeted for it).

I created a new branch off of develop (aligned_allocation) where we can try to figure this out. The latest commit there is working on Mac, but haven't had a chance to try it on Windows or Linux yet.

[ttsesm]

Thanks @sampotter, I've managed to make it compile in Windows (I do not have access to a linux machine currently). I had to make the following change though:

if errno == EINVAL:
     raise Exception('bad alignment')
elif errno == ENOMEM:

in line

python-embree/embree.pyx

Line 19 in 802b5f4

if errno == ENOMEM:

otherwise it was complaining in line

python-embree/embree.pyx

Line 803 in 802b5f4

if errno == EINVAL:

that EINVAL was not declared.

[sampotter]

Thanks for catching that, and glad you were able to get it to compile. If you put together a pull request I'll be happy to accept it.

[ttsesm]

@sampotter I had some time to play a bit with the provided code in one of your first responses. However, while embree module is compiled and installed it doesn't seem that I can import it. Any idea what could be the issue?

I've compiled and installed it running the following command from the python-embree root folder while being in my venv:

pip install --global-option=build_ext --global-option="-IC:\Program Files\Intel\Embree3\include" --global-option="-LC:\Program Files\Intel\Embree3\lib" .

As you can see, I can find it in my site packages in pycharm:

but if I try to import it so that to be able to use it, it doesn't find it.

[sampotter]

@ttsesm Sorry about the delay. Could you be a little more explicit about what's going wrong? It's hard for me to help with so little info.

[sampotter]

@ttsesm Closing this due to lack of activity.

[aluo-x]

I also had trouble importhing this package. I've isolated the issue to pytorch_geometric, importing works as long as I import embree first.

[adamgranthendry]

@sampotter I actually get the error:

ImportError: DLL load failed while importing embree: The specified module could not be found.

Any suggestions on how to fix this?

[sampotter]

Very hard to say without more information. Could you provide more information about your setup, what version of Python you have, whether you're using vanilla Python or Anaconda, what version of Embree itself, how you installed it, etc etc

[adam-grant-hendry]

Hi @sampotter,

Setup:
OS: Windows 10 x64 Professional, Build 1909
Python: 3.8.10 x64 (vanilla)
Numpy: numpy‑1.21.1+mkl‑cp38‑cp38‑win_amd64.whl from Christoph Gohlke's collection
Embree: 3.13.0 x64 (Standard Install Location: C:\Program Files\Intel\Embree3)

Steps to Reproduce:
(I use MSYS2 bash terminal below)

1. Create project folder project
2. Create virtual environment with venv in subfolder .venv (py -m venv .venv) and activate it
3. Install the numpy wheel (py -m pip install numpy‑1.21.1+mkl‑cp38‑cp38‑win_amd64.whl)
4. Install per your instructions:

py setup.py build_ext -I/c/Program\ Files/Intel/Embree3/include -L /c/Program\ Files/Intel/Embree3/lib
py setup.py install

5. Importing embree fails with:

ImportError: DLL load failed while importing embree: The specified module could not be found.

6. The appropriate .egg and .pyd files are created. Looking further at embree.py

C:\project\.venv\Lib\site-packages\embree-0.0.0-py3.8-win-amd64.egg
│   embree.cp38-win_amd64.pyd
│   embree.py
│
├───EGG-INFO
│       dependency_links.txt
│       native_libs.txt
│       not-zip-safe
│       PKG-INFO
│       SOURCES.txt
│       top_level.txt
│
└───__pycache__
        embree.cpython-38.pyc

So I open an ipython terminal and step through the __build__() function:

In [1]: cd C:\project\.venv\Lib\site-packages\embree-0.0.0-py3.8-win-amd64.egg
In [2]: import sys, pkg_resources, importlib.util
In [3]: # In this instance, `__name__` is `__main__`, not `embree`, just fyi
In [4]: __file__ = pkg_resources.resource_filename(__name__, 'embree.cp38-win_amd64.pyd')
In [5]: __file__
Out [5]: 'embree.cp38-win_amd64.pyd'
In [6]: spec = importlib.util.spec_from_file_location(__name__,__file__)
In [7]: spec
Out [7]: ModuleSpec(name='__main__', loader=<_frozen_importlib_external.ExtensionFileLoader object at 0x000002176D5FC7C0>, origin='C:\\project\\.venv\\Lib\\site-packages\\embree-0.0.0-py3.8-win-amd64.egg\\embree.cp38-win_amd64.pyd')
In [8]: mod = importlib.util.module_from_spec(spec)
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
C:\project\.venv\Lib\site-packages\embree-0.0.0-py3.8-win-amd64.egg\embree.cp38-win_amd64.pyd in <module>

C:\Program Files\Python38\lib\importlib\_bootstrap.py in module_from_spec(spec)

C:\Program Files\Python38\lib\importlib\_bootstrap_external.py in create_module(self, spec)

C:\Program Files\Python38\lib\importlib\_bootstrap.py in _call_with_frames_removed(f, *args, **kwds)

ImportError: DLL load failed while importing __main__: The specified module could not be found.

[sampotter]

Hmm, I looked through what you sent, but nothing jumps out at me. Unfortunately, I don't have a Windows machine to test with at the moment, and I don't use Python on Windows very often (if at all). Of course, I would be very happy to get this issue resolved, but I'm going to need to punt and ask you to try to figure this out yourself... If you can determine what the problem is and put it together in a pull request, I'll trust your judgment and happily merge it. Sorry I can't be more help at the moment.

[adamgranthendry]

@sampotter Not to put you on the spot, but you provided Windows install instructions. How did you verify the install works properly on a Windows machine if you don't use Windows?

[sampotter]

I had access to a Windows machine for work over a year ago, but no longer do. At the time, another person who was using this library (probably the OP for this thread) wanted some help, and I happened to have the time to give it a shot. Of course, this is an open source project. If these instructions don't work for you and you find a way to improve them so that they handle a broader set of cases, it would be very helpful to include them.

[adamgranthendry]

@sampotter I see now, that is completely understandable. Thank you for venturing into Windows for us! I think scikit-build with a CMakeLists.txt script might do nicely to fix the issue I'm seeing here. I'll see what magic I can conjure up and report back if I have any success.

[sampotter]

Sure, please give it a shot if you have time and I'll be happy to merge any PRs if they seem reasonable. Make sure to @ me when you post so I don't miss anything.

[sampotter]

@adam-grant-hendry Any luck with this?

[adam-grant-hendry]

@sampotter Not yet. Been busy with the day job. I got Embree2 to compile though. See pyvista #468

[sampotter]

No problem, and nice job---that's good to know... it will be helpful to let people know there's another option on Windows in case they run into trouble.