Fat Build on Windows #236
Comments
|
No, unfortunately not.
…On Nov 25, 2017 2:58 PM, "D.U.Hadler" ***@***.***> wrote:
Is there any way to produce a fat build on Windows with the MSVC toolchain?
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
<#236>, or mute the thread
<https://github.com/notifications/unsubscribe-auth/AAOzps9swiQucMWvlB0Zja3SzX55SuI6ks5s6B0UgaJpZM4Qqb6k>
.
|
|
Thanks for the quick response!. |
|
It's very ABI dependent. The Linux fat build system is thousands of lines
of hacks. It's a really nontrivial job and took me months to get working on
64 bit machines.
I don't know if it is theoretically possible on Windows. There are
requirements on the number of registers required and various build system
hacks are needed to make it work on Linux. If it is possible on Windows, I
doubt any of us know how to do it. It's also the first time we've been
asked, that I recall.
…On 25 November 2017 at 15:55, D.U.Hadler ***@***.***> wrote:
Thanks for the quick response!.
Just curious: what is the reason?
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#236 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/AAOzprqDD0vUBkqvM1i1HN_vuNsEXVeDks5s6CpogaJpZM4Qqb6k>
.
|
|
Theoretically it ought to be possible to build MPIR using autoconf/automake and cl.exe as the C compiler (that was the approach taken at some point by CGAL project). What should be much more feasible is to port MPIR to cmake, and cmake is able to create Windows makefiles to be run with nmake. (CGAL presently uses cmake). This would also largely alleviate the need for that VC metadata in build.vc*/ |
|
As well, it used to be perfectly possible to call VC compiler from Cygwin. Why would one want to jump hoops, if there is a working build system, just use the right command-line compiler? |
|
Windows programmers don't use Cygwin, autoconf or cmake!
At any rate, this doesn't make it any easier to do fat binaries. The ABI
and entire way fat binaries work is different on Windows. Autoconf can't
even build linux fat binaries, so Windows ones are certainly out of the
question.
…On 29 November 2017 at 13:23, Dima Pasechnik ***@***.***> wrote:
As well, it used to be perfectly possible call VC compiler from Cygwin.
Why would one want to jump hoops, if there is a working build system, just
use the right command-line compiler?
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#236 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/AAOzpr2vB3p3hx0NJ1AhEQBo2fwWu0rcks5s7UywgaJpZM4Qqb6k>
.
|
|
On Wed, Nov 29, 2017 at 1:35 PM, wbhart ***@***.***> wrote:
Windows programmers don't use Cygwin, autoconf or cmake!
is this an opening salvo in a flamewar? :-)
At any rate, this doesn't make it any easier to do fat binaries. The ABI
and entire way fat binaries work is different on Windows. Autoconf can't
even build linux fat binaries, so Windows ones are certainly out of the
question.
You do have `--enable-fat`, don't you?
Are you saying it's broken?
It certainly seems to work in 3.0.0, and all it takes is a
couple of lines in `configure.ac`.
|
|
On 29 November 2017 at 14:54, Dima Pasechnik <[email protected]>
wrote:
On Wed, Nov 29, 2017 at 1:35 PM, wbhart ***@***.***> wrote:
> Windows programmers don't use Cygwin, autoconf or cmake!
>
is this an opening salvo in a flamewar? :-)
No, of course not. Windows programmers use MSVC because that is their
ecosystem. Recommending they use Cygwin or various Linux tools is like
suggesting Linux people use Windows to configure their stuff.
You want to talk to Brian Gladman if you want to do something Windows
specific. He is fully up with what Windows users are actually making use of.
>
> At any rate, this doesn't make it any easier to do fat binaries. The ABI
> and entire way fat binaries work is different on Windows. Autoconf can't
> even build linux fat binaries, so Windows ones are certainly out of the
> question.
>
You do have `--enable-fat`, don't you?
Yes.
Are you saying it's broken?
No.
It certainly seems to work in 3.0.0, and all it takes is a
couple of lines in `configure.ac`.
Sorry, but you misunderstood. There are thousands of lines of code
dedicated to building fat binaries. And it is all *very* OS and arch
specific!! It's not just a matter of "enabling" that option on Windows.
It's 3 months of work, minimum!
|
|
One talks about different types of I understand now that MPIR is doing something else here. And as far as toolsets on Windows are concerned, one is not wedded to Visual Studio IDE. |
|
On 29 November 2017 at 19:57, Dima Pasechnik ***@***.***> wrote:
One talks about different types of enable-fat. One is simply about making
a binary that works on as many CPUs as possible (also known as universal).
This is well-supported on OSX via autoconf (or directly by OSX tools), by
the way. It used to be possible to make a binary that works on PPC and on
x86 and on x86_64. The right for the arch binary is then enabled at loading
time (not at runtime).
I understand now that MPIR is doing something else here.
MPIR is an assembly language library. I'm pretty sure autoconf doesn't
support building fat binaries for that.
And as far as toolsets on Windows are concerned, one is not wedded to
Visual Studio IDE.
You are misunderstanding. The VS files are in there because that is what
Windows developers want. The fact that some other option is possible is
irrelevant.
D is in there for development, not for deployment. By the way, I don't
understand why all these build.vs*/ are even getting into the main MPIR
repo. It would be perfectly fine for them to be in a git submodule, and not
being bundled into the main distribution either.
Because Windows developers want/need them. We are not the only project to
include VS files.
At this stage, I think you are trying to wind me up or something. Brian
Gladman has contributed Windows support for GMP/MPIR for years, and is
totally plugged in to what that community wants. That's part of the reason
MPIR even exists!
|
|
On Wed, Nov 29, 2017 at 7:04 PM, wbhart ***@***.***> wrote:
> And as far as toolsets on Windows are concerned, one is not wedded to
> Visual Studio IDE.
>
You are misunderstanding. The VS files are in there because that is what
Windows developers want. The fact that some other option is possible is
irrelevant.
I am merely expressing a surprise that there are no XCode metadata files,
Eclipse metadata files, IntelliJ metadata files, etc in the repo yet.
But seriously, the repo is hard to navigate. Like, 50% of the 700K diff
between the master and 3.0.0 consists of Windows metadata files.
Can't it get more modular? Why can't these build.vc* and mpir.net go to a
git submodule (or two submodules)? That's where they ought to be by right,
no? It won't make the setup for the VC folks harder.
> D is in there for development, not for deployment. By the way, I don't
> understand why all these build.vs*/ are even getting into the main MPIR
> repo. It would be perfectly fine for them to be in a git submodule, and
not
> being bundled into the main distribution either.
>
Because Windows developers want/need them. We are not the only project to
include VS files.
At this stage, I think you are trying to wind me up or something. Brian
Gladman has contributed Windows support for GMP/MPIR for years, and is
totally plugged in to what that community wants. That's part of the reason
MPIR even exists!
I certainly am not trying to wind anyone up. I am not saying that these
pieces must not be there, I'm merely pledging for some source modularity.
|
FYI, this might change in the future. Builtin CMake support is included in VS2017 IDE. (VS2017 can target older VS versions too). Microsoft also has a package manager for C/C++ projects called |
|
On Wed, Nov 29, 2017 at 7:44 PM, Isuru Fernando ***@***.***> wrote:
And as far as toolsets on Windows are concerned, one is not wedded to
Visual Studio IDE.
You are misunderstanding. The VS files are in there because that is what
Windows developers want. The fact that some other option is possible is
irrelevant.
FYI, this might change in the future. Builtin CMake support is included in
VS2017 IDE. (VS2017 can target older VS versions too). Microsoft also has a
package manager for C/C++ projects called vcpkg which is written using
cmake and integrates well with VS2015 and VS2017. (MPIR is installable
using vcpkg)
Hear, hear! I think this illustrates that Microsoft understands that Visual
Studio alone cannot be flexible enough tool for large-scale projects...
|
|
On 29 November 2017 at 20:49, Dima Pasechnik <[email protected]>
wrote:
On Wed, Nov 29, 2017 at 7:44 PM, Isuru Fernando ***@***.***>
wrote:
> And as far as toolsets on Windows are concerned, one is not wedded to
> Visual Studio IDE.
>
> You are misunderstanding. The VS files are in there because that is what
> Windows developers want. The fact that some other option is possible is
> irrelevant.
>
> FYI, this might change in the future. Builtin CMake support is included
in
> VS2017 IDE. (VS2017 can target older VS versions too). Microsoft also
has a
> package manager for C/C++ projects called vcpkg which is written using
> cmake and integrates well with VS2015 and VS2017. (MPIR is installable
> using vcpkg)
>
Hear, hear! I think this illustrates that Microsoft understands that Visual
Studio alone cannot be flexible enough tool for large-scale projects.
Please discuss this with @BrianGladman
I prefer not to second guess what he actually wants.
|
|
I am coming very late to this thread as I did not know of its existence earlier (I am the sole developer/maintainer of the Visual Studio build system for MPIR). As Bill has said, MPIR exists precisely because of the need for a build system for GMP that uses the native Microsoft build tools (an approach that the GMP developers rejected at the time of the fork). In my view a FAT binary build with Visual Studio would be possible. However, this is not a part of the current build system for two reasons: (1) there has not been demand for this in the community that builds MPIR with Visual Studio, and (2) the cost of developing and maintaining such a FAT build would be high. Contrary to the view expressed above, Visual Studio build tools have been delivering thousands of large scale projects on Windows for many years. To suggest that they don't have the flexibility to do this is completely incorrect. I suspect that this view is based on a profound misunderstanding of the different approaches for applications delivery in the *nix and Windows environments. Turing to CMake and vcpkg, Microsoft has been extending the reach of Visual Studio into the open source domain by adding support for these in Visual Studio 2017. This is an interesting development and one that it would be nice to harness if we had the resources to do so. But I am the only person working on the Visual Studio build system and, for several reasons I have not yet been able to invest in these new and potentially valuable developments. The Visual Studio build system for MPIR involves a lot of code, which will mean that moving it to CMake and then to vcpkg wil be a significant task, especially so if a FAT build were to be included. This will almost certainly need good knowledge of CMake but, surprisingly, I have not been able to find any accessible in-depth paper based documentation for it (yes I like to read such stuff on paper!) The Kitware book appears to meet this need but it is insanely expensive and I am unwilling to spend money on it as my work on MPIR is completely unfunded. So, although I would like to look at using CMake/vcpkg for the Visual Studio build, the lack of free in-depth paper based documentation of CMake has so far prevented any work by me on this. |
|
On Thu, Nov 30, 2017 at 9:33 AM, Brian Gladman ***@***.***> wrote:
I am coming very late to this thread as I did not know of its existence
earlier (I am the sole developer/maintainer of the Visual Studio build
system for MPIR).
As Bill has said, MPIR exists precisely because of the need for a build
system for GMP that uses the native Microsoft build tools (an approach that
the GMP developers rejected at the time of the fork).
In my view a FAT binary build with Visual Studio would be possible.
However, this is not a part of the current build system for two reasons:
(1) there has not been demand for this in the community that builds MPIR
with Visual Studio, and (2) the cost of developing and maintaining such a
FAT build would be high.
Contrary to the view expressed above, Visual Studio build tools have been
delivering thousands of large scale projects on Windows for many years. To
suggest that they don't have the flexibility to do this is completely
incorrect. I suspect that this view is based on a profound misunderstanding
of the different approaches for applications delivery in the *nix and
Windows environments.
I have no doubt that VS might be an adequate tool to deliver all sorts of
applications, but MPIR is not an application.
Suppose one wants an nmake-based way of building MPIR on Windows. Indeed,
there is at least one obvious use for this this, namely testing on AppVeyor
using the native Windows toolchain. This is currently lacking, probably for
the reason that most attention on Windows has been on a tool less than
perfect for the job.
Other use cases for this would be to use clang on Windows
https://clang.llvm.org/get_started.html
(Probably important for Julia/Nemo support? Probably needs cmake.)
Turing to CMake and vcpkg, Microsoft has been extending the reach of Visual
Studio into the open source domain by adding support for these in Visual
Studio 2017. This is an interesting development and one that it would be
nice to harness if we had the resources to do so. But I am the only person
working on the Visual Studio build system and, for several reasons I have
not yet been able to invest in these new and potentially valuable
developments.
it has simply come the full circle (with cmake replacing autotools) since
late 1990s-early 2000s (time I used Visual C++ quite a bit, crashing the
compiler dozens of time a day).
By the way, what do you think about
https://code.visualstudio.com/ ?
The Visual Studio build system for MPIR involves a lot of code, which will
mean that moving it to CMake and then to vcpkg wil be a significant task,
especially so if a FAT build were to be included. This will almost
certainly need good knowledge of CMake but, surprisingly, I have not been
able to find any accessible in-depth paper based documentation for it (yes
I like to read such stuff on paper!)
The Kitware book appears to meet this need but it is insanely expensive
and I am unwilling to spend money on it as my work on MPIR is completely
unfunded. So, although I would like to look at using CMake/vcpkg for the
Visual Studio build, the lack of free in-depth paper based documentation of
CMake has so far prevented any work by me on this.
We can arrange for you
https://www.amazon.com/Mastering-CMake-Ken-Martin/dp/1930934319 (if this is
the book you mean)
via a grant we have. Please email me your shipping address.
|
|
Thank you for your rapid and detailed response to my input. I do not doubt that the current build system for Windows is not going to meet everyone's current needs. However, it has been an effective way of building MPIR over the last decade and is only now showing its age as a result of the recent steps that Microsoft has made to extend its tools into the open source domain. It would certainly be very nice if it proved possible to make use of these recent developments but I honestly do not know whether I will have the time to undertake what I fear will be a large task. I will email you separately on this. |
|
I am very happy to learn that a FAT binary build with Visual Studio would (in principle) be possible! So I would like to suggest that this option will be included in future MPIR builds with Visual Studio. I am also very interested to work with the MPIR team (and I guess, in particular with Brain Gladman) to help developing and maintaining such a build. This might also involve some work regarding the current (MPIR 3.0.0) build using MSYS2 and MinGW-w64, which does not support a FAT build properly on Windows 64 bit (I had pointed this out earlier, see #213). The reason for wanting a FAT build in the first place is maybe best explained by looking at a Python project like gmpy2 (see https://pypi.python.org/pypi/gmpy2), which provides an Python interface to MPIR, MPFR and MPC. On Python for Windows, the expectation is to distribute a binary build, not just the source code. A binary build on 64 bit will have to use the AMD K8 configuration as the lowest common denominator. This does not really do justice to the support which MPIR provides for newer CPUs. The alternative would be to provide many more CPU-specific binary builds multiplied by the number of supported Python versions, which seems neither practical nor desirable. A 64 bit distribution of gmpy2 with a FAT binary build would solve this dilemma. |
|
@duhadler I should stress again that a creating a Visual Studio FAT build will be a VERY difficult task. I looked at this several years ago and it did not take me long to conclude that it was not something that I was very keen to take on! |
|
I don't have any idea how to do this on Windows. Once again, I'm not sure
it is even possible in the sense that it is meant in MPIR due to the lack
of registers.
There seems to be something else people refer to as fat binaries that might
be a possibility. I don't know how they work.
The MPIR fat binaries on *nix work by populating a table of function
pointers. The first time the function is called an assembly stub finds the
right version of the assembly function and puts a pointer in the table and
then calls it. Subsequent times the function in the table is called
directly. Obviously this means detecting the arch at runtime.
All the assembly functions for all the different arches are compiled with
different names (the name of the arch is appended to their global symbols
to distinguish them). This means all the versions are available at once,
but only the one in the table of function pointers is actually called.
I think it's about three months of full time work for someone to get this
to work, if it is even possible on Windows. I recall that the register
juggling was very tricky on AMD64. When the assembly function is first
called, the parameters for the function are in the registers. You can't
disturb these, but you need some registers to do the computation of which
assembly version should be called. The more juggling you do, the less
efficient the whole thing, and the times go back to the same as a generic C
build!
…On 1 December 2017 at 16:00, Brian Gladman ***@***.***> wrote:
@duhadler <https://github.com/duhadler> I should stress again that a
creating a Visual Studio FAT build will be a VERY difficult task. I looked
at this several years ago and it did not take me long to conclude that it
was not something that I was very keen to take on!
@wbhart <https://github.com/wbhart> Bill, when I looked at this several
years ago you were kind enough to set out in detail what I would have to do
to match how this is achieved with *nix/GCC. I am wondering if you might
still have this description around?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#236 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/AAOzppz1SZIG6t0W0OZjsyl6AUE4wm5gks5s8BSZgaJpZM4Qqb6k>
.
|
|
If, say, we want a FAT build involving N functions, I would have thought we would have each of the N functions called through a single function pointer in a table of N pointers which is set up on the first call to any of the N functions (i.e. all the pointers are initially pointing to initialisation routines that will set up all the pointers for the detected architecture before completing the called function). This first call will, of course, be costly but I can't see why the costs would be high on any subsequent calls since these will involve only one extra jump instruction. I also believe this could be done using an indexed jump instruction with a fixed index so it would not require an extra register. Am I missing something obvious here? |
|
MPIR 2.7.2 using MSYS2 and MinGW-w64 (GCC 4.8.3 or later) has full support for a FAT build on Windows 64 bit. The resulting DLL can be called from executables which depend on the GCC runtime, like R (Statistical System), or which do not depend on any C++ runtime, like C# or Visual Basic. However, this 64 bit DLL cannot be called from executables which depend on a current version of the MSVC runtime (2010 or later) if the code contains anything that the MSVC compiler believes needs an object initialisation (which is the reason why MinGW-w64 cannot be used for Python, Matlab, Java 64 bit on Windows, since these languages depend on recent versions of the MSVC runtime). Python, Matlab, Java 64 bit can however call the MPIR 2.7.2 DLL when it is built with the MSVC tool chain, and linked against the MSVC 2010 runtime or later. So I would assume that, since MinGW-w64 can produce a working FAT build on Windows 64 bit, this should in principle also be possible with the MSVC tool chain, unless recent versions of the MSVC runtime contain restrictions (registers etc) that make that impossible. Since the MinGW-w64 runtime itself links against an ancient version of the MSVC runtime (msvcrt.dll), I would think that this is worth a try. |
|
I don't know who added Fat binary support for MPIR on Windows, since it certainly wasn't me or Brian. The calling conventions are different on Windows, which means different registers are used to pass arguments. So if this actually works, I certainly don't know how. I can believe it builds, but I have a hard time believing that it actually works. But maybe someone added support for it, and I have just forgotten. |
|
@BrianGladman I don't recall if the entire table is set up on the first call to any function or if each function is set up individually on the first call. But either should work. There is still overhead because you need a short function to actually call the relevant assembly object through the table. This does have overhead. Even on Linux, a fat binary build is much slower than a native build. But it is still faster than a generic C build. |
|
I think there may also be some overhead since some of the optimisation is put out of kilter by the extra function call. On modern CPUs the indirect function call probably isn't a big source of slowdown. Neither of these sources of overhead is relevant here of course; they both occur on Linux and Windows. |
|
Both GMP 6.1.2 and MPIR 2.7.2, using MSYS2 and MinGW-w64 (GCC 4.8.3 or later), have full support for a FAT build on Windows 64 bit. They not only can be built, they actually work. This can be verified as follows: Download MSYS2 (64 bit) from www.msys2.org Install the downloaded package to C:\msys64 and follow the instructions from the homepage. Close MSYS2 and restart (i.e. double-click on mingw64.exe in the installation directory). pacman -S base-devel mingw-w64-x86_64-toolchain and install everything. Close MSYS2 and restart. Place the GMP 6.1.2 and MPIR 2.7.2 directories in C:\msys64\home\MP64. Within MSYS2, type ./configure --build=x86_64-w64-mingw32 --host=x86_64-w64-mingw32 --enable-fat --enable-static --disable-shared ABI=64 CFLAGS="-m64 -march=k8 -Wno-attributes" --prefix=/local/Win64 --exec-prefix=/local/Win64 to configure the GMP source files for compilation of a FAT 64 bit static library. Once this has been completed type |
|
As @isuruf said, CMake is pretty much going to be the default build system for new C/C++/Assembly projects going forward. This is because CMake works with all major operating systems (Windows/OSX/Linux), and most importantly, like autoconf, requires zero maintenance once set up (you don't need to regenerate build files manually for each VC version). Now, whether existing projects such as MPIR and flint2 will adopt CMake anytime soon is a different question. Switching build systems requires significant effort, and it will take time to bring the CMake build system up to the capabilities of the autoconf build system. |
|
@ghost, @dimpase, @isuruf - Resurrecting this old thread :) - I see a lot of talk about CMake here (for which we now have #186, #214, #215). I seem to be missing the link between CMake and a theoretical FAT build on Windows. Does CMake make this possible? @BrianGladman, @wbhart - Spitballing here - perhaps the FAT binary functionality could be achieved with the Microsoft toolchain using a different approach compared to the dynamic jump table you described in Linux: |
|
@KevinHake There is a general desire to have a CMake based build for MPIR on Windows, especially so, given that Visual Studio 2017 directly supports it (there is already such a build for the MPIR library using the generic C code). But I doubt that CMake is going to make any FAT build easier to achieve. Turning to your thought on a FAT binary, the idea you put forward is an interesting one We wouldn't have to have a special DLL since we can just have a routine to detect the underlying architecture after which we would simply dynamically load the fastest compatible one. I have not implemented dynamic loading with MPIR but I have done this with my AES library and it was not too hard to implement. As you say, the downside is a large expansion in the size of the distribution. Another alternative is to note that only a small number of routines are sped up using assembler so we could put all of these into a set of dynamically loaded optimised DLLs with another standard DLL for the remaining C code. This would considerably reduce the size of the distribution. We should be able to do something similar with static libraries. |
|
My biggest problem with this FAT business is the need for it in the open-source realm. Building MPIR on the host it has to run on is reasonably fast, needs to be done just once, and may be done with free tools. |
|
It could also be argued that an application meant for distribution, not just a one-off research project or what have you, which is also concerned about performance, should really use an installer. And that installer should really be responsible for detecting the local architecture and choosing the most optimized dll(s) to use. From a software architecture standpoint, FAT binaries in general are kind of a flawed approach; They're bigger, slower, more complicated: harder to develop and debug, compared to a single-architecture binary. That's why they haven't become standard practice, and never even implemented natively in Windows. |
Is there any way to produce a fat build on Windows with the MSVC toolchain?
The text was updated successfully, but these errors were encountered: