Skip to content

Commit

Permalink
Use the term "overlay" to describe trailing data
Browse files Browse the repository at this point in the history
  • Loading branch information
adang1345 committed Jul 31, 2023
1 parent 7a57e24 commit f8e5b7d
Show file tree
Hide file tree
Showing 2 changed files with 27 additions and 29 deletions.
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,7 @@ The path separator to use in the following options is `';'` on Windows and `':'`
- `-w`,`--wheel-dir`: directory to write the repaired wheel (default is `wheelhouse` relative to current working directory)
- `--no-mangle`: name(s) of DLL(s) not to mangle, path-separator-delimited
- `--no-mangle-all`: don't mangle any DLL names
- `--strip`: strip DLLs that contain trailing data when name-mangling. The GNU `strip` utility must be present in `PATH`.
- `--strip`: strip DLLs that contain an overlay when name-mangling. The GNU `strip` utility must be present in `PATH`.
- `-L`,`--lib-sdir`: subdirectory suffix to store vendored DLLs (default `.libs`). For example, if your wheel is named `mywheel-0.0.1-cp310-cp310-win_amd64.whl`, then the vendored DLLs are stored in `mywheel.libs` by default. If your wheel contains a top-level extension module that is not in any package, then this setting is ignored, and vendored DLLs are instead placed directly into `site-packages` when the wheel is installed.
- `--namespace-pkg`: namespace packages, specified in case-sensitive dot notation and delimited by the path separator. Normally, we patch or create `__init__.py` in each top-level package to add the vendored DLL location to the DLL search path at runtime. If you have a top-level namespace package that requires `__init__.py` to be absent or unmodified, then this technique can cause problems. This option tells `delvewheel` to use an alternate strategy that does not create or modify `__init__.py` at the root of the given namespace package(s). For example,
- `--namespace-pkg package1` declares `package1` as a namespace package.
Expand All @@ -71,7 +71,7 @@ The path separator to use in the following options is `';'` on Windows and `':'`

- `delvewheel` reads DLL file headers to determine which libraries a wheel depends on. DLLs that are loaded at runtime using `ctypes`/`cffi` (from Python) or `LoadLibrary` (from C/C++) will be missed. You can, however, specify additional DLLs to vendor into the wheel using the `--add-dll` option. If you elect to do this, it is your responsibility to ensure that the additional DLL is found at load time.
- Wheels created using `delvewheel` are not guaranteed to work on systems older than Windows 7 SP1. If you intend to create a wheel for an old Windows system, you should test the resultant wheel thoroughly. If it turns out that getting the wheel to work on an older system simply requires an extra DLL, you can use the `--add-dll` flag to vendor additional DLLs into the wheel.
- To avoid DLL hell, we mangle the file names of most DLLs that are vendored into the wheel. This way, a Python process that tries loading a vendored DLL does not end up using a different DLL with the same name. Due to a limitation in how name-mangling is performed, `delvewheel` is unable to name-mangle DLLs whose dependents contain insufficient internal padding to fit the mangled names and contain trailing data at the end of the binary. An exception will be raised if such a DLL is encountered. Commonly, trailing data consist of symbols that can be safely removed using the GNU `strip` utility, although there exist situations where the data must be present for the DLL to function properly. To remove the trailing data, execute `strip -s EXAMPLE.dll` or use the `--strip` flag. To keep the trailing data and skip name mangling, use the `--no-mangle` or `--no-mangle-all` flag.
- To avoid DLL hell, we mangle the file names of most DLLs that are vendored into the wheel. This way, a Python process that tries loading a vendored DLL does not end up using a different DLL with the same name. Due to a limitation in how name-mangling is performed, `delvewheel` is unable to name-mangle DLLs whose dependents contain insufficient internal padding to fit the mangled names and contain an overlay at the end of the binary. An exception will be raised if such a DLL is encountered. Commonly, the overlay consists of symbols that can be safely removed using the GNU `strip` utility, although there exist situations where the data must be present for the DLL to function properly. To remove the overlay, execute `strip -s EXAMPLE.dll` or use the `--strip` flag. To keep the overlay and skip name mangling, use the `--no-mangle` or `--no-mangle-all` flag.
- Any DLL containing an Authenticode signature will have its signature cleared if its dependencies are name-mangled.
- `delvewheel` cannot repair a wheel that contains extension modules targeting more than one CPU architecture (e.g. both `win32` and `win_amd64`). You should create a separate wheel for each CPU architecture and repair each individually.
- If your project has a [delay-load DLL dependency](https://learn.microsoft.com/en-us/cpp/build/reference/linker-support-for-delay-loaded-dlls), you must use a custom delay-load import hook when building the DLL that has the delay-load dependency. This ensures that the directory containing the vendored DLLs is included in the DLL search path when delay-loading. For convenience, we provide a suitable hook for Microsoft Visual C/C++ at [delayload/delayhook.c](delayload/delayhook.c). Add the file to your C/C++ project when building your DLL.
52 changes: 25 additions & 27 deletions delvewheel/_dll_utils.py
Original file line number Diff line number Diff line change
Expand Up @@ -378,8 +378,8 @@ def _are_characteristics_suitable(section: pefile.SectionStructure) -> bool:


def _get_pe_size_and_enough_padding(pe: pefile.PE, new_dlls: typing.Iterable[bytes]) -> typing.Tuple[int, bool]:
"""Determine the size of a PE file (excluding any trailing data) and
whether the file has enough padding for writing the elements of new_dlls.
"""Determine the size of a PE file (excluding any overlay) and whether the
file has enough padding for writing the elements of new_dlls.
Determining whether the file has enough padding is an instance of the NP-
complete bin packing problem, so we use the Next Fit approximation
Expand Down Expand Up @@ -413,8 +413,8 @@ def replace_needed(lib_path: str, old_deps: typing.List[str], name_map: typing.D
old_deps: a subset of the dependencies that lib_path has, in list form
name_map: a dict that maps an old dependency name to a new name, must
contain at least all the keys in old_deps
strip: whether to try to strip DLLs that contain trailing data if not
enough internal padding exists
strip: whether to try to strip DLLs that contain overlays if not enough
internal padding exists
verbose: verbosity level, 0 to 2
test: testing options for internal use"""
if not old_deps:
Expand All @@ -423,9 +423,9 @@ def replace_needed(lib_path: str, old_deps: typing.List[str], name_map: typing.D
name_map = {dep.lower().encode('utf-8'): name_map[dep].encode('utf-8') for dep in old_deps}
# keep only the DLLs that will be mangled

# If an attribute certificate table exists and is the only trailing data,
# remove the table. In this case, we end up removing all trailing data
# without needing to run strip.
# If an attribute certificate table exists and is the only thing in the
# overlay, remove the table. In this case, we end up removing the entire
# overlay without needing to run strip.
with PEContext(lib_path, None, False, verbose) as pe:
pe_size = max(section.PointerToRawData + section.SizeOfRawData for section in pe.sections)
cert_table = pe.OPTIONAL_HEADER.DATA_DIRECTORY[pefile.DIRECTORY_ENTRY['IMAGE_DIRECTORY_ENTRY_SECURITY']]
Expand All @@ -438,18 +438,18 @@ def replace_needed(lib_path: str, old_deps: typing.List[str], name_map: typing.D
# overwrite the bytes of the old dependency names. Determine whether the PE
# file has enough usable internal padding to use to write the new
# dependency names. If so, overwrite the padding. Otherwise, as long as the
# PE file contains no trailing data or the trailing data can be stripped,
# append a new PE section to store the new names. Determining whether
# enough internal padding exists is an instance of the bin packing problem
# in which the new dependency names are items and the contiguous padding
# runs are bins. The bin packing problem is NP-hard, so for simplicity, we
# use the Next Fit algorithm.
# PE file contains no overlay or the overlay can be stripped, append a new
# PE section to store the new names. Determining whether enough internal
# padding exists is an instance of the bin packing problem in which the new
# dependency names are items and the contiguous padding runs are bins. The
# bin packing problem is NP-hard, so for simplicity, we use the Next Fit
# algorithm.
with PEContext(lib_path, None, False, verbose) as pe:
pe_size, enough_padding = _get_pe_size_and_enough_padding(pe, name_map.values())
if 'not_enough_padding' in test:
enough_padding = False
if not enough_padding and pe_size < os.path.getsize(lib_path) and strip:
# try to strip the trailing data
# try to strip the overlay
try:
subprocess.check_call(['strip', '-s', lib_path])
except FileNotFoundError:
Expand All @@ -459,42 +459,40 @@ def replace_needed(lib_path: str, old_deps: typing.List[str], name_map: typing.D

lib_name = os.path.basename(lib_path)
if not enough_padding and pe_size < os.path.getsize(lib_path):
# cannot rename dependencies due to trailing data
# cannot rename dependencies due to overlay
if strip:
raise RuntimeError(textwrap.fill(
f'Unable to rename the dependencies of {lib_name} because '
'this DLL does not contain enough internal padding to fit the '
'new dependency names, and it contains trailing data after '
'the point where the DLL file specification ends. The GNU '
'new dependency names, and it contains an overlay. The GNU '
'strip utility was run automatically in attempt to remove the '
'trailing data but failed to remove all of it. Unless you '
'have knowledge to the contrary, you should assume that the '
'trailing data exist for an important reason and are not safe '
f'to remove. Include {os.pathsep.join(old_deps)} in the '
'overlay but failed to remove all of it. Unless you have '
'knowledge to the contrary, you should assume that the '
'overlay exists for an important reason and is not safe to '
f'remove. Include {os.pathsep.join(old_deps)} in the '
'--no-mangle flag to fix this error.',
initial_indent=' ' * len('RuntimeError: ')).lstrip())
else:
error_text = [
textwrap.fill(
f'Unable to rename the dependencies of {lib_name} because '
'this DLL does not contain enough internal padding to fit '
'the new dependency names, and it contains trailing data '
'after the point where the DLL file specification ends. '
'Commonly, the trailing data consist of symbols that can '
'be safely removed, although there exist situations where '
'the new dependency names, and it contains an overlay. '
'Commonly, the overlay consists of symbols that can be '
'safely removed, although there exist situations where '
'the data must be present for the DLL to function '
'properly. Here are your options.',
initial_indent=' ' * len('RuntimeError: ')).lstrip(),
'\n',
textwrap.fill(
'- Try to remove the trailing data using the GNU strip '
'- Try to remove the overlay using the GNU strip '
f"utility with the command `strip -s {lib_name}'.",
subsequent_indent=' '
),
'\n',
textwrap.fill(
'- Use the --strip flag to ask delvewheel to execute '
'strip automatically when trailing data are detected.',
'strip automatically when an overlay is detected.',
subsequent_indent=' '
),
'\n',
Expand Down

0 comments on commit f8e5b7d

Please sign in to comment.