Notes about specific Features¶
Ctypes Dependencies¶
Ctypes is a foreign function library for Python, that allows calling functions present in shared libraries. Those libraries are not imported as Python packages, because they are not picked up via Python imports: their path is passed to ctypes instead, which deals with the shared library directly; this caused <1.4 PyInstaller import detect machinery to miss those libraries, failing the goal to build self-contained PyInstaller executables:
from ctypes import *
# This will pass undetected under PyInstaller detect machinery,
# because it's not a direct import.
handle = CDLL("/usr/lib/library.so")
handle.function_call()
Solution in PyInstaller¶
PyInstaller contains a pragmatic implementation of Ctypes dependencies: it will search for simple standard usages of ctypes and automatically track and bundle the referenced libraries. The following usages will be correctly detected:
CDLL("library.so")
WinDLL("library.so")
ctypes.DLL("library.so")
cdll.library # Only valid under Windows - a limitation of ctypes, not PyInstaller's
windll.library # Only valid under Windows - a limitation of ctypes, not PyInstaller's
cdll.LoadLibrary("library.so")
windll.LoadLibrary("library.so")
More in detail, the following restrictions apply:
only libraries referenced by bare filenames (e.g. no leading paths) will be handled; handling absolute paths would be impossible without modifying the bytecode as well (remember that while running frozen, ctypes would keep searching the library at that very absolute location, whose presence on the host system nobody can guarantee), and relative paths handling would require recreating in the frozen executable the same hierarchy of directories leading to the library, in addition of keeping track of which the current working directory is;
only library paths represented by a literal string will be detected and included in the final executable: PyInstaller import detection works by inspecting raw Python bytecode, and since you can pass the library path to ctypes using a string (that can be represented by a literal in the code, but also by a variable, by the return value of an arbitrarily complex function, etc…), it’s not reasonably possible to detect all ctypes dependencies;
only libraries referenced in the same context of ctypes’ invocation will be handled.
We feel that it should be enough to cover most ctypes’ usages, with little or no modification required in your code.
If PyInstaller does not detect a library, you can add it to your
bundle by passing the respective information to --add-binary option or
listing it in the .spec-file. If your frozen
application will be able to pick up the library at run-time can not be
guaranteed as it depends on the detailed implementation.
Gotchas¶
The ctypes detection system at Analysis time
is based on ctypes.util.find_library().
This means that you have to make sure
that while performing Analysis and running frozen,
all the environment values find_library() uses to search libraries
are aligned to those when running un-frozen.
Examples include using LD_LIBRARY_PATH or DYLD_LIBRARY_PATH to
widen find_library() scope.
SWIG support¶
PyInstaller tries to detect binary modules created by SWIG. This detection requires:
The Python wrapper module must be imported somewhere in your application (or by any of the modules it uses).
The wrapper module must be available as source-code and it’s first line must contain the text
automatically generated by SWIG.The C-module must have the same name as the wrapper module prefixed with an underscore (
_). (This is a SWIG restriction already.)The C-module must sit just beside the wrapper module (thus a relative import would work).
Also some restrictions apply, due to the way the SWIG wrapper is implemented:
The C-module will become a global module. As a consequence, you can not use two SWIG modules with the same basename (e.g.
pkg1._cmodandpkg2._cmod), as one would overwrite the other.
Cython support¶
PyInstaller can follow import statements that refer to Cython C object modules and bundle them – like for any other module implemented in C.
But – again, as for any other module implemented in C – PyInstaller can not
determine if the Cython C object module is importing some Python module.
These will typically show up as in a traceback like this
(mind the .pyx extension):
Traceback (most recent call last):
[…]
File "myapp\cython_module.pyx", line 3, in init myapp.cython_module
ModuleNotFoundError: No module named 'csv'
So if you are using a Cython C object module, which imports Python modules,
you will have to list these as --hidden-import.
macOS multi-arch support¶
With the introduction of Apple Silicon M1, there are now several architecture options available for python:
single-arch
x86_64with thin binaries: older python.org builds, Homebrew python running natively on Intel Macs or under rosetta2 on M1 Macssingle-arch
arm64with thin binaries: Homebrew python running natively on M1 macsmulti-arch
universal2with fat binaries (i.e., containing bothx86_64andarm64slices): recentuniversal2python.org builds
PyInstaller aims to support all possible combinations stemming from the above options:
single-arch application created using corresponding single-arch python
universal2application created usinguniversal2pythonsingle-arch application created using
universal2python (i.e., reducinguniversal2fat binaries into eitherx86_64orarm64thin binaries)
By default, PyInstaller targets the current running architecture
and produces a single-arch binary (x86_64 when running on Intel Mac
or under rosetta2 on M1 Mac, or arm64 when running on M1 Mac). The
reason for that is that even with a universal2 python environment,
some packages may end up providing only single-arch binaries, making it
impossible to create a functional universal2 frozen application.
The alternative options, such as creating a universal2 version
of frozen application, or creating a non-native single-arch version using
universal2 environment, must therefore be explicitly enabled. This
can be done either by specifying the target architecture in the .spec
file via the target_arch= argument to EXE(), or on command-line
via the --target-arch switch. Valid values are x86_64, arm64,
and universal2.
Architecture validation during binary collection¶
To prevent run-time issues caused by missing or mismatched architecture slices in binaries, the binary collection process performs strict architecture validation. It checks whether collected binary files contain required arch slice(s), and if not, the build process is aborted with an error message about the problematic binary.
In such cases, creating frozen application for the selected target
architecture will not be possible unless the problem of missing arch slices
is manually addressed (for example, by downloading the wheel corresponding to
the missing architecture, and stiching the offending binary files together
using the lipo utility).
Changed in version 4.10: In earlier PyInstaller versions, the architecture validation was performed on all collected binaries, such as python extension modules and the shared libraries referenced by those extensions. As of PyInstaller 4.10, the architecture validation is limited to only python extension modules.
The individual architecture slices in a multi-arch universal2 extension
may be linked against (slices in) universal2 shared libraries, or
against distinct single-arch thin shared libraries. This latter case makes
it impossible to reliably validate architecture of the collected shared
libraries w.r.t. the target application architecture.
However, the extension modules do need to be fully compatible with the target application architecture. Therefore, their continued validation should hopefully suffice to detect attempts at using incompatible single-arch python packages *.
- *
Although nothing really prevents a package from having distinct, architecture-specific extension modules…
Trimming fat binaries for single-arch targets¶
When targeting a single architecture, the build process extracts the
corresponding arch slice from any collected fat binaries, including the
bootloader. This results in a completely thin build even when building
in universal2 python environment.
macOS binary code signing¶
With Apple Silicon M1 architecture, macOS introduced mandatory code signing,
even if ad-hoc (i.e., without actual code-signing identity). This means
that arm64 arch slices (but possibly also x86_64 ones, especially
in universal2 binaries) in collected binaries always come with signature.
The processing of binaries done by PyInstaller (e.g., library path rewriting in binaries’ headers) invalidates their signatures. Therefore, the signatures need to be re-generated, otherwise the OS refuses to load a binary.
By default, PyInstaller ad-hoc (re)signs all collected binaries and
the generated executable itself. Instead of ad-hoc signing, it is also
possible to use real code-signing identity. To do so, either specify your
identity in the .spec file via codesign_identity= argument to
EXE() , or on command-line via the --codesign-identity switch.
Being able to provide codesign identity allows user to ensure that all
collected binaries in either onefile or onedir build are signed
with their identity. This is useful because for onefile builds,
signing of embedded binaries cannot be performed in a post-processing step.
Note
When codesign identity is specified, PyInstaller also turns on
hardened runtime by passing --options=runtime to the codesign
command. This requires the codesign identity to be a valid Apple-issued
code signing certificate, and will not work with self-signed certificates.
Trying to use self-signed certificate as a codesign identity will result in shared libraries failing to load, with the following reason reported:
[libname]: code signature in ([libname]) not valid for use in process using Library Validation: mapped file has no Team ID and is not a platform binary (signed with custom identity or adhoc?)
Furthermore, it is possible to specify entitlements file to be used
when signing the collected binaries and the executable. This can be
done in the .spec file via entitlements_file= argument to
EXE(), or on command-line via the --osx-entitlements-file switch.
App bundles¶
PyInstaller also automatically attempts to sign .app bundles, either
using ad-hoc identity or actual signing identity, if provided via
--codesign-identity switch. In addition to passing same options as
when signing collected binaries (identity, hardened runtime, entitlement),
deep signing is also enabled via by passing --deep option to the
codesign utility.
Should the signing of the bundle fail for whatever reason, the error
message from the codesign utility will be printed to the console,
along with a warning that manual intervention and manual signing of the
bundle are required.
macOS event forwarding and argv emulation in app bundles¶
The user interaction with macOS app bundles takes place via so called
Apple Events. When the user double clicks on the application’s icon, the
application is started and receives an Open Application ('oapp') event.
Dragging a document on the application’s icon or attempting to open
an application-registered file generates an Open Document ('odoc') event.
Similarly, launching an URL with application-registered schema generates
a Launch/Get URL ('GURL') event. Typically, a long-running UI application
installs Carbon or Cocoa event handlers (or their equivalents provided
by higher-level UI toolkit) to handle these requests during its runtime.
PyInstaller provides two aspects of support for macOS event handling;
automatic event forwarding, which enables frozen aplication to receive
events in onefile mode, and optional argv emulation for converting
initial opening event into sys.argv arguments. Both aspects apply only
to app bundles (i.e., the windowed bootloader variant) and not to
POSIX (command-line) frozen applications.
Changed in version 5.0: In earlier PyInstaller versions, argv emulation was always enabled
in onefile mode and was unavailable in onedir mode.
As PyInstaller 5.0, argv emulation must be explicitly opted-in,
and is available in both onefile and onedir mode.
Event forwarding¶
In PyInstaller onedir bundles, the application runs as a single
process, and therefore receives Apple Events normally, as other macOS
applications would.
In onefile bundles, the application has a parent launcher process and
the child process; the open document requests generated by user are
received by the parent process, and are automatically forwarded to
the child process, where the frozen python code is running.
Event forwarding is implemented for the following types of Apple Events:
kAEOpenDocuments('odoc'): open document request
kAEGetURL('GURL'): open/launch URL request
kAEReopenApplication('rapp'): reopen application
kAEActivate('actv'): activate application (bring to front)
Optional argv emulation¶
PyInstaller implements an optional feature called argv emulation,
which can be toggled via argv_emulation= argument to EXE()
in the .spec file, or enabled on command-line
via --argv-emulation flag.
If enabled, the bootloader performs initial Apple Event handling to
intercept events during the application’s start-up sequence, and appends
file paths or URLs received via Open Document/URL (‘odoc’ and ‘GURL’)
events to sys.argv, as if they were received via command-line.
This feature is intended for simple applications that do not implement
the event handling, but still wish to process initial open document
request. This applies only to initial open events; events that occur
after the frozen python code is started are dispatched via event queue
(in onedir mode directly, and forwarded to child process in onefile
mode.) and as such need to be handled via event handlers.
Note
This feature is not suitable for long-running applications that may need to service multiple open requests during their lifetime. Such applications will require proper event handling anyay, and therefore do not benefit from having initial events processed by argv emulation.
Warning
The initial event processing performed by bootloader in onedir mode
may interfere with UI toolkit used by frozen python application, such
as Tcl/Tk via tkinter module. The symptoms may range from window
not being brought to front when the application startup to application
crash with segmentation fault.
While PyInstaller tries to mitigate the issue on its end, we recommend against using argv emulation in combination with UI toolkits.
Practical examples¶
This section provides some practical examples on handling file and URL open events in macOS application bundles, via argv emulation in a simple one-shot program, or via installed event handlers in a GUI application.
Registering supported file types and custom URL schemas¶
In order for macOS application bundle to handle open operations
on files and custom URL schemas, the OS needs to be informed what
file types and what URL schemas the application supports. This
is done in the bundle’s Info.plist file, via CFBundleDocumentTypes
and CFBundleURLTypes entries:
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
[...] <!-- preceding entries --->
<key>CFBundleDocumentTypes</key>
<array>
<dict>
<key>CFBundleTypeName</key>
<string>MyCustomFileType</string>
<key>CFBundleTypeExtensions</key>
<array>
<string>mcf</string>
</array>
<key>CFBundleTypeRole</key>
<string>Viewer</string>
</dict>
</array>
<key>CFBundleURLTypes</key>
<array>
<dict>
<key>CFBundleURLName</key>
<string>MyCustomUrlSchema</string>
<key>CFBundleTypeRole</key>
<string>Viewer</string>
<key>CFBundleURLSchemes</key>
<array>
<string>my-url</string>
</array>
</dict>
</array>
</dict>
</plist>
In the above example, the application declares itself a viewer for
made-up .mcf files, and as a viewer for URLs beginning with
my-url://.
PyInstaller automatically generates an Info.plist file for your
application bundle; to have it include the entries shown above, add the
info_plist argument to the BUNDLE() directive in the
.spec file, and set its content as follows:
app = BUNDLE(
# [...]
info_plist={
'CFBundleURLTypes': [{
'CFBundleURLName': 'MyCustomUrlSchema',
'CFBundleTypeRole': 'Viewer',
'CFBundleURLSchemes': ['my-url', ],
}],
'CFBundleDocumentTypes': [{
'CFBundleTypeName': 'MyCustomFileType',
'CFBundleTypeExtensions': ['mcf', ],
'CFBundleTypeRole': "Viewer",
}],
}
)
Open event handling with argv emulation¶
Consider the following python script that began its life as a command-line utility, to be invoked from the terminal:
python3 img2gray.py image1.png image2.png ...
The script processes each passed image, converts it to grayscale, and saves it next to the original, with -gray appended to the file name:
# img2gray.py
import sys
import os
import PIL.Image
if len(sys.argv) < 2:
print(f"Usage: {sys.argv[0]} <filename> [filenames...]")
sys.exit(1)
# Convert all given files
for input_filename in sys.argv[1:]:
filename, ext = os.path.splitext(input_filename)
output_filename = filename + '-gray' + ext
img = PIL.Image.open(input_filename)
img_g = img.convert('L')
img_g.save(output_filename)
If you generate an application bundle (as opposed to a command-line
POSIX application), the most likely way of user interaction will be
dragging image files onto the bundle’s icon or using Open with...
entry from the image file’s context menu. Such interaction generates
open file events, and in general requires your application code to
implement event handling.
Enabling argv emulation in PyInstaller causes its bootloader to
process events during the application startup, and extend sys.argv
with any file paths or URLs that might have been received via open file
or URL requests. This allows your application to process the received
filenames as if they were passed via command-line, without any
modifications to the code itself.
The following .spec file provides
a complete example for a onedir application bundle that allows
conversion of .png and .jpg images:
# img2gray.spec
a = Analysis(['img2gray.py'], )
pyz = PYZ(a.pure, a.zipped_data)
exe = EXE(
pyz,
a.scripts,
exclude_binaries=True,
name='img2gray',
debug=False,
bootloader_ignore_signals=False,
strip=False,
upx=False,
console=False,
argv_emulation=True, # enable argv emulation
)
coll = COLLECT(
exe,
a.binaries,
a.zipfiles,
a.datas,
strip=False,
upx=False,
upx_exclude=[],
name='img2gray'
)
app = BUNDLE(
coll,
name='img2gray.app',
# Register .png and .jpg as supported file types
info_plist={
'CFBundleDocumentTypes': [{
'CFBundleTypeName': "Convertable image types",
'CFBundleTypeExtensions': [
'png', 'jpg',
],
'CFBundleTypeRole': "Viewer",
}],
}
)
The user can now drag image file(s) onto the icon of the resulting img2gray
application bundle, or select img2gray under the Open with... entry
in the image file’s context menu.
Note
The argv emulation handles only initial open event, which is received before your frozen python code is started. If you wish to handle subsequent open requests while the application is still running, you need to implement proper event handling in your python code.
Open event handling in a tkinter-based GUI application¶
The Tcl/Tk framework used by tkinter allows application to
provide event handlers for pre-defined types of Apple Events, by
registering macOS-specific commands.
The handler for open file events can be registered via
::tk::mac::OpenDocument command, while the handler for open URL
events can be registered via ::tk::mac::LaunchURL command. The
latter is available starting with Tcl/Tk 8.6.10 †.
- †
At the time of writing, python.org builds use Tcl/Tk 8.6.5, except for the Python 3.9.x macOS 64-bit universal2 installer builds, which use Tcl/Tk 8.6.10. Homebrew Python requires
tkinterto be explicitly installed aspython-tk, and uses latest version of Tcl/Tk, 8.6.11. Registering::tk::mac::LaunchURLcommand with versions of Tcl/Tk older than 8.6.10 is essentially no-op.
The following application illustrates the event handling using tkinter,
by logging all received open file/URL events into a scrollable text
widget:
# eventlogger_tk.py
import sys
import tkinter
import tkinter.scrolledtext
class Application:
def __init__(self):
# Create UI
self.window = tkinter.Tk()
self.window.geometry('800x600')
self.window.title("Tk-based event logger")
self.text_view = tkinter.scrolledtext.ScrolledText()
self.text_view.pack(fill=tkinter.BOTH, expand=1)
self.text_view.configure(state='disabled')
# Register event handlers
# See https://tcl.tk/man/tcl/TkCmd/tk_mac.html for list of
# macOS-specific commands
self.window.createcommand("::tk::mac::OpenDocument", self.open_document_handler)
self.window.createcommand("::tk::mac::LaunchURL", self.open_url_handler) # works with Tcl/Tk >= 8.6.10
def append_message(self, msg):
"""Append message to text view."""
self.text_view.configure(state='normal')
self.text_view.insert('end', msg + '\n')
self.text_view.configure(state='disabled')
def run(self):
"""Run the main loop."""
app.append_message("Application started!")
app.append_message(f"Args: {sys.argv[1:]}")
self.window.mainloop()
# Event handlers
def open_document_handler(self, *args):
app.append_message(f"Open document event: {args}")
def open_url_handler(self, *args):
app.append_message(f"Open URL event: {args}")
if __name__ == '__main__':
app = Application()
app.run()
The corresponding .spec file that builds
a onedir application bundle with a custom file association
(.pyi_tk) and a custom URL schema (pyi-tk://):
a = Analysis(['eventlogger_tk.py'])
pyz = PYZ(a.pure, a.zipped_data)
exe = EXE(
pyz,
a.scripts,
exclude_binaries=True,
name='eventlogger_tk',
debug=False,
bootloader_ignore_signals=False,
strip=False,
upx=False,
console=False,
argv_emulation=False, # unnecessary as app handles events
)
coll = COLLECT(
exe,
a.binaries,
a.zipfiles,
a.datas,
strip=False,
upx=False,
name='eventlogger_tk'
)
app = BUNDLE(
coll,
name='eventlogger_tk.app',
# Register custom protocol handler and custom file extension
info_plist={
'CFBundleURLTypes': [{
'CFBundleURLName': 'MyCustomUrlSchemaTk',
'CFBundleTypeRole': 'Viewer',
'CFBundleURLSchemes': ['pyi-tk'],
}],
'CFBundleDocumentTypes': [{
'CFBundleTypeName': 'MyCustomFileTypeTk',
'CFBundleTypeExtensions': [
'pyi_tk',
],
'CFBundleTypeRole': "Viewer",
}],
}
)
Once running, the application logs all received open file and open URL
requests. These are generated either by trying to open a file with
.pyi_tk extension using the UI, or using open command from
the terminal:
$ touch file1.pyi_tk file2.pyi_tk file3.pyi_tk file4.pyi_tk
$ open file1.pyi_tk
$ open file2.pyi_tk
$ open pyi-tk://test1
$ open pyi-tk://test2
$ open file3.pyi_tk file4.pyi_tk
Open event handling in a Qt-based GUI application¶
In Qt-based applications, open file and open URL requests are handled by installing application-wide event filter for QFileOpenEvent.
This event abstracts both open file and open URL request, with file
open requests having file:// URL schema. An event contains a
single file name or URL, so an open request containing multiple
targets generates corresponding number of QFileOpenEvent events.
Below is an example application and its corresponding .spec file:
# eventlogger_qt.py
import sys
import signal
from PySide2 import QtCore, QtWidgets
class Application(QtWidgets.QApplication):
"""
QtWidgets.QApplication with extra handling for macOS Open
document/URL events.
"""
openFileRequest = QtCore.Signal(QtCore.QUrl, name='openFileRequest')
def event(self, event):
if event.type() == QtCore.QEvent.FileOpen:
# Emit signal so that main window can handle the given URL.
# Or open a new application window for the file, or whatever
# is appropriate action for your application.
self.openFileRequest.emit(event.url())
return True
return super().event(event)
class MainWindow(QtWidgets.QMainWindow):
"""
Main window.
"""
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
self.resize(800, 600)
self.setWindowTitle("Qt-based event logger")
# Construct the UI
self.scroll_area = QtWidgets.QScrollArea()
self.scroll_area.setWidgetResizable(True)
self.setCentralWidget(self.scroll_area)
self.text_edit = QtWidgets.QTextEdit()
self.scroll_area.setWidget(self.text_edit)
self.text_edit.setReadOnly(True)
def append_message(self, msg):
"""
Append message to text view.
"""
self.text_edit.append(msg)
def handle_open_file_request(self, url):
self.append_message(f"Open request: {url.toString()}")
if __name__ == '__main__':
# Make Ctrl+C work
signal.signal(signal.SIGINT, signal.SIG_DFL)
app = Application(list(sys.argv))
window = MainWindow()
window.show()
window.append_message("Application started!")
window.append_message(f"Args: {sys.argv[1:]}")
app.openFileRequest.connect(window.handle_open_file_request)
app.exec_()
# eventlogger_qt.spec
a = Analysis(['eventlogger_qt.py'])
pyz = PYZ(a.pure, a.zipped_data)
exe = EXE(
pyz,
a.scripts,
exclude_binaries=True,
name='eventlogger_qt',
debug=False,
bootloader_ignore_signals=False,
strip=False,
upx=False,
console=False,
argv_emulation=False, # unnecessary as app handles events
)
coll = COLLECT(
exe,
a.binaries,
a.zipfiles,
a.datas,
strip=False,
upx=False,
name='eventlogger_qt'
)
app = BUNDLE(
coll,
name='eventlogger_qt.app',
# Register custom protocol handler and custom file extension
info_plist={
'CFBundleURLTypes': [{
'CFBundleURLName': 'MyCustomUrlSchemaQt',
'CFBundleTypeRole': 'Viewer',
'CFBundleURLSchemes': ['pyi-qt'],
}],
'CFBundleDocumentTypes': [{
'CFBundleTypeName': 'MyCustomFileTypeQt',
'CFBundleTypeExtensions': [
'pyi_qt',
],
'CFBundleTypeRole': "Viewer",
}],
}
)
The application behaves in the same way as its tkinter-based
counterpart, except that the associated file extension and URL
schema have been adjusted to prevent interference between the two
example applications.
Initial open event¶
This section contains notes about behavior of the initial open event received by appliation, as seen by the frozen python code (or the UI toolkit it uses).
When application is opened normally, this is done via Open Application
('oapp') event, which is the first event received by the application.
If application is opened in response to open document or open URL request
(i.e., it is not yet running when request is made), then the first
received event is 'odoc' or 'GURL', respectively.
In PyInstaller-frozen onefile bundles, the child process always
starts with 'oapp' event, regardless how the application was
started. This is because the child is always started “normally”, and
it is the parent who receives the actual opening event; if the parent
was opened with 'odoc' or 'GURL' event, then event is either
forwarded to child or converted to sys.argv that is passed to the
child, depending on whether argv emulation is enabled or not.
Therefore, in onefile mode, argv emulation has no direct effect
on the initial open event (as seen by the frozen python code), which is
always 'oapp'.
In onedir bundles, there application consists of single process,
which receives the events. Without argv emulation, the initial open
event (as seen by the frozen python code) may be either 'oapp',
'odoc', or 'GURL', depending on how application was started.
However, if argv emulation is enabled in a onedir bundle, its
processing of initial event leaves the event queue empty. The lack
of initial open event seems to cause segmentation fault with Tcl/Tk 8.6.11
and Homebrew Python 3.9.6 (#5581). As a work-around, the
bootloader attempts to submit an 'oapp' event to itself, so that
when the frozen python code inspects the event queue, it finds an
initial open event (i.e., 'oapp'). These potential side effects
of argv emulation on UI toolkits are the reason why we recommend
against using them together.