Relative imports in Python 3

unfortunately, this module needs to be inside the package, and it also
needs to be runnable as a script, sometimes. Any idea how I could
achieve that?

Its quite common to have a layout like this…

main.py
mypackage/
    __init__.py
    mymodule.py
    myothermodule.py

…with a mymodule.py like this…

#!/usr/bin/env python3

# Exported function
def as_int(a):
    return int(a)

# Test function for module  
def _test():
    assert as_int(1) == 1

if __name__ == __main__:
    _test()

…a myothermodule.py like this…

#!/usr/bin/env python3

from .mymodule import as_int

# Exported function
def add(a, b):
    return as_int(a) + as_int(b)

# Test function for module  
def _test():
    assert add(1, 1) == 2

if __name__ == __main__:
    _test()

…and a main.py like this…

#!/usr/bin/env python3

from mypackage.myothermodule import add

def main():
    print(add(1, 1))

if __name__ == __main__:
    main()

…which works fine when you run main.py or mypackage/mymodule.py, but fails with mypackage/myothermodule.py, due to the relative import…

from .mymodule import as_int

The way youre supposed to run it is…

python3 -m mypackage.myothermodule

…but its somewhat verbose, and doesnt mix well with a shebang line like #!/usr/bin/env python3.

The simplest fix for this case, assuming the name mymodule is globally unique, would be to avoid using relative imports, and just use…

from mymodule import as_int

…although, if its not unique, or your package structure is more complex, youll need to include the directory containing your package directory in PYTHONPATH, and do it like this…

from mypackage.mymodule import as_int

…or if you want it to work out of the box, you can frob the PYTHONPATH in code first with this…

import sys
import os

SCRIPT_DIR = os.path.dirname(os.path.abspath(__file__))
sys.path.append(os.path.dirname(SCRIPT_DIR))

from mypackage.mymodule import as_int

Its kind of a pain, but theres a clue as to why in an email written by a certain Guido van Rossum…

Im -1 on this and on any other proposed twiddlings of the __main__
machinery. The only use case seems to be running scripts that happen
to be living inside a modules directory, which Ive always seen as an
antipattern. To make me change my mind youd have to convince me that
it isnt.

Whether running scripts inside a package is an antipattern or not is subjective, but personally I find it really useful in a package I have which contains some custom wxPython widgets, so I can run the script for any of the source files to display a wx.Frame containing only that widget for testing purposes.

Explanation

From PEP 328

Relative imports use a modules __name__ attribute to determine that
modules position in the package hierarchy. If the modules name does
not contain any package information (e.g. it is set to __main__)
then relative imports are resolved as if the module were a top level
module, regardless of where the module is actually located on the file
system.

At some point PEP 338 conflicted with PEP 328:

… relative imports rely on __name__ to determine the current
modules position in the package hierarchy. In a main module, the
value of __name__ is always __main__, so explicit relative imports
will always fail (as they only work for a module inside a package)

and to address the issue, PEP 366 introduced the top level variable __package__:

By adding a new module level attribute, this PEP allows relative
imports to work automatically if the module is executed using the -m
switch. A small amount of boilerplate in the module itself will allow
the relative imports to work when the file is executed by name. […] When it [the attribute] is present, relative imports will be based on this attribute
rather than the module __name__ attribute. […] When the main module is specified by its filename, then the __package__ attribute will be set to None. […] When the import system encounters an explicit relative import in a
module without __package__ set (or with it set to None), it will
calculate and store the correct value (__name__.rpartition(.)[0]
for normal modules and __name__ for package initialisation modules)

(emphasis mine)

If the __name__ is __main__, __name__.rpartition(.)[0] returns empty string. This is why theres empty string literal in the error description:

SystemError: Parent module  not loaded, cannot perform relative import

The relevant part of the CPythons PyImport_ImportModuleLevelObject function:

if (PyDict_GetItem(interp->modules, package) == NULL) {
    PyErr_Format(PyExc_SystemError,
            Parent module %R not loaded, cannot perform relative 
            import, package);
    goto error;
}

CPython raises this exception if it was unable to find package (the name of the package) in interp->modules (accessible as sys.modules). Since sys.modules is a dictionary that maps module names to modules which have already been loaded, its now clear that the parent module must be explicitly absolute-imported before performing relative import.

Note: The patch from the issue 18018 has added another if block, which will be executed before the code above:

if (PyUnicode_CompareWithASCIIString(package, ) == 0) {
    PyErr_SetString(PyExc_ImportError,
            attempted relative import with no known parent package);
    goto error;
} /* else if (PyDict_GetItem(interp->modules, package) == NULL) {
    ...
*/

If package (same as above) is empty string, the error message will be

ImportError: attempted relative import with no known parent package

However, you will only see this in Python 3.6 or newer.

Solution #1: Run your script using -m

Consider a directory (which is a Python package):

.
├── package
│   ├── __init__.py
│   ├── module.py
│   └── standalone.py

All of the files in package begin with the same 2 lines of code:

from pathlib import Path
print(Running if __name__ == __main__ else Importing, Path(__file__).resolve())

Im including these two lines only to make the order of operations obvious. We can ignore them completely, since they dont affect the execution.

__init__.py and module.py contain only those two lines (i.e., they are effectively empty).

standalone.py additionally attempts to import module.py via relative import:

from . import module  # explicit relative import

Were well aware that /path/to/python/interpreter package/standalone.py will fail. However, we can run the module with the -m command line option that will search sys.path for the named module and execute its contents as the __main__ module:

[email protected]:~$ python3 -i -m package.standalone
Importing /home/vaultah/package/__init__.py
Running /home/vaultah/package/standalone.py
Importing /home/vaultah/package/module.py
>>> __file__
/home/vaultah/package/standalone.py
>>> __package__
package
>>> # The __package__ has been correctly set and module.py has been imported.
... # Whats inside sys.modules?
... import sys
>>> sys.modules[__main__]
<module package.standalone from /home/vaultah/package/standalone.py>
>>> sys.modules[package.module]
<module package.module from /home/vaultah/package/module.py>
>>> sys.modules[package]
<module package from /home/vaultah/package/__init__.py>

-m does all the importing stuff for you and automatically sets __package__, but you can do that yourself in the

Solution #2: Set __package__ manually

Please treat it as a proof of concept rather than an actual solution. It isnt well-suited for use in real-world code.

PEP 366 has a workaround to this problem, however, its incomplete, because setting __package__ alone is not enough. Youre going to need to import at least N preceding packages in the module hierarchy, where N is the number of parent directories (relative to the directory of the script) that will be searched for the module being imported.

Thus,

1. Add the parent directory of the Nth predecessor of the current module to sys.path

2. Remove the current files directory from sys.path

3. Import the parent module of the current module using its fully-qualified name

4. Set __package__ to the fully-qualified name from 2

5. Perform the relative import

Ill borrow files from the Solution #1 and add some more subpackages:

package
├── __init__.py
├── module.py
└── subpackage
    ├── __init__.py
    └── subsubpackage
        ├── __init__.py
        └── standalone.py

This time standalone.py will import module.py from the package package using the following relative import

from ... import module  # N = 3

Well need to precede that line with the boilerplate code, to make it work.

import sys
from pathlib import Path

if __name__ == __main__ and __package__ is None:
    file = Path(__file__).resolve()
    parent, top = file.parent, file.parents[3]

    sys.path.append(str(top))
    try:
        sys.path.remove(str(parent))
    except ValueError: # Already removed
        pass

    import package.subpackage.subsubpackage
    __package__ = package.subpackage.subsubpackage

from ... import module # N = 3

It allows us to execute standalone.py by filename:

[email protected]:~$ python3 package/subpackage/subsubpackage/standalone.py
Running /home/vaultah/package/subpackage/subsubpackage/standalone.py
Importing /home/vaultah/package/__init__.py
Importing /home/vaultah/package/subpackage/__init__.py
Importing /home/vaultah/package/subpackage/subsubpackage/__init__.py
Importing /home/vaultah/package/module.py

A more general solution wrapped in a function can be found here. Example usage:

if __name__ == __main__ and __package__ is None:
    import_parents(level=3) # N = 3

from ... import module
from ...module.submodule import thing

Solution #3: Use absolute imports and setuptools

The steps are –

1. Replace explicit relative imports with equivalent absolute imports

2. Install package to make it importable

For instance, the directory structure may be as follows

.
├── project
│   ├── package
│   │   ├── __init__.py
│   │   ├── module.py
│   │   └── standalone.py
│   └── setup.py

where setup.py is

from setuptools import setup, find_packages
setup(
    name = your_package_name,
    packages = find_packages(),
)

The rest of the files were borrowed from the Solution #1.

Installation will allow you to import the package regardless of your working directory (assuming therell be no naming issues).

We can modify standalone.py to use this advantage (step 1):

from package import module  # absolute import

Change your working directory to project and run /path/to/python/interpreter setup.py install --user (--user installs the package in your site-packages directory) (step 2):

[email protected]:~$ cd project
[email protected]:~/project$ python3 setup.py install --user

Lets verify that its now possible to run standalone.py as a script:

[email protected]:~/project$ python3 -i package/standalone.py
Running /home/vaultah/project/package/standalone.py
Importing /home/vaultah/.local/lib/python3.6/site-packages/your_package_name-0.0.0-py3.6.egg/package/__init__.py
Importing /home/vaultah/.local/lib/python3.6/site-packages/your_package_name-0.0.0-py3.6.egg/package/module.py
>>> module
<module package.module from /home/vaultah/.local/lib/python3.6/site-packages/your_package_name-0.0.0-py3.6.egg/package/module.py>
>>> import sys
>>> sys.modules[package]
<module package from /home/vaultah/.local/lib/python3.6/site-packages/your_package_name-0.0.0-py3.6.egg/package/__init__.py>
>>> sys.modules[package.module]
<module package.module from /home/vaultah/.local/lib/python3.6/site-packages/your_package_name-0.0.0-py3.6.egg/package/module.py>

Note: If you decide to go down this route, youd be better off using virtual environments to install packages in isolation.

Solution #4: Use absolute imports and some boilerplate code

Frankly, the installation is not necessary – you could add some boilerplate code to your script to make absolute imports work.

Im going to borrow files from Solution #1 and change standalone.py:

1. Add the parent directory of package to sys.path before attempting to import anything from package using absolute imports:

   import sys
   from pathlib import Path # if you havent already done so
   file = Path(__file__).resolve()
   parent, root = file.parent, file.parents[1]
   sys.path.append(str(root))
   
   # Additionally remove the current files directory from sys.path
   try:
       sys.path.remove(str(parent))
   except ValueError: # Already removed
       pass
   

2. Replace the relative import by the absolute import:

   from package import module  # absolute import
   

standalone.py runs without problems:

[email protected]:~$ python3 -i package/standalone.py
Running /home/vaultah/package/standalone.py
Importing /home/vaultah/package/__init__.py
Importing /home/vaultah/package/module.py
>>> module
<module package.module from /home/vaultah/package/module.py>
>>> import sys
>>> sys.modules[package]
<module package from /home/vaultah/package/__init__.py>
>>> sys.modules[package.module]
<module package.module from /home/vaultah/package/module.py>

I feel that I should warn you: try not to do this, especially if your project has a complex structure.

---

As a side note, PEP 8 recommends the use of absolute imports, but states that in some scenarios explicit relative imports are acceptable:

Absolute imports are recommended, as they are usually more readable
and tend to be better behaved (or at least give better error
messages). […] However, explicit relative imports are an acceptable
alternative to absolute imports, especially when dealing with complex
package layouts where using absolute imports would be unnecessarily
verbose.

Relative imports in Python 3

Put this inside your packages __init__.py file:

# For relative imports to work in Python 3.6
import os, sys; sys.path.append(os.path.dirname(os.path.realpath(__file__)))

Assuming your package is like this:

├── project
│   ├── package
│   │   ├── __init__.py
│   │   ├── module1.py
│   │   └── module2.py
│   └── setup.py

Now use regular imports in you package, like:

# in module2.py
from module1 import class1

This works in both python 2 and 3.