invoke start is failing

[jifox]

I used this configuration with an unmodified develompent/dev.env

export NAUTOBOT_VER=1.4.4
export PYTHON_VER=3.7

When executing invoke start the following traceback is displayed.

 *  Executing task in folder nautobot-plugindev: docker logs --tail 1000 -f 6c66bd41055f812cce1490f6d181d5f9203d5aaadbc891094775a71332a0a021 

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/nautobot/extras/plugins/utils.py", line 64, in load_plugin
    plugin = importlib.import_module(plugin_name)
  File "/usr/local/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 965, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'example_plugin'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/nautobot/core/runner/runner.py", line 120, in settings_callback
    "settings": settings,
  File "/usr/local/lib/python3.7/site-packages/nautobot/core/cli.py", line 176, in _configure_settings
    load_plugins(settings)
  File "/usr/local/lib/python3.7/site-packages/nautobot/extras/plugins/utils.py", line 54, in load_plugins
    load_plugin(plugin_name, settings)
  File "/usr/local/lib/python3.7/site-packages/nautobot/extras/plugins/utils.py", line 70, in load_plugin
    ) from err
nautobot.extras.plugins.exceptions.PluginNotFound: Unable to import plugin example_plugin: Module not found. Check that the plugin module has been installed within the correct Python environment.
⏳ Waiting on DB... (0s / 30s)
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/nautobot/extras/plugins/utils.py", line 64, in load_plugin
    plugin = importlib.import_module(plugin_name)
  File "/usr/local/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 965, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'example_plugin'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/nautobot/core/runner/runner.py", line 120, in settings_callback
    "settings": settings,
  File "/usr/local/lib/python3.7/site-packages/nautobot/core/cli.py", line 176, in _configure_settings
    load_plugins(settings)
  File "/usr/local/lib/python3.7/site-packages/nautobot/extras/plugins/utils.py", line 54, in load_plugins
    load_plugin(plugin_name, settings)
  File "/usr/local/lib/python3.7/site-packages/nautobot/extras/plugins/utils.py", line 70, in load_plugin
    ) from err
nautobot.extras.plugins.exceptions.PluginNotFound: Unable to import plugin example_plugin: Module not found. Check that the plugin module has been installed within the correct Python environment.
⏳ Waiting on DB... (3s / 30s)
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/nautobot/extras/plugins/utils.py", line 64, in load_plugin
    plugin = importlib.import_module(plugin_name)
  File "/usr/local/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 965, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'example_plugin'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/nautobot/core/runner/runner.py", line 120, in settings_callback
    "settings": settings,
  File "/usr/local/lib/python3.7/site-packages/nautobot/core/cli.py", line 176, in _configure_settings
    load_plugins(settings)
  File "/usr/local/lib/python3.7/site-packages/nautobot/extras/plugins/utils.py", line 54, in load_plugins
    load_plugin(plugin_name, settings)
  File "/usr/local/lib/python3.7/site-packages/nautobot/extras/plugins/utils.py", line 70, in load_plugin
    ) from err
nautobot.extras.plugins.exceptions.PluginNotFound: Unable to import plugin example_plugin: Module not found. Check that the plugin module has been installed within the correct Python environment.
⏳ Waiting on DB... (6s / 30s)
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/nautobot/extras/plugins/utils.py", line 64, in load_plugin
    plugin = importlib.import_module(plugin_name)
  File "/usr/local/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 965, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'example_plugin'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/nautobot/core/runner/runner.py", line 120, in settings_callback
    "settings": settings,
  File "/usr/local/lib/python3.7/site-packages/nautobot/core/cli.py", line 176, in _configure_settings
    load_plugins(settings)
  File "/usr/local/lib/python3.7/site-packages/nautobot/extras/plugins/utils.py", line 54, in load_plugins
    load_plugin(plugin_name, settings)
  File "/usr/local/lib/python3.7/site-packages/nautobot/extras/plugins/utils.py", line 70, in load_plugin
    ) from err
nautobot.extras.plugins.exceptions.PluginNotFound: Unable to import plugin example_plugin: Module not found. Check that the plugin module has been installed within the correct Python environment.
⏳ Waiting on DB... (9s / 30s)
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/nautobot/extras/plugins/utils.py", line 64, in load_plugin
    plugin = importlib.import_module(plugin_name)
  File "/usr/local/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 965, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'example_plugin'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/nautobot/core/runner/runner.py", line 120, in settings_callback
    "settings": settings,
  File "/usr/local/lib/python3.7/site-packages/nautobot/core/cli.py", line 176, in _configure_settings
    load_plugins(settings)
  File "/usr/local/lib/python3.7/site-packages/nautobot/extras/plugins/utils.py", line 54, in load_plugins
    load_plugin(plugin_name, settings)
  File "/usr/local/lib/python3.7/site-packages/nautobot/extras/plugins/utils.py", line 70, in load_plugin
    ) from err
nautobot.extras.plugins.exceptions.PluginNotFound: Unable to import plugin example_plugin: Module not found. Check that the plugin module has been installed within the correct Python environment.
⏳ Waiting on DB... (12s / 30s)
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/nautobot/extras/plugins/utils.py", line 64, in load_plugin
    plugin = importlib.import_module(plugin_name)
  File "/usr/local/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 965, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'example_plugin'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/nautobot/core/runner/runner.py", line 120, in settings_callback
    "settings": settings,
  File "/usr/local/lib/python3.7/site-packages/nautobot/core/cli.py", line 176, in _configure_settings
    load_plugins(settings)
  File "/usr/local/lib/python3.7/site-packages/nautobot/extras/plugins/utils.py", line 54, in load_plugins
    load_plugin(plugin_name, settings)
  File "/usr/local/lib/python3.7/site-packages/nautobot/extras/plugins/utils.py", line 70, in load_plugin
    ) from err
nautobot.extras.plugins.exceptions.PluginNotFound: Unable to import plugin example_plugin: Module not found. Check that the plugin module has been installed within the correct Python environment.
⏳ Waiting on DB... (15s / 30s)
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/nautobot/extras/plugins/utils.py", line 64, in load_plugin
    plugin = importlib.import_module(plugin_name)
  File "/usr/local/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 965, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'example_plugin'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/nautobot/core/runner/runner.py", line 120, in settings_callback
    "settings": settings,
  File "/usr/local/lib/python3.7/site-packages/nautobot/core/cli.py", line 176, in _configure_settings
    load_plugins(settings)
  File "/usr/local/lib/python3.7/site-packages/nautobot/extras/plugins/utils.py", line 54, in load_plugins
    load_plugin(plugin_name, settings)
  File "/usr/local/lib/python3.7/site-packages/nautobot/extras/plugins/utils.py", line 70, in load_plugin
    ) from err
nautobot.extras.plugins.exceptions.PluginNotFound: Unable to import plugin example_plugin: Module not found. Check that the plugin module has been installed within the correct Python environment.
⏳ Waiting on DB... (18s / 30s)
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/nautobot/extras/plugins/utils.py", line 64, in load_plugin
    plugin = importlib.import_module(plugin_name)
  File "/usr/local/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 965, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'example_plugin'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/nautobot/core/runner/runner.py", line 120, in settings_callback
    "settings": settings,
  File "/usr/local/lib/python3.7/site-packages/nautobot/core/cli.py", line 176, in _configure_settings
    load_plugins(settings)
  File "/usr/local/lib/python3.7/site-packages/nautobot/extras/plugins/utils.py", line 54, in load_plugins
    load_plugin(plugin_name, settings)
  File "/usr/local/lib/python3.7/site-packages/nautobot/extras/plugins/utils.py", line 70, in load_plugin
    ) from err
nautobot.extras.plugins.exceptions.PluginNotFound: Unable to import plugin example_plugin: Module not found. Check that the plugin module has been installed within the correct Python environment.
⏳ Waiting on DB... (21s / 30s)
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/nautobot/extras/plugins/utils.py", line 64, in load_plugin
    plugin = importlib.import_module(plugin_name)
  File "/usr/local/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 965, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'example_plugin'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/nautobot/core/runner/runner.py", line 120, in settings_callback
    "settings": settings,
  File "/usr/local/lib/python3.7/site-packages/nautobot/core/cli.py", line 176, in _configure_settings
    load_plugins(settings)
  File "/usr/local/lib/python3.7/site-packages/nautobot/extras/plugins/utils.py", line 54, in load_plugins
    load_plugin(plugin_name, settings)
  File "/usr/local/lib/python3.7/site-packages/nautobot/extras/plugins/utils.py", line 70, in load_plugin
    ) from err
nautobot.extras.plugins.exceptions.PluginNotFound: Unable to import plugin example_plugin: Module not found. Check that the plugin module has been installed within the correct Python environment.
⏳ Waiting on DB... (24s / 30s)
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/nautobot/extras/plugins/utils.py", line 64, in load_plugin
    plugin = importlib.import_module(plugin_name)
  File "/usr/local/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 965, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'example_plugin'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/nautobot/core/runner/runner.py", line 120, in settings_callback
    "settings": settings,
  File "/usr/local/lib/python3.7/site-packages/nautobot/core/cli.py", line 176, in _configure_settings
    load_plugins(settings)
  File "/usr/local/lib/python3.7/site-packages/nautobot/extras/plugins/utils.py", line 54, in load_plugins
    load_plugin(plugin_name, settings)
  File "/usr/local/lib/python3.7/site-packages/nautobot/extras/plugins/utils.py", line 70, in load_plugin
    ) from err
nautobot.extras.plugins.exceptions.PluginNotFound: Unable to import plugin example_plugin: Module not found. Check that the plugin module has been installed within the correct Python environment.
⏳ Waiting on DB... (27s / 30s)
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/nautobot/extras/plugins/utils.py", line 64, in load_plugin
    plugin = importlib.import_module(plugin_name)
  File "/usr/local/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 965, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'example_plugin'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/nautobot/core/runner/runner.py", line 120, in settings_callback
    "settings": settings,
  File "/usr/local/lib/python3.7/site-packages/nautobot/core/cli.py", line 176, in _configure_settings
    load_plugins(settings)
  File "/usr/local/lib/python3.7/site-packages/nautobot/extras/plugins/utils.py", line 54, in load_plugins
    load_plugin(plugin_name, settings)
  File "/usr/local/lib/python3.7/site-packages/nautobot/extras/plugins/utils.py", line 70, in load_plugin
    ) from err
nautobot.extras.plugins.exceptions.PluginNotFound: Unable to import plugin example_plugin: Module not found. Check that the plugin module has been installed within the correct Python environment.
❌ Waited 30s or more for the DB to become ready.

[jvanderaa]

Definitely interesting, really two errors that I'm seeing here. One is that it is all of a sudden waiting on the DB. Second is the unable to import the example plugin.

@bryanculver have you seen this before?

[jifox]

As a workaround I've mounted the nautobot_config.py in docker-compose and set the PLUGINS=[]

I was able to successfully execute invoke start after that.

[jvanderaa]

Good work around on it. I'll still pursue getting the plugins working, as that is our example to be able to test the SDK against it.

[joewesch]

I don't think this is necessarily an issue with pynautobot. This same thing happens if you run any Nautobot dev image. For example docker run --rm -it ghcr.io/nautobot/nautobot-dev:1.4.7-py3.10.

[joewesch]

I have reported this as a bug in the nautobot repo: nautobot/nautobot#2671