The runner project

The runner project

The runner project contains the greater part of the project, with the actual bindings for all the classes in uibase, and C++-python converters.

Forewords

The runner project uses boost::python for the interface, so you might want to get the documentation nearby if you intend to modify it.

In the following, I will start by explaining the organization of the project, detailing what the main 4 components (files) contain, and then explain how to do the most common things (adding a new plugin type, adding a new wrappers, etc.).

Project organization

The parts of the project you are most likely going to modify are the following:

• proxypluginwrappers: contains wrappers for the plugins.
• gamefeatureswrappers: contains wrappers for the game features.
• pythonrunner: contains the actual declaration of the mobase module with all the exposed classes.
• uibasewrappers: contains wrappers for some uibase classes that are neither plugins nor game features.

Note: A wrapper is a C++ class that implements a C++ interface and delegates virtual function calls to a python object. A wrapper is needed for any class that can be extended through python (plugins, game features, some uibase classes).

The runner projects also contains the following utility files:

• converters.h contains utility functions to easily create C++/Python conversion functions for Qt or standard classes. Some useful functions:
  • utils::register_qclass_converter to register a Qt class.
  • utils::register_qflags_converter to register a QFlags.
  • utils::register_sequence_container to register a sequence container (QList, std::vector, std::list, ...).
• tuple_helper.h and variant_helper.h contain converters for tuples and variants (works for both std and boost version). The register functions are in the boost::python namespace.
• error.h contains custom exceptions that you can throw. In particular, the PythonError, that will fetch the error message from python (does not always work... ).
• pythonwrapperutilities.h contains utility function that can be used when implementing wrappers.

Those are (should be?) fully documented, so I will not explain them in detail here. You can find examples for those in the 4 main files.

pythonrunner

The pythonrunner.cpp file contains the actual mobase module within BOOST_PYTHON_MODULE(mobase):

BOOST_PYTHON_MODULE(mobase)
{
  // I have no idea why this is required (or even if this is required).
  PyEval_InitThreads();

  // We need to import PyQt5 here, otherwize boost will fail to convert default
  // argument for Qt class.
  bpy::import("PyQt5.QtCore");
  bpy::import("PyQt5.QtWidgets");

  // Registering converters (see below).

  // Exposing classes (See below).

  // Register game features, see the "gamefeatureswrappers" section.
  registerGameFeaturesPythonConverters();
}

Note: bpy is a namespace alias for boost::python.

Registering converters

The beginning of the module declaration contains registration for all converters: Qt classes that are used between Python and C++, QFlags, containers, smart pointers, etc. If a function is exposed through boost::python, all the argument types and return type must have registered converters. If you have a std::function that has arguments, you also needs to register its arguments or return type. For instance, if we were to expose the following function:

std::vector<QString> mapToString(
    std::vector<std::tuple<QDateTime, QUrl>>,
    std::function<QString(std::set<int>, std::map<int, double>, QString)>
);

You would need to register the following:

// For QString, this is a special one, see FAQ at the end.
utils::register_qstring_converter();

// Register the Qt class:
utils::register_qclass_converter<QDateTime>();
utils::register_qclass_converter<QUrl>();

// Register the tuple:
bpy::register_tuple<std::tuple<QDateTime, QUrl>>();

// Register the containers:
bpy::register_sequence_container<std::vector<QString>>();
bpy::register_sequence_container<std::vector<std::tuple<QDateTime, QUrl>>>();
bpy::register_set_container<std::set<int>>();
bpy::register_associative_container<std::map<int, double>>();

// Register the function:
bpy::register_functor_converter<QString(std::set<int>, std::map<int, double>, QString)>();

You only need to register converters if those are not already registered: do not duplicate converter registration.

A few notes on converters:

• register_tuple will convert from any Python sequence of the right length and always to Python tuple.
• register_variant will convert from and to any of the Python types the variant contains (those must have registered converters). This is different from QVariant!.
• register_sequence_container will convert from any Python sequence, but always to Python list.
• register_set_container has the same behavior as register_sequence_container. It does not convert to Python set due to the differences between std::set and set (set is more like std::unordered_set in C++ than std::set).
• register_associative_container will convert from dict-like object and to Python dict.
• register_functor_converter will convert from any Python callable with the right number of arguments.
  • Only the number of arguments is checked when converting, not their types nor the return type (this is not possible without executing the function), so the conversion could succeed while the actual call might fail.
  • None python object are converted to default-constructed std::function.
  • register_functor_converter does not register a to-Python converter!

Checking for python sequences or dict is made using PySequence_Check and PyDict_Check, so any Python types with the proper requirements should work (e.g. defaultdict and OrderedDict are valid dictionaries).

Some warnings:

• These converters will not work with pointers (as arguments or return object), e.g. the following cannot be done:

std::vector<int>* myVec(std::set<Qstring> const*);

• Except for sequence containers, the order of elements is lost when converting.

The register_qclass_converter is different and will register both value and pointer conversions (it does not register value for non-copyable type).

If you need to register an enumeration with an associated Q_FLAGS declaration, you should expose the base enumeration through bpy::enum_ (see below) and then use register_qflags_converter on the actual QFlags, e.g.:

utils::register_qflags_converter<IPluginList::PluginStates>();

bpy::enum_<IPluginList::PluginState>("PluginState")
  .value("MISSING", IPluginList::STATE_MISSING)
  .value("INACTIVE", IPluginList::STATE_INACTIVE)
  .value("ACTIVE", IPluginList::STATE_ACTIVE)
  ;

Exposing classes

Classes, functions and enumerations are exposed using respectively bpy::class_, bpy::def and bpy::enum_. I will not go into details about everything here since the boost documentation for this should be sufficient.

Enumerations are exposed through bpy::enum_. Enumeration values should be ALL_UPPER_CASES to follow python convention and not be exported in the mobase module (see bpy::enum_::export_values).

bpy::enum_<MOBase::IPluginInstaller::EInstallResult>("InstallResult")
  .value("SUCCESS", MOBase::IPluginInstaller::RESULT_SUCCESS)
  .value("FAILED", MOBase::IPluginInstaller::RESULT_FAILED)
  .value("CANCELED", MOBase::IPluginInstaller::RESULT_CANCELED)
  .value("MANUAL_REQUESTED", MOBase::IPluginInstaller::RESULT_MANUALREQUESTED)
  .value("NOT_ATTEMPTED", MOBase::IPluginInstaller::RESULT_NOTATTEMPTED)

  // .export_values() - Don't do that, unless you are in a class scope.
  ;

You can export enumeration values into a class for inner enumerations. This allows typing IFileTree.FILE instead of IFileTree.FileType.FILE. You can find examples of this in the FileTreeEntry and IFileTree classes.

Classes are exposed through bpy::class_. If your class needs a wrapper (i.e. it needs to be extended in python), you will expose the wrapper rather than the interface.

There are many examples in the actual code so I will not list everything here.

• If you expose a wrapper, you can mark pure-virtual functions with bpy::pure_virtual. You can provide default implementation for those if you want (see the wrapper section).
  • Currently, the only default implementation are for retro-compatibility with existing plugins. It should be pretty rare to need default implementations.
  • bpy::pure_virtual is not mandatory. Its only effect is changing the error message you get (in Python) if you call a non-implemented pure-virtual method.
• If you expose a wrapper for a class that can be extended in python, and the original interface has protected member-functions or variables, you need to bring those in the public scope of the wrapper to expose them to python. By convention, protected methods should start with a _, e.g. _parentWidget().
• Some classes need to expose Qt-specific members in python. This cannot be done using bpy::bases since Qt classes are not exposed through boost::python. You can use the Q_DELEGATE for those (see FAQ).
• If you expose a (member-)function that has in-out parameters (a non-const reference or pointer), you might need to modify the actual signature (see below).

Be careful when returning objects to pythonr, specifically regarding the return_value_policy. There are many examples in pythonrunner.cpp. In particular QWidget* uses a bpy::return_by_value policy even if we actually reference existing objects. This is due to how we interface with sip.

Here is a mini-example:

bpy::class_<MyPluginWrapper, bpy::bases<IPlugin>, boost::noncopyable>("MyPlugin")

  // Define a method that inheriting classes need to implement:
  .def("methodToImplement", bpy::pure_virtual(&MyPluginWrapper::methodToImplement))

  // A basic method that does not require an implementation and returns the global IOrganizer, so
  // we use reference_existing_object:
  .def("organizer", &MyPluginWrapper::organizer,
      bpy::return_value_policy<bpy::reference_existing_object>())

  // Expose a protected member. MyPluginWrapper should have brough `protectedMethod`
  // in the public scope.
  .def("_protectedMethod", &MyPluginWrapper::protectedMethod)

  // Returning a QWidget*, we need to use return_by_value.
  .def("_parentWiget", &MyPluginWrapper::parentWidget,
      bpy::return_value_policy<bpy::return_by_value>())

  // Delegate QObject stuff, e.g. if MyPlugin defines Qt signals.
  Q_DELEGATE(MyPlugin, QObject, "_object")

If a function takes, e.g., a int& to be modified, you need to modify the signature. The usual way of exposing such functions to python is to make the function returns the int instead of modifying it. If the function already returns something, you can transform it to a tuple, or allow returning either the original return type (if the int& was not modified) or a tuple.

Note: If the reference argument can be modified in python (e.g., the argument does not need to be re-assigned), you can pass it to python using boost::ref.

A full example of this is the IPluginInstallerSimple::install method, you can check its implementation. Below is a mini-example:

// We want to expose: double Foo::bar(QString, int&) const;

// If Foo needs to be extended in Python, we need a wrapper:
class FooWrapper: /* see below */ {
public:
  virtual double bar(QString q, int& i) const override {
    // We will allow Python method to return either a double, if i was not modified, or both
    // a double and the new value for i - Do not forget to register converters for both the
    // tuple and the variant:
    using return_type = std::variant<double, std::tuple<double, int>>;

    // We call the python method (see the proxypluginwrappers section):
    auto result = basicWrapperFunctionImplementation<FooWrapper, return_type>(this, "bar", q, i);

    // We use std::visit and update i (if modified) and return d:
    return std::visit([&](auto const& t) {
      using type = std::decay_t<decltype(t)>;

      // The python function returned only d, so i is not modified:
      if constexpr (std::is_same_v<type, double>) {
        return t;
      }
      // The python function returned (d, i):
      else if constexpr (std::is_same_v<type, std::tuple<double, int>>) {
        // Retrieve i:
        i = std::get<1>(t);
        return std::get<0>(t);
      }
    }, result);
  }
};

// When exposing the bar, we need to also return a tuple:
bpy::class_<FooWrapper>("Foo")
  // We use a lambda converted to a function-pointer (+) - If this was not a member function, we would
  // not have the first argument:
  .def("bar", +[](FooWrapper *foo, QString q, int& i) {
     // Call the original foo:
    double d = foo->bar(q, i);

    // Return a tuple containing both the original return value (d) and the argument (i):
    return std::make_tuple(d, i);
  })
  ;

proxypluginwrappers

These two files contains wrapper for all the plugins defined in uibase. Since wrapper needs to delegate all methods, even the ones from the parent IPlugin, a utility macro COMMON_I_PLUGIN_WRAPPER_DECLARATIONS is provided.

Here is a typical wrapper declaration for a plugin (in proxypluginwrappers.h):

// The class should inherit the actual plugin type and the corresponding wrapper. If
// the actual plugin type does not inherit MOBase::IPlugin, the wrapper should also
// inherit MOBase::IPlugin (see IPluginDiagnoseWrapper).
class IPluginShinyWrapper:
    public MOBase::IPluginShiny,
    public boost::python::wrapper<MOBase::IPluginShiny> {

  // Add Qt declaration for the plugin:
  Q_OBJECT
  Q_INTERFACES(MOBase::IPlugin MOBase::IPluginShiny)
  // Add declaration for common plugin methods:
  COMMON_I_PLUGIN_WRAPPER_DECLARATIONS

public:
  // Add a static className, for logging purpose:
  static constexpr const char* className = "IPluginShinyWrapper";

  // Bring get_override:
  using boost::python::wrapper<MOBase::IPluginShiny>::get_override;

  // If the parent plugin has constructors, bring them here:
  using IPluginShiny::IPluginShiny;

  // If the parent plugin has protected methods, bring them here (required
  // to be able to expose them with bpy::class_):
  using IPluginShiny::superProtectedMethod;

  // Add implementation for all pure-virtual methods:
  virtual int isShiny() const override;
  virtual void darken() override;
};

And here is the typical definitions (in proxypluginwrappers.cpp):

/// IPluginShiny Wrapper

// Define the common methods:
COMMON_I_PLUGIN_WRAPPER_DEFINITIONS(IPluginShiny)

// Define the overriden methods. The `pythonwrapperutilities.h` header defines
// multiple utility functions so you should use them instead of raw `get_override`
// for better exception handling.
int IPluginShinyWrapper::isSiny() const {
    return basicWrapperFunctionImplementation<IPluginShinyWrapper, int>(
		    this, "isShiny");
}

void IPluginShinyWrapper::darken() {
    basicWrapperFunctionImplementation<IPluginShinyWrapper, void>(this);
}

There are (currently) 4 functions available in pythonwrapperutilities.h:

• The first basicWrapperFunctionImplementation overload is the basic one that will call the python method and try to extract an object of the given return type. If the python method is not found, a MissingImplementation exception is thrown, and if an error occurs, PythonError or UnknownException are thrown.
• The second overload is similar except that it takes a reference to a bpy::object in which the result of get_override will be stored. This is very useful if you want to keep the returned python object alive. See the uibasewrappers section or the FAQ for more details on why you need this.
• Two basicWrapperFunctionImplementationWithDefault overloads are provided. Those can be used similarly to basicWrapperFunctionImplementation except that instead of throwing a MissingImplementation if the python method is not found, they will call a default method that you can provide.
  • The two overloads (Wrapper * and Wrapper * const) are provided because the default member function can be const-qualified. It does not really matter for the other functions in this header since get_override does not care about const-qualification.

gamefeatureswrappers

These two files contains wrappers for the game features. Note that unlike proxypluginwrappers, these also contain a registerGameFeaturesPythonConverters that register the actual wrappers.

You can modify existing wrapper as you would modify a wrapper for a plugin but if you add a new game feature, you need to add it to the MpGameFeaturesList list at the top of gamefeatureswrapper.h. This is required to get proper extraction from and to the the map of game features of the IPluginGame plugins.

uibasewrappers

This file contains a few wrappers that are neither plugins nor game features. If you use one of the existing wrapper in this file (or if you create a new one and use it), you have to be careful: you need to keep the initial Python object alive to be able to delegate calls to python.

This issue is not present with plugins or game features because plugins are kept alive in PythonRunner::m_PythonObjects and game features are hold by the actual game plugin.

If you do the following in C++:

// ISaveGame is wrapped using ISaveGameWrapper in uibasewrappers.h
ISaveGame* getSaveGame() {
  // Simply call the python function:
  return basicWrapperFunctionImplementation<MyWrapper, ISaveGame*>(this, "getSaveGame");
}

And in python you implement getSaveGame:

def getSaveGame(self):
  # We return a new object!
  return MySaveGameImplementation()

The code will most likely not work. Why? Because as soon as you get out of getSaveGame, you lose the initial Python object that holds the ISaveGameWrapper*. You can still use the returned object, but any attempt to access the MySaveGameImplementation python object will fail. In particular, trying to use get_override to call a method overriden in MySaveGameImplementation will result in a missing implementation exception.

The only way to prevent this is to store the actual bpy::object somewhere, e.g., in a member variable of the C++ wrapper. See for instance the implementation of SaveGameInfoWrapper::getSaveGameInfo or SaveGameInfoWrapper::getSaveGameWidget.

Modifying or adding plugin types

You can modify existing plugin wrappers without too much worry. As with C++ plugin interfaces, it is a good idea to be backward compatible for plugin types that are often used. In particular, if a new methods has been added, it is a good practice to add a default implementation for it in order to allow existing plugins to still work (unless the method is now mandatory).

Warning: Even if the C++ interface has a default-implementation of a virtual method, it is necessary to use the basicWrapperFunctionImplementationWithDefault:

class IPluginShinyWrapper: /* ... */ {
public:

    // Adding a method that has a default implementation in the parent interface:
    virtual int theNewMethod() override {
        return basicWrapperFunctionImplementationWithDefault<IPluginShinyWrapper, int>(
            this, &IPluginShinyWrapper::theNewMethod_Default);
    }

    // By convention, I add _Default to default method:
    int theNewMethod_Default() {
        // Simply call the parent method:
        return IPluginShiny::theNewMethod();
    }
};

// And when exposing the class:
bpy::class_<IPluginShinyWrapper>("IPluginShiny")
  .def'("theNewMethod", &IPluginShiny::theNewMethod, &IPluginShinyWrapper::theNewMethod_Default)
  ;

If you add a new plugin types, it is necessary to add it to the list of tried plugins in PythonRunner::instantiate:

appendIfInstance<IPluginShiny>(pluginObj, interfaceList);

Note: If the plugin does not inherit IPlugin, you need to check appendIfInstance with the wrapper.

Modifying or adding game features

Modifying game features is similar to modifying plugins. Try to avoid breaking compatibility with existing features and provide default with new methods.

If you add a new game feature, you need to add it to the MpGameFeaturesList list at the top of gamefeatureswrappers.h. This list is used in various places to perform conversion from and to python.