Embedding Python in Objective-C: Part 2

Two years ago, I wrote about how I embedded Python in Objective-C through PyObjC. It worked well and everything was fine. The only problem I had with PyObjC was that I felt that such a massive library (pyobjc-core has over 5,000 Python SLOC and over 55,000 Objective-C SLOC, not counting tests) was overkill for what I used of it. The whole library was a black box to me and that made me a bit uneasy.

Then, a few weeks ago, I started getting report of weird crashes which seemed related to 64-bit and the presence on the system of some 3rd party app with specific settings. I could never reproduce the damn crashes on my computer, but they came around exactly at the same moment as dupeGuru 3.3.0 which updated PyObjC to fix another 64-bit related crash, so I could only assume that PyObjC (and/or libffi, which is used by PyObjC) was the cause. The problem was that I didn't know where to start the bug hunt because of my inability to work within PyObjC's codebase.

So, out of curiosity for its feasability and to know if it was going to fix my crashes, I started working on ObjP. Instead of dynamically dispatching function call between Python and Objective-C, I would generate source code for static wrappers which would simply use Python's C API. Early tests were encouraging because it fixed the crashes on my users' machines, so I continued working on it and a few days ago I completed dupeGuru's conversion from PyObjC to ObjP and published dupeGuru 3.3.2. So far I haven't heard from crashes, so I guess it works well.

What is ObjP?

ObjP is a code generation utility. It generates Objective-C code that bridges Python and Objective-C together using Python's (3.2+) C API. There are two types of wrappers it can generate: Python-->Objective-C wrappers (p2o), or Objective-C-->Python wrappers (o2p). To generate an o2p wrapper class, it needs to be supplied with a Python class with proper annotations like this:

class MyClass:
    def hello_(self, name: str) -> str:
        return "Hello {}".format(name)

This would generate an Objective-C class with this interface:

@interface MyClass : NSObject {}
- (NSString *)hello:(NSString *)name;
@end

For p2o, you give it an Objective-C header (or even an annotated Python class that describes the Objective-C interface) and it generates a Python extension module that wraps that class.

The number of data structure it supports is limited (bool, int, str, list and dict) but I don't need more, and it works rather well.

How to use ObjP?

The technicalities are a bit cumbersome to lay out, so I'm going to stay at a higher level of explanations, but you can always look at dupeGuru's code for a real life example. For a simpler example, there's guiskel which I converted to ObjP.

The bridging unit

So if you want to create an Objective-C app that embeds Python using ObjP, you're going to have to start by creating a python bridging unit very similar to what you'd do if you used PyObjC. The difference is that instead of using @signature decorators, you use Python annotations (a feature introduced in Python 3). A small difference is that in PyObjC, as long as arguments and return values were NSObject subclasses, you didn't have to specify a signature, but with ObjP you can't do that because it requires you to differentiate between str, list and dict.

Generating o2p code

Once you have that bridging unit, you can generate your code using objp.o2p. Here's an example:

import objp.o2p
from bridgeunit import Class1, Class2
objp.o2p.generate_objc_code(Class1, 'autogen')
objp.o2p.generate_objc_code(Class2, 'autogen')

Now you're going to have Class1.h|m and Class2.h|m files in the autogen folder. Here's an example of what one of your method wrappers could look like:

- (BOOL)canMoveRows:(NSArray *)rows to:(NSInteger)position
{
    OBJP_LOCKGIL;
    PyObject *pFunc = PyObject_GetAttrString(py, "canMoveRows_to_");
    OBJP_ERRCHECK(pFunc);
    PyObject *prows = ObjP_list_o2p(rows);
    PyObject *pposition = ObjP_int_o2p(position);
    PyObject *pResult = PyObject_CallFunctionObjArgs(pFunc, prows, pposition, NULL);
    Py_DECREF(prows);
    Py_DECREF(pposition);
    OBJP_ERRCHECK(pResult);
    Py_DECREF(pFunc);
    BOOL result = ObjP_bool_p2o(pResult);
    Py_DECREF(pResult);
    OBJP_UNLOCKGIL;
    return result;
}

Now, you can't use them in your Objective-C app yet because it didn't initialize Python. Yup, you have to embed Python in your XCode project.

Python in XCode

The first thing you need is a python dylib that can be embedded. For this, you need to copy your own python dylib and update its install name with install_name_tool to something like "@rpath/Python". This way, XCode will know where to look for the Python dylib it linked to at compile time when comes the time to run the app on your user's machine. I created a little utility function in pluginbuilder, copy_embeddable_python_dylib(), to take care of that whole @rpath thing.

Once you have that dylib, you can add it in your XCode project, make sure that you link to it and that it's copied in your Frameworks resource subfolder. You're also going to have to add "@executable_path/../Frameworks" to your project's "Runpath Search Paths".

Next is Python headers. You're going to have to make sure that XCode can find Python's headers. Personally, I use pluginbuilder's get_python_header_folder() to create a symlink in my project folder and add that symlink to my XCode project's "Header Search Paths".

Collecting dependencies

For your app to run on another machine, you're going to have to collect dependencies for your Python code and bundle it with your app. You can use pluginbuilder for that. Normally, it creates the whole ".plugin" shebang, but there's collect_dependencies() that you can use instead of build_plugin() that will just perform the dependency collection phase. Put this all in a folder (I call mine "py") and add that folder to your XCode project's "Resources" folder.

Initializing Python

Before you can instantiate Class1 and Class2 in your code, you still need to initialize Python. I do this in my main.m unit, just before NSApplicationMain. This is what my unit looks like:

#import <Cocoa/Cocoa.h>
#import <Python.h>
#import <wchar.h>
#import <locale.h>

int main(int argc, char *argv[])
{
    NSAutoreleasePool *pool = [[NSAutoreleasePool alloc] init];
    /* We have to set the locate to UTF8 for mbstowcs() to correctly convert non-ascii chars in paths */
    setlocale(LC_ALL, "en_US.UTF-8");
    NSString *respath = [[NSBundle mainBundle] resourcePath];
    NSString *mainpy = [respath stringByAppendingPathComponent:@"bridgeunit.py"];
    wchar_t wPythonPath[PATH_MAX+1];
    NSString *pypath = [respath stringByAppendingPathComponent:@"py"];
    mbstowcs(wPythonPath, [pypath fileSystemRepresentation], PATH_MAX+1);
    Py_SetPath(wPythonPath);
    Py_SetPythonHome(wPythonPath);
    Py_Initialize();
    PyEval_InitThreads();
    PyGILState_STATE gilState = PyGILState_Ensure();
    FILE* fp = fopen([mainpy UTF8String], "r");
    PyRun_SimpleFile(fp, [mainpy UTF8String]);
    fclose(fp);
    PyGILState_Release(gilState);
    if (gilState == PyGILState_LOCKED) {
        PyThreadState_Swap(NULL);
        PyEval_ReleaseLock();
    }
    int result = NSApplicationMain(argc,  (const char **) argv);
    Py_Finalize();
    [pool release];
    return result;
}

Besides the basic python initialization calls, you can notice the use of Py_SetPath, which allows us to point Python to our collected dependencies which we've copied in our application bundle. There's also the setlocale thing which is a workaround to a bug causing Python's initialization to fail if the path from which we launch our application contains a non-ascii character.

Reaping the rewards

After you've done all this, you can finally access your Python code through your ObjP generated wrappers. Just call [[Class1 alloc] init] and use the class like any other Objective-C class. And there's no hocus pocus. If you want, you can look at Class1.m and see exactly what is being done when you call a method.

Beyond the basics

In my own code, I stretch ObjP's capabilities a bit further than what I've exposed there (it involves passing PyObject references around and stuff), but that's gonna be for another article.