Refined Object Model and Python Bindings

May 11, 2020

Hi everyone,

In the past two months, as I was trying to analyze and share information about the state of the current pandemic, I've started to (re-)learn two well-known scientific visualization libraries: d3 in Javascript, and matplotlib in Python.

Learning new libraries is always a good source of inspiration when you're writing a library yourself, and it prompted me to finally take the time to properly fix an annoying problem I was having with the Python bindings of VGC: memory management of tree data structures.

Description of the change (a bit technical)

In programming terminology, a "tree" simply refers to a hierarchy of objects, like an organization of folders in your hard drive, or a family tree. There is a "root" object (the unique ancestor of all objects in the tree), and this root can have one or more children, which can themselves have one or more children, etc.

None of this is really complicated to implement in C++ alone, or in Python alone, but it is a bit tricky in our case where the implementation is in C++ but users can create, access, and modify the tree in Python too. The reason it is tricky is that Python and C++ have fundamentally different ways to manage memory: Python has what is called a "garbage collector", while C++ has not. A garbage collector is a mechanism which automatically releases the memory taken by an object once it isn't accessible anymore, that is, when no variables point to the object anymore.

The problem in our case is that because the data structure is implemented in C++, this means that Python isn't aware of the internal references of the tree, which doesn't play well with its garbage collector. The previous solution in VGC was to give Python "shared ownership" of all objects in the tree, which is quite a standard solution when writing Python bindings of C++ code, but this had the side effects that the C++ parent of each object isn't anymore its "unique owner", which creates all sorts of problems, such as making it harder or impossible to enforce that some desirable properties stay true (= "invariants" in programming terminology).

The new solution which I have just finished to implement is a more refined memory ownership model, where Python is only given a "weak reference" to objects in the tree, which doesn't guarantee that the referenced object itself stays alive, but guarantees instead that the root of this object stays alive. If a Python script attempts to access an object which doesn't exist anymore, then a Python exception is raised.

This was quite a significant change, which took a long time to design, and around 1,500 lines of code were added or modified. It doesn't change anything to the user interface, but it was one of the few important pieces of architecture that was yet to be done, and I'm glad this is over.

What's next?

Up to May 31, I will actually spend 100% of my time making a short video explaining in an intuitive way some of the technology behind VPaint/VGC, which should be featured in the end credits of the Hand Drawn Documentary (if all goes well). Hopefully, this might help a few animators to be aware of this project :-)

I should then go back to developing VGC in June.

As always, thanks a lot for your support!

Boris

Become a sponsor

Excited about VGC? Join the sponsorship program to help us make it happen!

Indeed, we made the choice to be funded by future users like yourself rather than business investors, to ensure that your interests always come first.

How does it work?

For only $1 per month (Bronze Sponsor), you get instant access to all exclusive news as well as all alpha and beta versions of VGC. This allows you to try out VGC right now, which is a unique opportunity to provide early feedback and shape the tools the way you want them to be. Welcome to the team!

For $3 per month (Silver Sponsor), you also get a perpetual license key for the upcoming VGC Illustration 1.0 and VGC Animation 1.0. Higher amounts (Gold Sponsor and above) also give you software upgrades for up to 10 years! It's basically like pre-ordering VGC at a discounted early bird price.

You can either pay in USD via Patreon, or in EUR via Tipeee. You can cancel your monthly payments at any time, and still keep your rewards, no questions asked: we trust you to contribute for a number of months which seems fair to you. Alternatively, you can also choose to make a one-time payment via Paypal rather than monthly contributions. Just click on the chosen button and follow the instructions. Thank you!