Is clone pattern supported with classes overloaded in python?

[yesint]

In C++ clone pattern is used in many APIs:

class A {
public:
   A();
   A(const A& other)
   virtual void func()=0;
   virtual A* clone() const { return new A(*this);  }
}

class A_trampoline: public A {
 // Usual things for pure virtual func()
}

// function dependent on clone functionality
void work_on_many_instances(const shared_ptr<A>& inst){
    std::vector<shared_ptr<A>> tasks;
    for(int i=0; i<100; ++i) tasks.push_back( shared_ptr<A>(inst->clone()) );
    // Do something with clonned instances
}

Suppose one extends class A in python using standard trampoline facilities and exposes it as usual:

py::class_<A,A_trampoline,std::shared_ptr<A>>(m, "A") 
....
;

// Accepts PyObject with class derived in python from A
m.def("work_on_many_instances", [](const py::object& o){
        auto p = o.cast<shared_ptr<A>>();
        work_on_many_instances(p); // UPS! clone will try to create A, not the derived class...
   });

When clone() is called inside work_on_many_instances() it creates an instance of A, which is an abstract class and, of course, fails. Unless I'm missing something obvious I can't figure out how to clone an object of derived class correctly. As far as I understand the trampoline should make a deep copy of existing python derived class instance on C++ side somehow but I have absolutely no idea how. Is it possible at all?

[YannickJadoul]

Isn't the whole point of a virtual clone() just to avoid this, i.e. to have a polymorphic copy-constructor that you can override in the derived classes?

Why don't you just add virtual A_trampoline *clone() const { return new A_trampoline(*this); } to the trampoline class?

[yesint]

Yes, this is what I did initially. Unfortunately it doesn't work:

what(): Tried to call pure virtual function "A::func"

That is what I can't understand, actually.

[YannickJadoul]

When exactly does that happen? You have implemented a non-pure A_trampoline::func, right?

[YannickJadoul]

Wait, I'm now wondering if o.cast<shared_ptr<A>>(); is the right thing to do. Since there will be a shared_ptr<A_trampoline> inside, no?

[dean0x7d]

@yesint Can you post a complete code example (both C++ and Python sides) which reproduces the error that you're seeing?

[yesint]

@dean0x7d Here is minimal example to reproduce:

//-----------------------------
#include <pybind11/pybind11.h>

using namespace std;
namespace py = pybind11;

class A {
public:
   A(){}
   A(const A& other){}
   virtual void func()=0;
   virtual A* clone() const =0;
};

class A_trampoline: public A {
    using A::A;

    void func() override {
        PYBIND11_OVERLOAD_PURE(
            void, /* Return type */
            A,      /* Parent class */
            func,          /* Name of function in C++ (must match Python name) */
                          /* Argument(s) */
        );
    }

    virtual A_trampoline* clone() const {
        return new A_trampoline(*this);
    }
};

// function dependent on clone functionality
void work_on_many_instances(const shared_ptr<A>& inst){
    std::vector<shared_ptr<A>> tasks;
    for(int i=0; i<100; ++i) tasks.push_back( shared_ptr<A>(inst->clone()) );
    for(int i=0; i<100; ++i) tasks[i]->func(); // Call python overload for all instances
}


PYBIND11_MODULE(module1, m) {

    py::class_<A,A_trampoline,std::shared_ptr<A>>(m, "A")
            .def(py::init<>())
            .def("func",&A::func)
    ;

    // Accepts PyObject with class derived in python from A
    m.def("work_on_many_instances", [](const py::object& o){
            auto p = o.cast<shared_ptr<A>>();  //shared_ptr<A_trampoline> gives the same error
            work_on_many_instances(p); // UPS! clone will try to create A, not the derived class...
       });
}

And the python code to test:

from module1 import *

class B(A):
    def func(self):
        print "hi!"
        
inst = B()
work_on_many_instances(inst)

The error:

$ python test_1.py 
Traceback (most recent call last):
  File "test_1.py", line 8, in <module>
    work_on_many_instances(inst)
RuntimeError: Tried to call pure virtual function "A::func"

[yesint]

Wait, I'm now wondering if o.cast<shared_ptr>(); is the right thing to do. Since there will be a shared_ptr<A_trampoline> inside, no?

Unless I call clone() this cast gives me pointer to correct object, which is able to call python-derived func():

m.def("work_on_many_instances", [](const py::object& o){
            auto p = o.cast<shared_ptr<A>>();
            p->func();           
       });

// prints: hi! when called from python

[yesint]

Correct me if I'm wrong (I don't know internals of pybind11) but it seems that when clone() is called only C++ A_trampoline object is created but not the B instance on the python side. As a result the trampoline redirects the virtual call to nothing ending up in calling pure-virtual A::func(). So the idea as far as I understand is to create B instance in python interpreter as usual and access it from A_trampoline::clone instead of allocating A_trampoline at C++ side. But I have no idea how to implement this.

[YannickJadoul]

Oh, yes, of course! You are right; I'd failed to notice that complexity!
So the A_trampoline::func() does get called, but there's nothing it can do because it's indeed not a B...

One option is then of course to override the clone() in the same way on the Python side, but that somehow does not feel very Pythonic?

Another option is to use the type of the actual Python object like this:

    virtual A_trampoline* clone() const {
	auto this_object = py::cast(this);
        return py::cast<A_trampoline*>(this_object.get_type()(this_object).release());
    }

and then add

            .def(py::init_alias<const A_trampoline &>())

The downside of this approach is that when you implement __init__ on the Python derived class, that it should also have a 'copy constructor overload' that's cascaded back to A.

EDIT:
Either way, I've just tested the latter approach, and that does seem to work for me.

[YannickJadoul]

Wait, actually, this might be nastier than it seemed at first. Because ... where is the actual Python object now stored (including its attributes)? I guess this either means something's leaking, or that call to func() is unsafe?

[yesint]

Well, this black magic works :) At least the program now behaves as expected.
However I don't quit understand the point with init:

from module1 import *

class B(A):
    
    def __init__(self):
        A.__init__(self)
        self.x = 1
        
    def func(self):
        print "hi!",self.x
        
inst = B()
work_on_many_instances(inst)

Gives:

$ python test_1.py 
Traceback (most recent call last):
  File "test_1.py", line 12, in <module>
    work_on_many_instances(inst)
TypeError: __init__() takes exactly 1 argument (2 given)

[YannickJadoul]

Yeah, there's the thing with this_object.get_type()(this_object), which is a call to that constructor of B, but with the object as argument.

Another option would to try and see if you can replace this copy-constructor thing with Python's copy.copy() or copy.deepcopy() ?

Edit: just doing this works, btw:

     def __init__(self, *args):
        A.__init__(self, *args)

[YannickJadoul]

But also, I'm still not convinced that this py::cast<A_trampoline*>(<...>.release()); is ok. Maybe @dean0x7d has a better idea of what would be happening inside pybind11, and why this approach magically doesn't loose the Python object's attributes?

[yesint]

Well, it seems that the attributes are not copied and have to be assigned in python-side "copy-constructor":

def __init__(self,other=None):
        if other:
            A.__init__(self,other)
            self.x = other.x # Without this x is not cloned
        else:
            A.__init__(self)
            self.x = 0

I'm not familiar with deepcopy() enough to figure out how to use it in __init__ but in the case of deepcopy what will happen with C++ object base?

[yesint]

Btw, both copy() and deepcopy() fails on B:

inst = B()
copy.deepcopy(inst)

> Traceback (most recent call last):
  File "test_1.py", line 21, in <module>
    copy.deepcopy(inst)
  File "/usr/lib/python2.7/copy.py", line 190, in deepcopy
    y = _reconstruct(x, rv, 1, memo)
  File "/usr/lib/python2.7/copy.py", line 329, in _reconstruct
    y = callable(*args)
  File "/usr/lib/python2.7/copy_reg.py", line 93, in __newobj__
    return cls.__new__(cls, *args)
TypeError: pybind11_object.__new__(B) is not safe, use object.__new__()