Everything is pickled twice

[crusaderky]

import distributed
import logging

def log(msg):
    with open('poc.log', 'a') as fh:
        fh.write(msg + '\n')

class Dummy:
    def __getstate__(self):
        log('pickle')
        return self.__dict__
    
    def __setstate__(self, state):
        self.__dict__ = state
        log('unpickle')

cluster = distributed.LocalCluster(n_workers=1, threads_per_worker=1, processes=True)
client = distributed.Client(cluster)

client.submit(lambda x: x, Dummy()).result()

In poc.log:

pickle
pickle
unpickle
pickle
pickle
unpickle

Am I missing something obvious?

[mrocklin]

We can isolate this as follows:

In [1]: class Dummy:
   ...:     def __getstate__(self):
   ...:         print('pickle')
   ...:         return self.__dict__
   ...:     
   ...:     def __setstate__(self, state):
   ...:         self.__dict__ = state
   ...:         print('unpickle')
   ...: 
   ...: from distributed.protocol.pickle import dumps, loads
   ...: dummy = Dummy()
   ...: dumps(dummy)
   ...: 
pickle
pickle
Out[1]: b'\x80\x04\x95\x07\x02\x00\x00\x00\x00\x00\x00\x8c\x17cloudpickle.cloudpickle\x94\x8c\x19_rehydrate_skeleton_class\x94\x93\x94(h\x00\x8c\r_builtin_type\x94\x93\x94\x8c\tClassType\x94\x85\x94R\x94\x8c\x05Dummy\x94\x8c\x08builtins\x94\x8c\x06object\x94\x93\x94\x85\x94}\x94\x8c\x07__doc__\x94Ns\x87\x94R\x94}\x94(\x8c\n__module__\x94\x8c\x08__main__\x94\x8c\x0c__getstate__\x94h\x00\x8c\x0e_fill_function\x94\x93\x94(h\x00\x8c\x0f_make_skel_func\x94\x93\x94h\x04\x8c\x08CodeType\x94\x85\x94R\x94(K\x01K\x00K\x01K\x02KCC\x0et\x00d\x01\x83\x01\x01\x00|\x00j\x01S\x00\x94N\x8c\x06pickle\x94\x86\x94\x8c\x05print\x94\x8c\x08__dict__\x94\x86\x94\x8c\x04self\x94\x85\x94\x8c\x1e<ipython-input-1-46d87eae6452>\x94h\x14K\x02C\x04\x00\x01\x08\x01\x94))t\x94R\x94J\xff\xff\xff\xff}\x94\x87\x94R\x94}\x94N}\x94NtR\x8c\x0c__setstate__\x94h\x16(h\x18h\x1b(K\x02K\x00K\x02K\x02KCC\x12|\x01|\x00_\x00t\x01d\x01\x83\x01\x01\x00d\x00S\x00\x94N\x8c\x08unpickle\x94\x86\x94h h\x1f\x86\x94h"\x8c\x05state\x94\x86\x94h$h-K\x06C\x04\x00\x01\x06\x01\x94))t\x94R\x94J\xff\xff\xff\xffh(\x87\x94R\x94}\x94N}\x94NtRutR)\x81\x94}\x94b.'

There is some complex behaviors in distributed.protocol.pickle that optimize over a few cases. There is no universal way to make everything fast all the time unfortunately.

def dumps(x):
    """ Manage between cloudpickle and pickle

    1.  Try pickle
    2.  If it is short then check if it contains __main__
    3.  If it is long, then first check type, then check __main__
    """
    try:
        result = pickle.dumps(x, protocol=pickle.HIGHEST_PROTOCOL)
        if len(result) < 1000:
            if b'__main__' in result:
                return cloudpickle.dumps(x, protocol=pickle.HIGHEST_PROTOCOL)
            else:
                return result
        else:
            if _always_use_pickle_for(x) or b'__main__' not in result:
                return result
            else:
                return cloudpickle.dumps(x, protocol=pickle.HIGHEST_PROTOCOL)
    except Exception:
        try:
            return cloudpickle.dumps(x, protocol=pickle.HIGHEST_PROTOCOL)
        except Exception as e:
            logger.info("Failed to serialize %s. Exception: %s", x, e)
            raise

[crusaderky]

Sorry I forgot this open - I can see how the current behaviour is sane.

[mrocklin]

Thanks for closing!

…

On Sun, Jun 17, 2018 at 4:11 AM, crusaderky ***@***.***> wrote: Sorry I forgot this open - I can see how the current behaviour is sane. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#1371 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AASszA7ve0sLRHAph9auVv5trdsxep3aks5t9g8_gaJpZM4PLRAD> .