Keras 2 model conversion failure

I have a Keras 2 model, it seems to work correctly in Python / Keras / TensorFlow back end (it's giving correct classificatios when the test script is run). Conversion to CoreML, on the other hand, fails with a mysterious stack trace (bad marshal). Any idea what is going on? Any pointer appreciated.


File "/Users/***/Documents/Models/coreml_*********/lib/python2.7/site-packages/coremltools/converters/keras/_keras_*********.py", line 477, in convert

predicted_feature_name = predicted_feature_name)

File "/Users/***/Documents/Models/coreml_*********/lib/python2.7/site-packages/coremltools/converters/keras/_keras2_*********.py", line 149, in _convert

model = _keras.models.load_model(model)

File "/Users/***/Documents/Models/coreml_*********/lib/python2.7/site-packages/keras/models.py", line 240, in load_model

model = model_from_config(model_config, custom_objects=custom_objects)

File "/Users/***/Documents/Models/coreml_*********/lib/python2.7/site-packages/keras/models.py", line 304, in model_from_config

return layer_module.deserialize(config, custom_objects=custom_objects)

File "/Users/***/Documents/Models/coreml_*********/lib/python2.7/site-packages/keras/layers/__init__.py", line 54, in deserialize

printable_module_name='layer')

File "/Users/***/Documents/Models/coreml_*********/lib/python2.7/site-packages/keras/utils/generic_utils.py", line 140, in deserialize_keras_object

list(custom_objects.items())))

File "/Users/***/Documents/Models/coreml_*********/lib/python2.7/site-packages/keras/engine/topology.py", line 2416, in from_config

process_layer(layer_data)

File "/Users/***/Documents/Models/coreml_*********/lib/python2.7/site-packages/keras/engine/topology.py", line 2385, in process_layer

custom_objects=custom_objects)

File "/Users/***/Documents/Models/coreml_*********/lib/python2.7/site-packages/keras/layers/__init__.py", line 54, in deserialize

printable_module_name='layer')

File "/Users/***/Documents/Models/coreml_*********/lib/python2.7/site-packages/keras/utils/generic_utils.py", line 140, in deserialize_keras_object

list(custom_objects.items())))

File "/Users/***/Documents/Models/coreml_*********/lib/python2.7/site-packages/keras/layers/core.py", line 706, in from_config

function = func_load(config['function'], globs=globs)

File "/Users/***/Documents/Models/coreml_*********/lib/python2.7/site-packages/keras/utils/generic_utils.py", line 198, in func_load

code = marshal.loads(code.encode('raw_unicode_escape'))

ValueError: bad marshal data (unknown type code)

Up vote post of eppilo Down vote post of eppilo
3.8k views
Posted by
eppilo
Reply
Add a Comment

Accepted Reply

The best way is to check the CoreML Protobuf specification. Here is an excerpt (ignore the "params", those are the supported layer types):


message NeuralNetworkLayer {
    oneof layer {
        ConvolutionLayerParams convolution = 100;

        PoolingLayerParams pooling = 120;

        ActivationParams activation = 130;

        InnerProductLayerParams innerProduct = 140;
        EmbeddingLayerParams embedding = 150;

        //normalization related layers
        BatchnormLayerParams batchnorm = 160;
        MeanVarianceNormalizeLayerParams mvn = 165;
        L2NormalizeLayerParams l2normalize = 170;
        SoftmaxLayerParams softmax = 175;
        LRNLayerParams lrn = 180;

        CropLayerParams crop = 190;
        PaddingLayerParams padding = 200;
        UpsampleLayerParams upsample = 210;

        UnaryFunctionLayerParams unary = 220;

        //elementwise operations
        AddLayerParams add = 230;
        MultiplyLayerParams multiply = 231;

        AverageLayerParams average = 240;
        ScaleLayerParams scale = 245;

        BiasLayerParams bias = 250;
        MaxLayerParams max = 260;
        MinLayerParams min = 261;

        DotProductLayerParams dot = 270;
        ReduceLayerParams reduce = 280;
        LoadConstantLayerParams loadConstant = 290;

        //data reorganization
        ReshapeLayerParams reshape = 300;
        FlattenLayerParams flatten = 301;
        PermuteLayerParams permute = 310;
        ConcatLayerParams concat = 320;
        SplitLayerParams split = 330;
        SequenceRepeatLayerParams sequenceRepeat = 340;

        //Recurrent Layers
        SimpleRecurrentLayerParams simpleRecurrent = 400;
        GRULayerParams gru = 410;
        UniDirectionalLSTMLayerParams uniDirectionalLSTM = 420;
        BiDirectionalLSTMLayerParams biDirectionalLSTM = 430;
    }
}


You can also find the supported activators in there:


message ActivationParams {
    oneof NonlinearityType {
        ActivationLinear linear = 5;

        ActivationReLU ReLU = 10;
        ActivationLeakyReLU leakyReLU = 15;
        ActivationThresholdedReLU thresholdedReLU = 20;
        ActivationPReLU PReLU = 25;

        ActivationTanh tanh = 30;
        ActivationScaledTanh scaledTanh = 31;

        ActivationSigmoid sigmoid = 40;
        ActivationSigmoidHard sigmoidHard = 41;

        ActivationELU ELU = 50;

        ActivationSoftsign softsign = 60;
        ActivationSoftplus softplus = 70;
        ActivationParametricSoftplus parametricSoftplus = 71;
    }
}
Posted by
FrankSchlegel
Up vote reply of FrankSchlegel Down vote reply of FrankSchlegel
Post marked as solved
Add a Comment

Replies

The `custom_objects` in the stracktrace is a good hint. Likely your model contains a layer type that is not supported by Core ML.

Posted by
kerfuffle
Up vote reply of kerfuffle Down vote reply of kerfuffle
Add a Comment

Thanks a lot for the answer, the model in question is YOLO (not Tiny YOLO). So if I understand correctly what you are saying, I need to look at the model configuration and find out if some layers are not supported by CoreML. Could it be the REORG layer? I am a bit lost as to which layers CoreML actually supports: how to find out?

Posted by
eppilo
Up vote reply of eppilo Down vote reply of eppilo
Add a Comment

The best way is to check the CoreML Protobuf specification. Here is an excerpt (ignore the "params", those are the supported layer types):


message NeuralNetworkLayer {
    oneof layer {
        ConvolutionLayerParams convolution = 100;

        PoolingLayerParams pooling = 120;

        ActivationParams activation = 130;

        InnerProductLayerParams innerProduct = 140;
        EmbeddingLayerParams embedding = 150;

        //normalization related layers
        BatchnormLayerParams batchnorm = 160;
        MeanVarianceNormalizeLayerParams mvn = 165;
        L2NormalizeLayerParams l2normalize = 170;
        SoftmaxLayerParams softmax = 175;
        LRNLayerParams lrn = 180;

        CropLayerParams crop = 190;
        PaddingLayerParams padding = 200;
        UpsampleLayerParams upsample = 210;

        UnaryFunctionLayerParams unary = 220;

        //elementwise operations
        AddLayerParams add = 230;
        MultiplyLayerParams multiply = 231;

        AverageLayerParams average = 240;
        ScaleLayerParams scale = 245;

        BiasLayerParams bias = 250;
        MaxLayerParams max = 260;
        MinLayerParams min = 261;

        DotProductLayerParams dot = 270;
        ReduceLayerParams reduce = 280;
        LoadConstantLayerParams loadConstant = 290;

        //data reorganization
        ReshapeLayerParams reshape = 300;
        FlattenLayerParams flatten = 301;
        PermuteLayerParams permute = 310;
        ConcatLayerParams concat = 320;
        SplitLayerParams split = 330;
        SequenceRepeatLayerParams sequenceRepeat = 340;

        //Recurrent Layers
        SimpleRecurrentLayerParams simpleRecurrent = 400;
        GRULayerParams gru = 410;
        UniDirectionalLSTMLayerParams uniDirectionalLSTM = 420;
        BiDirectionalLSTMLayerParams biDirectionalLSTM = 430;
    }
}


You can also find the supported activators in there:


message ActivationParams {
    oneof NonlinearityType {
        ActivationLinear linear = 5;

        ActivationReLU ReLU = 10;
        ActivationLeakyReLU leakyReLU = 15;
        ActivationThresholdedReLU thresholdedReLU = 20;
        ActivationPReLU PReLU = 25;

        ActivationTanh tanh = 30;
        ActivationScaledTanh scaledTanh = 31;

        ActivationSigmoid sigmoid = 40;
        ActivationSigmoidHard sigmoidHard = 41;

        ActivationELU ELU = 50;

        ActivationSoftsign softsign = 60;
        ActivationSoftplus softplus = 70;
        ActivationParametricSoftplus parametricSoftplus = 71;
    }
}
Posted by
FrankSchlegel
Up vote reply of FrankSchlegel Down vote reply of FrankSchlegel
Post marked as solved
Add a Comment

Same problems with yours. And it seems that the problems can be solved by changing the .cfg file.


Did you solve it? Can you tell me how to solve it?🙂

Posted by
lu.meng
Up vote reply of lu.meng Down vote reply of lu.meng
Add a Comment

No I haven't. I marked an answer as correct because it helps understanding a bit potential issues but not because it solves the problem. I am not an expert in Darknet models, but changong the .cfg file sounds wrong. The model comes pre-trained so you have to rely on the same model configuration that is producing the weights. Did you get anywhere?

Posted by
eppilo
Up vote reply of eppilo Down vote reply of eppilo
Add a Comment

I had luck getting past the bad marshalling error by using CoreML in Python 3. You can do this by cloning the coremltools repo and building from source. After building from source, add the repo's directory to your PYTHONPATH.

Posted by
hidoodle
Up vote reply of hidoodle Down vote reply of hidoodle
Add a Comment