1 Reply
      Latest reply on Dec 7, 2017 1:16 PM by allenlin
      rmsmith Level 1 Level 1 (0 points)

        Hi, having issues with CoreML and Keras for sequence data which is limiting what can be achieved for audio tasks. Although the scripts convert using coremltools (v0.5.1) they do not run.

         

        Starting with something more basic, I have a neural network that is simple input -> FC -> Softmax as follows:

        input_data = Input(name='the_input', shape=(None, input_dim))  # >>(?, time, 26)
        x = input_data
        layercount = 3
        
        
        for l in range(layercount):
              x = TimeDistributed(Dense(fc_size, name='fc_{}'.format(l+1), kernel_initializer=init,
                                        bias_initializer=init, activation='relu'))(x)  # >>(?, time, fc_size)
        
        y_pred = TimeDistributed(Dense(output_dim, name="y_pred", activation="softmax"))(x) # >>(?, time, 29)
        
        
        
        
        
        

         

        This compiles, trains, converts using Coremltools, and is loaded into iOS and works fine. Problem is, Dense layers aren't that good for time series, therefore accuracy and WER is low. Much more useful is return_sequences=True using RNN for something like audio data.

         

        Let's add a type of RNN (GRU/LSTM/SimpleRNN) are all the same result here:

        input_data = Input(name='the_input', shape=(None, input_dim))  # >>(?, time, 26)
        x = input_data
        layercount = 3
        
        for l in range(layercount):
          x = TimeDistributed(Dense(fc_size, name='fc_{}'.format(l+1), kernel_initializer=init,
                                        bias_initializer=init, activation='relu'))(x)  # >>(?, time, fc_size)
        
        x = GRU(rnn_size, return_sequences=True, activation='relu', name='rnn1')(x)  # >> (?, time, rnn_size)
        y_pred = TimeDistributed(Dense(output_dim, name="y_pred", activation="softmax"))(x)  # >> (?, time, 29)
        
        
        
        
        
        

         

        Again, this compiles in Keras, trains, converts with coremltools however in iOS errors on runtime with:

         

        [coreml] Different batch numbers for input features.

        [coreml] Failure in resetSizes.

        error: Unexpected runtime error

        Thread 1: EXC_BAD_INSTRUCTION (code=EXC_I386_INVOP, subcode=0x0)

         



        Here is a list of things that I have tried.

        1. Forcing a reshape so that it's guarenteed to be a certain size before and after RNN
        2. Implementing 1Dconv CNN before/after

         

        Any ideas or hints welcome as this is causing a real headache. Is there anything I can do with this to get it to run?


        • Re: Error on any Keras RNN type with return_sequences=True
          allenlin Apple Staff Apple Staff (20 points)

          Hi, rmsmith

           

          Have you tried later versions of coremltools? I tried your script in coremltools 0.6.3 and 0.7, Keras result and CoreML result looks consistent. Here's the script I ran. Thanks!

           

          import keras
          from keras.layers import Input, Dense, GRU
          from keras.layers.wrappers import Bidirectional, TimeDistributed
          from keras.models import Model
          
          def get_model_1(input_dim, seq_len, output_dim):
              input_data = Input(name='the_input', shape=(None, input_dim))
              x = input_data
              layercount = 3
          
              for l in range(layercount):
                x = TimeDistributed(Dense(6, name='fc_{}'.format(l+1), activation='relu'))(x) 
              x = GRU(8, return_sequences=True, activation='relu', name='rnn1')(x)
              y_pred = TimeDistributed(Dense(output_dim, name="y_pred", activation="softmax"))(x)
          
              model = Model([input_data],[y_pred])
              return model
          
          input_dim = 4
          seq_len = 3
          output_dim = 3
          
          model = get_model_1(input_dim,seq_len, output_dim)
          import coremltools
          mlmodel = coremltools.converters.keras.convert(model)
          
          import numpy as np
          cdata = np.random.rand(seq_len,1,input_dim)
          kdata = cdata.transpose((1,0,2))
          kres = model.predict([kdata])
          
          cres_dict = mlmodel.predict({'input1':cdata})
          cres = cres_dict['output1']
          
          print(kres)
          print('\n')
          print(cres)