Error on any Keras RNN type with return_sequences=True

Question

Hi, having issues with CoreML and Keras for sequence data which is limiting what can be achieved for audio tasks. Although the scripts convert using coremltools (v0.5.1) they do not run.

Starting with something more basic, I have a neural network that is simple input -> FC -> Softmax as follows:

input_data = Input(name='the_input', shape=(None, input_dim))  # >>(?, time, 26)
x = input_data
layercount = 3


for l in range(layercount):
      x = TimeDistributed(Dense(fc_size, name='fc_{}'.format(l+1), kernel_initializer=init,
                                bias_initializer=init, activation='relu'))(x)  # >>(?, time, fc_size)

y_pred = TimeDistributed(Dense(output_dim, name="y_pred", activation="softmax"))(x) # >>(?, time, 29)

This compiles, trains, converts using Coremltools, and is loaded into iOS and works fine. Problem is, Dense layers aren't that good for time series, therefore accuracy and WER is low. Much more useful is return_sequences=True using RNN for something like audio data.

Let's add a type of RNN (GRU/LSTM/SimpleRNN) are all the same result here:

input_data = Input(name='the_input', shape=(None, input_dim))  # >>(?, time, 26)
x = input_data
layercount = 3

for l in range(layercount):
  x = TimeDistributed(Dense(fc_size, name='fc_{}'.format(l+1), kernel_initializer=init,
                                bias_initializer=init, activation='relu'))(x)  # >>(?, time, fc_size)

x = GRU(rnn_size, return_sequences=True, activation='relu', name='rnn1')(x)  # >> (?, time, rnn_size)
y_pred = TimeDistributed(Dense(output_dim, name="y_pred", activation="softmax"))(x)  # >> (?, time, 29)

Again, this compiles in Keras, trains, converts with coremltools however in iOS errors on runtime with:

[coreml] Different batch numbers for input features.

[coreml] Failure in resetSizes.

error: Unexpected runtime error

Thread 1: EXC_BAD_INSTRUCTION (code=EXC_I386_INVOP, subcode=0x0)

Here is a list of things that I have tried.

Forcing a reshape so that it's guarenteed to be a certain size before and after RNN
Implementing 1Dconv CNN before/after

Any ideas or hints welcome as this is causing a real headache. Is there anything I can do with this to get it to run?

Core ML

2.2k

Posted by

rmsmith

Reply

Add a Comment

Answer 1

Hi, rmsmith

Have you tried later versions of coremltools? I tried your script in coremltools 0.6.3 and 0.7, Keras result and CoreML result looks consistent. Here's the script I ran. Thanks!

import keras
from keras.layers import Input, Dense, GRU
from keras.layers.wrappers import Bidirectional, TimeDistributed
from keras.models import Model

def get_model_1(input_dim, seq_len, output_dim):
    input_data = Input(name='the_input', shape=(None, input_dim))
    x = input_data
    layercount = 3

    for l in range(layercount):
      x = TimeDistributed(Dense(6, name='fc_{}'.format(l+1), activation='relu'))(x) 
    x = GRU(8, return_sequences=True, activation='relu', name='rnn1')(x)
    y_pred = TimeDistributed(Dense(output_dim, name="y_pred", activation="softmax"))(x)

    model = Model([input_data],[y_pred])
    return model

input_dim = 4
seq_len = 3
output_dim = 3

model = get_model_1(input_dim,seq_len, output_dim)
import coremltools
mlmodel = coremltools.converters.keras.convert(model)

import numpy as np
cdata = np.random.rand(seq_len,1,input_dim)
kdata = cdata.transpose((1,0,2))
kres = model.predict([kdata])

cres_dict = mlmodel.predict({'input1':cdata})
cres = cres_dict['output1']

print(kres)
print('\n')
print(cres)

Posted by

allenlin

Add a Comment

Answer 2

Hi rmsmith, I ran into the same problem about "Different batch numbers for input features" / "Failure in resetSizes". Did you ever find any solution? I've been searching without luck. If you have any info, I'll appreciate it very much.

Posted by

Lesser Goldfinch

Add a Comment

Error on any Keras RNN type with return_sequences=True

Replies