Getting a Check failed: IsAligned() ptr on tensorflow-metal for the deepdream demo

When running the code on the sample notebook for deepdream — https://www.tensorflow.org/tutorials/generative/deepdream — I get the error below when I try executing the tiled version (cell [18] in the notebook).

2021-12-17 15:56:05.988238: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:112] Plugin optimizer for device_type GPU is enabled.
2021-12-17 15:56:06.768675: F tensorflow/core/framework/tensor.cc:681] Check failed: IsAligned() ptr = 0x7fb6115606f0

I don't get the error on the other cells of the notebook, and I don't get the error running the notebook on CPU, or running it on other accelerators (e.g., NVIDIA on Linux). I'm running tensorflow-metal on a 27-inch, 2017 iMac with a 8GB Radeon Pro 580 and macOS 11.6.1 (20G224).

Answered by Frameworks Engineer in 702814022

The issue comes from core Tensorflow: https://github.com/tensorflow/tensorflow/issues/49202

We are working on this issue and will update when we have a fix.

Same for me on 2020 MBP (M1) and keras_retinanet

Same for me when working with Tensorflow inside a separate class. When uninstalling Tensorflow-metal form my python environment it works again.

MacBook Pro M1 Pro (10/16) 16 GB Memory

Python 3.9.9 Tensorflow-metal 0.3.0

Same for me on 2020 M1 Mac mini and attempting to run ai-benchmark. Defeats the whole point of even benchmarking if the library is busted.

Fails on test 17/19, Pixel-RNN, due to the alignment issue with the allocator. The tensorflow-opencl software had a similar issue back in 2017 that had to be solved by enforcing memory alignment in the allocator(s).

I have the same problem. Imac 2019, 8G ram.

SAME ISSUE HERE. MBP 16 2021 M1 MAX python          3.9.9 tensorflow-deps      2.7.0         tensorflow-estimator   2.7.0          tensorflow-macos     2.7.0          tensorflow-metal     0.3.0          

Hi @EduardoV and others,

Thanks for reporting this issue. We have reproduced this locally and are looking into it. I will update here once the solution is out.

The same issue with LSTM layers, but it correctly works with Dense layers even using GPU. MAC PRO 2019, 28-core Intel Xeon W, AMD Radeon Pro Vega II Duo 32

Same error messages while running code from Deep Learning with Python, listing 6.23. Dies in both Jupyter Lab and command line python at the model.fit() line.

M1 iMac, 16 GB, tensorflow-macos 2.7, metal plugin, Python 3.9.5.

model = Sequential()
model.add(Embedding(max_features, 32))
model.add(SimpleRNN(32))
model.add(Dense(1, activation='sigmoid'))

model.compile(optimizer='rmsprop', loss='binary_crossentropy', metrics=['acc'])
history = model.fit(input_train, y_train,
                    epochs=10,
                    batch_size=128,
                    validation_split=0.2)

I have the same issue but only if I try to add a LTCCell to my model.

model = Sequential()
model.add(RNN(ltc_cell, return_sequences=False, input_shape=(config.num_steps, config.input_size)))
model.add(Dense(128))
model.add(Dense(config.output_size))
model.compile(optimizer=optimizer, loss=loss_function, metrics=['Accuracy'])
model.build()
model.summary()
history = model.fit(train_X, train_y, batch_size=config.batch_size, epochs=config.init_epoch,
                    verbose=verbosity_mode)

When I delete the Metal plugin I can train my network but it would take forever. For a LTCCell with 64 inter_neurons and 32 motor_neurons the estimated time is 2 hours per Epoch. M1 MacBook Pro, 8 GB, tensorflow-macos 2.7, metal plugin 0.3.0, Python 3.9.

Accepted Answer

The issue comes from core Tensorflow: https://github.com/tensorflow/tensorflow/issues/49202

We are working on this issue and will update when we have a fix.

It seems to be working, I only ran a few trials though.

Error still persists in tensorflow-macos=2.9.2 and tensorflow-metal=0.5.0.

Error occurred while attempting to run inference using a Tensorflow 2 version of the Mask R-CNN deep learning neural network architecture for instance segmentation:

2022-08-16 04:06:15.497243: F tensorflow/core/framework/tensor.cc:728] Check failed: IsAligned() ptr = 0x2d5a5cf60

As this thread is 8 months old and multiple versions of both tensorflow-macos and tensorflow-metal for Apple Silicon have been released, one would have imagined there would have been a fix by now...

Having the same error:

2022-08-26 10:13:48.413402: F tensorflow/core/framework/tensor.cc:728] Check failed: IsAligned() ptr = 0x17cc64950

with:

tensorflow-macos==2.9.2
tensorflow-metal==0.5.1
tensorflow-probability==0.14.0

Any update on this?

BTW: Runs without tensorflow-metal installed.

Getting the same error when working with 64-bit float variables. Minimal example:

import tensorflow as tf

DTYPE = "float64"
def mre():
    P = tf.Variable(tf.ones(shape=(), dtype=DTYPE))
    delta = tf.ones(dtype=DTYPE, shape=P.shape)
    print(f"{P.numpy()=}")
    print(f"{P.dtype=}")
    P.assign_add(delta)  # crashes here
    print(P.numpy())

if __name__ == "__main__":
    mre()

Environment:

Python 3.11.7

tensorflow                   2.15.0
tensorflow-estimator         2.15.0
tensorflow-io-gcs-filesystem 0.36.0
tensorflow-macos             2.15.0
tensorflow-metal             1.1.0

Chip: M3 Pro
OS: 14.4.1 (23E224)
Getting a Check failed: IsAligned() ptr on tensorflow-metal for the deepdream demo
 
 
Q