Post

Replies

Boosts

Views

Activity

Reply to Sklearn is unstable on Apple Silicon
I reinstalled the sklearn and scipy, after that sklearn can be imported correctly. Package                  Version absl-py                  0.10.0 aiohttp                  3.7.4.post0 anyio                    2.2.0 appnope                  0.1.2 argon2-cffi              20.1.0 astroid                  2.6.6 astunparse               1.6.3 async-generator          1.10 async-timeout            3.0.1 attrs                    21.2.0 autopep8                 1.5.6 Babel                    2.9.1 backcall                 0.2.0 beniget                  0.3.0 biopython                1.79 bleach                   4.0.0 blinker                  1.4 Bottleneck               1.3.2 brotlipy                 0.7.0 cached-property          1.5.2 cachetools               4.2.2 certifi                  2021.5.30 cffi                     1.14.5 chardet                  4.0.0 charset-normalizer       2.0.4 clang                    5.0 click                    8.0.3 cloudpickle              2.0.0 conda                    4.10.3 conda-package-handling   1.7.3 cryptography             3.4.7 cycler                   0.11.0 Cython                   0.29.24 decorator                5.1.0 defusedxml               0.7.1 deprecation              2.1.0 dill                     0.3.4 entrypoints              0.3 et-xmlfile               1.1.0 flake8                   3.9.0 flatbuffers              1.12 future                   0.18.2 gast                     0.4.0 google-auth              1.33.0 google-auth-oauthlib     0.4.1 google-pasta             0.2.0 googleapis-common-protos 1.53.0 grpcio                   1.38.1 gym                      0.21.0 h5py                     3.1.0 idna                     3.2 importlib-metadata       4.8.1 ipykernel                5.3.4 ipython                  7.26.0 ipython-genutils         0.2.0 isort                    5.9.3 jdcal                    1.4.1 jedi                     0.17.2 Jinja2                   3.0.2 joblib                   1.1.0 json5                    0.9.6 jsonschema               3.2.0 jupyter-client           7.0.1 jupyter-core             4.7.1 jupyter-http-over-ws     0.0.8 jupyter-lsp              1.5.0 jupyter-packaging        0.10.4 jupyter-server           1.6.4 jupyterlab               3.1.7 jupyterlab-lsp           3.9.1 jupyterlab-pygments      0.1.2 jupyterlab-server        2.8.2 keras                    2.6.0 Keras-Preprocessing      1.1.2 kiwisolver               1.3.2 lazy-object-proxy        1.6.0 Markdown                 3.3.4 MarkupSafe               2.0.1 matplotlib               3.4.2 matplotlib-inline        0.1.2 mccabe                   0.6.1 mistune                  0.8.4 msgpack                  1.0.2 msgpack-numpy            0.4.7.1 multidict                5.1.0 nbclassic                0.2.6 nbclient                 0.5.3 nbconvert                6.1.0 nbformat                 5.1.3 nest-asyncio             1.5.1 networkx                 2.6.3 notebook                 6.4.5 numexpr                  2.7.3 numpy                    1.19.5 oauthlib                 3.1.1 olefile                  0.46 openpyxl                 3.0.5 opt-einsum               3.3.0 packaging                21.0 pandas                   1.3.1 pandocfilters            1.4.3 parso                    0.7.0 pexpect                  4.8.0 pickleshare              0.7.5 Pillow                   8.4.0 pip                      21.3 pluggy                   0.13.1 ply                      3.11 prometheus-client        0.11.0 promise                  2.3 prompt-toolkit           3.0.20 protobuf                 3.17.2 ptyprocess               0.7.0 pyasn1                   0.4.8 pyasn1-modules           0.2.8 pybind11                 2.8.1 pycodestyle              2.6.0 pycosat                  0.6.3 pycparser                2.20 pydocstyle               6.1.1 pyflakes                 2.2.0 pyglet                   1.5.21 Pygments                 2.10.0 PyJWT                    2.1.0 pylint                   2.9.6 pyOpenSSL                21.0.0 pyparsing                3.0.4 pyrsistent               0.18.0 PySocks                  1.7.1 python-dateutil          2.8.2 python-jsonrpc-server    0.4.0 python-language-server   0.36.2 pythran                  0.9.11 pytz                     2021.3 pyzmq                    20.0.0 requests                 2.26.0 requests-oauthlib        1.3.0 rope                     0.19.0 rsa                      4.7.2 ruamel-yaml-conda        0.15.100 scikit-learn             1.0.1 scipy                    1.7.1 seaborn                  0.11.2 Send2Trash               1.8.0 setuptools               52.0.0.post20210825 six                      1.15.0 sniffio                  1.2.0 snowballstemmer          2.1.0 tensorboard              2.6.0 tensorboard-data-server  0.6.0 tensorboard-plugin-wit   1.6.0 tensorflow-datasets      4.4.0 tensorflow-estimator     2.7.0 tensorflow-macos         2.6.0 tensorflow-metadata      1.4.0 tensorflow-metal         0.2.0 Tensorforce              0.6.5 termcolor                1.1.0 terminado                0.9.4 testpath                 0.5.0 threadpoolctl            3.0.0 toml                     0.10.2 tomlkit                  0.7.0 tornado                  6.1 tqdm                     4.62.3 traitlets                5.1.0 typing-extensions        3.7.4.3 ujson                    4.0.2 urllib3                  1.26.7 wcwidth                  0.2.5 webencodings             0.5.1 Werkzeug                 2.0.2 wheel                    0.35.1 wrapt                    1.12.1 yapf                     0.31.0 yarl                     1.6.3 zipp                     3.6.0
Machine Learning & AI General
Nov ’21
Reply to Sklearn is unstable on Apple Silicon
Here is the pip list. and I cannot run python -c "import sklearn" on my terminal. Package                  Version absl-py                  0.10.0 aiohttp                  3.7.4.post0 anyio                    2.2.0 appnope                  0.1.2 argon2-cffi              20.1.0 astroid                  2.6.6 astunparse               1.6.3 async-generator          1.10 async-timeout            3.0.1 attrs                    21.2.0 autopep8                 1.5.6 Babel                    2.9.1 backcall                 0.2.0 beniget                  0.3.0 biopython                1.79 bleach                   4.0.0 blinker                  1.4 Bottleneck               1.3.2 brotlipy                 0.7.0 cached-property          1.5.2 cachetools               4.2.2 certifi                  2021.5.30 cffi                     1.14.5 chardet                  4.0.0 charset-normalizer       2.0.4 clang                    5.0 click                    8.0.3 cloudpickle              2.0.0 conda                    4.10.3 conda-package-handling   1.7.3 cryptography             3.4.7 cycler                   0.11.0 Cython                   0.29.24 decorator                5.1.0 defusedxml               0.7.1 deprecation              2.1.0 dill                     0.3.4 entrypoints              0.3 et-xmlfile               1.1.0 flake8                   3.9.0 flatbuffers              1.12 future                   0.18.2 gast                     0.4.0 google-auth              1.33.0 google-auth-oauthlib     0.4.1 google-pasta             0.2.0 googleapis-common-protos 1.53.0 grpcio                   1.38.1 gym                      0.21.0 h5py                     3.1.0 idna                     3.2 importlib-metadata       4.8.1 ipykernel                5.3.4 ipython                  7.26.0 ipython-genutils         0.2.0 isort                    5.9.3 jdcal                    1.4.1 jedi                     0.17.2 Jinja2                   3.0.2 joblib                   1.1.0 json5                    0.9.6 jsonschema               3.2.0 jupyter-client           7.0.1 jupyter-core             4.7.1 jupyter-http-over-ws     0.0.8 jupyter-lsp              1.5.0 jupyter-packaging        0.10.4 jupyter-server           1.6.4 jupyterlab               3.1.7 jupyterlab-lsp           3.9.1 jupyterlab-pygments      0.1.2 jupyterlab-server        2.8.2 keras                    2.6.0 Keras-Preprocessing      1.1.2 kiwisolver               1.3.2 lazy-object-proxy        1.6.0 Markdown                 3.3.4 MarkupSafe               2.0.1 matplotlib               3.4.2 matplotlib-inline        0.1.2 mccabe                   0.6.1 mistune                  0.8.4 msgpack                  1.0.2 msgpack-numpy            0.4.7.1 multidict                5.1.0 nbclassic                0.2.6 nbclient                 0.5.3 nbconvert                6.1.0 nbformat                 5.1.3 nest-asyncio             1.5.1 networkx                 2.6.3 notebook                 6.4.5 numexpr                  2.7.3 numpy                    1.19.5 oauthlib                 3.1.1 olefile                  0.46 openpyxl                 3.0.5 opt-einsum               3.3.0 packaging                21.0 pandas                   1.3.1 pandocfilters            1.4.3 parso                    0.7.0 pexpect                  4.8.0 pickleshare              0.7.5 Pillow                   8.4.0 pip                      21.3 pluggy                   0.13.1 ply                      3.11 prometheus-client        0.11.0 promise                  2.3 prompt-toolkit           3.0.20 protobuf                 3.17.2 ptyprocess               0.7.0 pyasn1                   0.4.8 pyasn1-modules           0.2.8 pybind11                 2.8.1 pycodestyle              2.6.0 pycosat                  0.6.3 pycparser                2.20 pydocstyle               6.1.1 pyflakes                 2.2.0 pyglet                   1.5.21 Pygments                 2.10.0 PyJWT                    2.1.0 pylint                   2.9.6 pyOpenSSL                21.0.0 pyparsing                3.0.4 pyrsistent               0.18.0 PySocks                  1.7.1 python-dateutil          2.8.2 python-jsonrpc-server    0.4.0 python-language-server   0.36.2 pythran                  0.9.11 pytz                     2021.3 pyzmq                    20.0.0 requests                 2.26.0 requests-oauthlib        1.3.0 rope                     0.19.0 rsa                      4.7.2 ruamel-yaml-conda        0.15.100 scikit-learn             1.0.1 scipy                    1.8.0.dev0+1921.c30fefc seaborn                  0.11.2 Send2Trash               1.8.0 setuptools               52.0.0.post20210825 six                      1.15.0 sniffio                  1.2.0 snowballstemmer          2.1.0 tensorboard              2.6.0 tensorboard-data-server  0.6.0 tensorboard-plugin-wit   1.6.0 tensorflow-datasets      4.4.0 tensorflow-estimator     2.7.0 tensorflow-macos         2.6.0 tensorflow-metadata      1.4.0 tensorflow-metal         0.2.0 Tensorforce              0.6.5 termcolor                1.1.0 terminado                0.9.4 testpath                 0.5.0 threadpoolctl            3.0.0 toml                     0.10.2 tomlkit                  0.7.0 tornado                  6.1 tqdm                     4.62.3 traitlets                5.1.0 typing-extensions        3.7.4.3 ujson                    4.0.2 urllib3                  1.26.7 wcwidth                  0.2.5 webencodings             0.5.1 Werkzeug                 2.0.2 wheel                    0.35.1 wrapt                    1.12.1 yapf                     0.31.0 yarl                     1.6.3 zipp                     3.6.0
Machine Learning & AI General
Nov ’21
Reply to M1 GPU is extremely slow, how can I enable CPU to train my NNs?
It seems like the small batch size will reduce the performance of GPU https://developer.apple.com/forums/thread/685623 so I increased the batch size from 256 to 1024, which reduced the running time from 40 minutes to 10 minutes per epoch. However, again, one epoch only takes around 2 minutes with CPU. I am so confused now, it seems like I need to increase the batch size from 1024 to 1024 * 5 so that the running time will be reduced to 2 minutes per epoch.....
Machine Learning & AI General
Nov ’21
Reply to tensorflow-macos cannot be installed on Python 3.8 virtual environment
Hi yuliya80, Thank you for your help. Let me give more details: My macOS version is 12.0.1. the code sample is MINIST toy example, from tensorflow website: https://www.tensorflow.org/datasets/keras_example I tried it again and occurs same error. what I can feel is the tensorflow-macos is still unstable..... but thank you at least give me a direction otherwise I am totally freak out....
Machine Learning & AI General
Nov ’21
Reply to miniforge3 doesn't include sklearn?
I followed the official guideline (https://scikit-learn.org/stable/developers/advanced_installation.html), but somehow still cannot work. so I took almost one day to solve this problem, combining the solutions from: https://github.com/scikit-learn/scikit-learn/issues/19137 and https://github.com/scikit-learn-contrib/hdbscan/issues/457and it works. python -m pip install --no-cache --no-use-pep517 pythran cython pybind11 gast"==0.4.0" pyenv rehash pip install --pre -i https://pypi.anaconda.org/scipy-wheels-nightly/simple scipy python -m pip install --no-use-pep517 scikit-learn --no-binary :all:
Machine Learning & AI General
Nov ’21
Reply to Help! Cannot train any NNs with tensorflow on M1 Chip
ouputs when uncommon following code: from tensorflow.python.framework.ops import disable_eager_execution disable_eager_execution() 2021-10-31 18:54:21.304180: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: SSE4.2 To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. 2021-10-31 18:54:21.305317: I tensorflow/core/common_runtime/pluggable_device/pluggable_device_factory.cc:305] Could not identify NUMA node of platform GPU ID 0, defaulting to 0. Your kernel may not have been built with NUMA support. 2021-10-31 18:54:21.306071: I tensorflow/core/common_runtime/pluggable_device/pluggable_device_factory.cc:271] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 0 MB memory) -> physical PluggableDevice (device: 0, name: METAL, pci bus id: <undefined>) 2021-10-31 18:54:21.392880: I tensorflow/core/common_runtime/pluggable_device/pluggable_device_factory.cc:305] Could not identify NUMA node of platform GPU ID 0, defaulting to 0. Your kernel may not have been built with NUMA support. 2021-10-31 18:54:21.392913: I tensorflow/core/common_runtime/pluggable_device/pluggable_device_factory.cc:271] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 0 MB memory) -> physical PluggableDevice (device: 0, name: METAL, pci bus id: <undefined>) 2021-10-31 18:54:21.486630: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:112] Plugin optimizer for device_type GPU is enabled. 2021-10-31 18:54:21.499261: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:112] Plugin optimizer for device_type GPU is enabled. Metal device set to: Apple M1 Pro systemMemory: 16.00 GB maxCacheSize: 5.33 GB 2021-10-31 18:54:21.560669: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:112] Plugin optimizer for device_type GPU is enabled. 2021-10-31 18:54:21.577903: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:112] Plugin optimizer for device_type GPU is enabled. 2021-10-31 18:54:21.671397: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:112] Plugin optimizer for device_type GPU is enabled. 2021-10-31 18:54:21.690163: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:112] Plugin optimizer for device_type GPU is enabled. 2021-10-31 18:54:21.725452: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:112] Plugin optimizer for device_type GPU is enabled. 2021-10-31 18:54:21.745059: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:112] Plugin optimizer for device_type GPU is enabled. 2021-10-31 18:54:21.762898: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:185] None of the MLIR Optimization Passes are enabled (registered 2) 2021-10-31 18:54:21.782170: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:112] Plugin optimizer for device_type GPU is enabled. 2021-10-31 18:54:21.796932: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:112] Plugin optimizer for device_type GPU is enabled. 2021-10-31 18:54:21.839 python[42307:1235107] -[MPSGraph adamUpdateWithLearningRateTensor:beta1Tensor:beta2Tensor:epsilonTensor:beta1PowerTensor:beta2PowerTensor:valuesTensor:momentumTensor:velocityTensor:gradientTensor:name:]: unrecognized selector sent to instance 0x6000036816c0 Train on 469 steps, validate on 79 steps Epoch 1/10 --------------------------------------------------------------------------- RuntimeError Traceback (most recent call last) /var/folders/mq/pnk708vd76xfjjjmy_4yn2lw0000gn/T/ipykernel_42307/2486947832.py in <module> 58 59 start_time = time.time() ---> 60 model.fit( 61 ds_train, 62 epochs=10, /opt/anaconda3/lib/python3.8/site-packages/keras/engine/training_v1.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, validation_freq, max_queue_size, workers, use_multiprocessing, **kwargs) 775 776 func = self._select_training_loop(x) --> 777 return func.fit( 778 self, 779 x=x, /opt/anaconda3/lib/python3.8/site-packages/keras/engine/training_arrays_v1.py in fit(self, model, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, validation_freq, **kwargs) 638 val_x, val_y, val_sample_weights = None, None, None 639 --> 640 return fit_loop( 641 model, 642 inputs=x, /opt/anaconda3/lib/python3.8/site-packages/keras/engine/training_arrays_v1.py in model_iteration(model, inputs, targets, sample_weights, batch_size, epochs, verbose, callbacks, val_inputs, val_targets, val_sample_weights, shuffle, initial_epoch, steps_per_epoch, validation_steps, validation_freq, mode, validation_in_fit, prepared_feed_values_from_dataset, steps_name, **kwargs) 284 else: 285 actual_inputs = ins() --> 286 batch_outs = f(actual_inputs) 287 except tf.errors.OutOfRangeError: 288 if is_dataset: /opt/anaconda3/lib/python3.8/site-packages/keras/backend.py in __call__(self, inputs) 4027 feed_symbols != self._feed_symbols or self.fetches != self._fetches or 4028 session != self._session): -> 4029 self._make_callable(feed_arrays, feed_symbols, symbol_vals, session) 4030 4031 fetched = self._callable_fn(*array_vals, /opt/anaconda3/lib/python3.8/site-packages/keras/backend.py in _make_callable(self, feed_arrays, feed_symbols, symbol_vals, session) 3963 callable_opts.run_options.CopyFrom(self.run_options) 3964 # Create callable. -> 3965 callable_fn = session._make_callable_from_options(callable_opts) 3966 # Cache parameters corresponding to the generated callable, so that 3967 # we can detect future mismatches and refresh the callable. /opt/anaconda3/lib/python3.8/site-packages/tensorflow/python/client/session.py in _make_callable_from_options(self, callable_options) 1509 """ 1510 self._extend_graph() -> 1511 return BaseSession._Callable(self, callable_options) 1512 1513 /opt/anaconda3/lib/python3.8/site-packages/tensorflow/python/client/session.py in __init__(self, session, callable_options) 1467 compat.as_bytes(callable_options.SerializeToString())) 1468 try: -> 1469 self._handle = tf_session.TF_SessionMakeCallable( 1470 session._session, options_ptr) 1471 finally: RuntimeError: Caught an unknown exception!
Machine Learning & AI General
Oct ’21