Sklearn is unstable on Apple Silicon

Question

dkjdjdfdskln OP

Created Nov ’21

Replies 6

Boosts 1

Views 7.6k

Participants 10

Hi,

I installed skearn successfully and ran the MINIST toy example successfully.

then I started to run my project. The finning thing everything seems good at the start point (at least no ImportError occurs). but when I made some changes of my code and try to run all cells (I use jupyter lab) again, ImportError occurs.....

ImportError: dlopen(/Users/a/miniforge3/lib/python3.9/site-packages/scipy/spatial/qhull.cpython-39-darwin.so, 0x0002): Library not loaded: @rpath/liblapack.3.dylib

  Referenced from: /Users/a/miniforge3/lib/python3.9/site-packages/scipy/spatial/qhull.cpython-39-darwin.so

  Reason: tried: '/Users/a/miniforge3/lib/liblapack.3.dylib' (no such file), '/Users/a/miniforge3/lib/liblapack.3.dylib' (no such file), '/Users/a/miniforge3/lib/python3.9/site-packages/scipy/spatial/../../../../liblapack.3.dylib' (no such file), '/Users/a/miniforge3/lib/liblapack.3.dylib' (no such file), '/Users/a/miniforge3/lib/liblapack.3.dylib' (no such file), '/Users/a/miniforge3/lib/python3.9/site-packages/scipy/spatial/../../../../liblapack.3.dylib' (no such file), '/Users/a/miniforge3/lib/liblapack.3.dylib' (no such file), '/Users/a/miniforge3/bin/../lib/liblapack.3.dylib' (no such file), '/Users/a/miniforge3/lib/liblapack.3.dylib' (no such file), '/Users/a/miniforge3/bin/../lib/liblapack.3.dylib' (no such file), '/usr/local/lib/liblapack.3.dylib' (no such file), '/usr/lib/liblapack.3.dylib' (no such file)

then I have to uninstall scipy, sklearn, etc and reinstall all of them. and my code can be ran again.....

Magically I hate to say, anyone knows how to permanently solve this problem? make skearn more stable?

Boost

Answer 1

Frameworks Engineer OP

Apple

Nov ’21

Hi,

Can you paste pip list when it runs successfully vs when it fails ? Also in your Jupyter cells, are you installing any packages or changing PYTHONPATH? If you could provide a small repro case, that might help give some idea.

0

Answer 2

dkjdjdfdskln OP

Nov ’21

Here is the pip list. and I cannot run python -c "import sklearn" on my terminal.

Package Version

absl-py 0.10.0

aiohttp 3.7.4.post0

anyio 2.2.0

appnope 0.1.2

argon2-cffi 20.1.0

astroid 2.6.6

astunparse 1.6.3

async-generator 1.10

async-timeout 3.0.1

attrs 21.2.0

autopep8 1.5.6

Babel 2.9.1

backcall 0.2.0

beniget 0.3.0

biopython 1.79

bleach 4.0.0

blinker 1.4

Bottleneck 1.3.2

brotlipy 0.7.0

cached-property 1.5.2

cachetools 4.2.2

certifi 2021.5.30

cffi 1.14.5

chardet 4.0.0

charset-normalizer 2.0.4

clang 5.0

click 8.0.3

cloudpickle 2.0.0

conda 4.10.3

conda-package-handling 1.7.3

cryptography 3.4.7

cycler 0.11.0

Cython 0.29.24

decorator 5.1.0

defusedxml 0.7.1

deprecation 2.1.0

dill 0.3.4

entrypoints 0.3

et-xmlfile 1.1.0

flake8 3.9.0

flatbuffers 1.12

future 0.18.2

gast 0.4.0

google-auth 1.33.0

google-auth-oauthlib 0.4.1

google-pasta 0.2.0

googleapis-common-protos 1.53.0

grpcio 1.38.1

gym 0.21.0

h5py 3.1.0

idna 3.2

importlib-metadata 4.8.1

ipykernel 5.3.4

ipython 7.26.0

ipython-genutils 0.2.0

isort 5.9.3

jdcal 1.4.1

jedi 0.17.2

Jinja2 3.0.2

joblib 1.1.0

json5 0.9.6

jsonschema 3.2.0

jupyter-client 7.0.1

jupyter-core 4.7.1

jupyter-http-over-ws 0.0.8

jupyter-lsp 1.5.0

jupyter-packaging 0.10.4

jupyter-server 1.6.4

jupyterlab 3.1.7

jupyterlab-lsp 3.9.1

jupyterlab-pygments 0.1.2

jupyterlab-server 2.8.2

keras 2.6.0

Keras-Preprocessing 1.1.2

kiwisolver 1.3.2

lazy-object-proxy 1.6.0

Markdown 3.3.4

MarkupSafe 2.0.1

matplotlib 3.4.2

matplotlib-inline 0.1.2

mccabe 0.6.1

mistune 0.8.4

msgpack 1.0.2

msgpack-numpy 0.4.7.1

multidict 5.1.0

nbclassic 0.2.6

nbclient 0.5.3

nbconvert 6.1.0

nbformat 5.1.3

nest-asyncio 1.5.1

networkx 2.6.3

notebook 6.4.5

numexpr 2.7.3

numpy 1.19.5

oauthlib 3.1.1

olefile 0.46

openpyxl 3.0.5

opt-einsum 3.3.0

packaging 21.0

pandas 1.3.1

pandocfilters 1.4.3

parso 0.7.0

pexpect 4.8.0

pickleshare 0.7.5

Pillow 8.4.0

pip 21.3

pluggy 0.13.1

ply 3.11

prometheus-client 0.11.0

promise 2.3

prompt-toolkit 3.0.20

protobuf 3.17.2

ptyprocess 0.7.0

pyasn1 0.4.8

pyasn1-modules 0.2.8

pybind11 2.8.1

pycodestyle 2.6.0

pycosat 0.6.3

pycparser 2.20

pydocstyle 6.1.1

pyflakes 2.2.0

pyglet 1.5.21

Pygments 2.10.0

PyJWT 2.1.0

pylint 2.9.6

pyOpenSSL 21.0.0

pyparsing 3.0.4

pyrsistent 0.18.0

PySocks 1.7.1

python-dateutil 2.8.2

python-jsonrpc-server 0.4.0

python-language-server 0.36.2

pythran 0.9.11

pytz 2021.3

pyzmq 20.0.0

requests 2.26.0

requests-oauthlib 1.3.0

rope 0.19.0

rsa 4.7.2

ruamel-yaml-conda 0.15.100

scikit-learn 1.0.1

scipy 1.8.0.dev0+1921.c30fefc

seaborn 0.11.2

Send2Trash 1.8.0

setuptools 52.0.0.post20210825

six 1.15.0

sniffio 1.2.0

snowballstemmer 2.1.0

tensorboard 2.6.0

tensorboard-data-server 0.6.0

tensorboard-plugin-wit 1.6.0

tensorflow-datasets 4.4.0

tensorflow-estimator 2.7.0

tensorflow-macos 2.6.0

tensorflow-metadata 1.4.0

tensorflow-metal 0.2.0

Tensorforce 0.6.5

termcolor 1.1.0

terminado 0.9.4

testpath 0.5.0

threadpoolctl 3.0.0

toml 0.10.2

tomlkit 0.7.0

tornado 6.1

tqdm 4.62.3

traitlets 5.1.0

typing-extensions 3.7.4.3

ujson 4.0.2

urllib3 1.26.7

wcwidth 0.2.5

webencodings 0.5.1

Werkzeug 2.0.2

wheel 0.35.1

wrapt 1.12.1

yapf 0.31.0

yarl 1.6.3

zipp 3.6.0

0

Answer 3

dkjdjdfdskln OP

Nov ’21

I reinstalled the sklearn and scipy, after that sklearn can be imported correctly.

Package Version

absl-py 0.10.0

aiohttp 3.7.4.post0

anyio 2.2.0

appnope 0.1.2

argon2-cffi 20.1.0

astroid 2.6.6

astunparse 1.6.3

async-generator 1.10

async-timeout 3.0.1

attrs 21.2.0

autopep8 1.5.6

Babel 2.9.1

backcall 0.2.0

beniget 0.3.0

biopython 1.79

bleach 4.0.0

blinker 1.4

Bottleneck 1.3.2

brotlipy 0.7.0

cached-property 1.5.2

cachetools 4.2.2

certifi 2021.5.30

cffi 1.14.5

chardet 4.0.0

charset-normalizer 2.0.4

clang 5.0

click 8.0.3

cloudpickle 2.0.0

conda 4.10.3

conda-package-handling 1.7.3

cryptography 3.4.7

cycler 0.11.0

Cython 0.29.24

decorator 5.1.0

defusedxml 0.7.1

deprecation 2.1.0

dill 0.3.4

entrypoints 0.3

et-xmlfile 1.1.0

flake8 3.9.0

flatbuffers 1.12

future 0.18.2

gast 0.4.0

google-auth 1.33.0

google-auth-oauthlib 0.4.1

google-pasta 0.2.0

googleapis-common-protos 1.53.0

grpcio 1.38.1

gym 0.21.0

h5py 3.1.0

idna 3.2

importlib-metadata 4.8.1

ipykernel 5.3.4

ipython 7.26.0

ipython-genutils 0.2.0

isort 5.9.3

jdcal 1.4.1

jedi 0.17.2

Jinja2 3.0.2

joblib 1.1.0

json5 0.9.6

jsonschema 3.2.0

jupyter-client 7.0.1

jupyter-core 4.7.1

jupyter-http-over-ws 0.0.8

jupyter-lsp 1.5.0

jupyter-packaging 0.10.4

jupyter-server 1.6.4

jupyterlab 3.1.7

jupyterlab-lsp 3.9.1

jupyterlab-pygments 0.1.2

jupyterlab-server 2.8.2

keras 2.6.0

Keras-Preprocessing 1.1.2

kiwisolver 1.3.2

lazy-object-proxy 1.6.0

Markdown 3.3.4

MarkupSafe 2.0.1

matplotlib 3.4.2

matplotlib-inline 0.1.2

mccabe 0.6.1

mistune 0.8.4

msgpack 1.0.2

msgpack-numpy 0.4.7.1

multidict 5.1.0

nbclassic 0.2.6

nbclient 0.5.3

nbconvert 6.1.0

nbformat 5.1.3

nest-asyncio 1.5.1

networkx 2.6.3

notebook 6.4.5

numexpr 2.7.3

numpy 1.19.5

oauthlib 3.1.1

olefile 0.46

openpyxl 3.0.5

opt-einsum 3.3.0

packaging 21.0

pandas 1.3.1

pandocfilters 1.4.3

parso 0.7.0

pexpect 4.8.0

pickleshare 0.7.5

Pillow 8.4.0

pip 21.3

pluggy 0.13.1

ply 3.11

prometheus-client 0.11.0

promise 2.3

prompt-toolkit 3.0.20

protobuf 3.17.2

ptyprocess 0.7.0

pyasn1 0.4.8

pyasn1-modules 0.2.8

pybind11 2.8.1

pycodestyle 2.6.0

pycosat 0.6.3

pycparser 2.20

pydocstyle 6.1.1

pyflakes 2.2.0

pyglet 1.5.21

Pygments 2.10.0

PyJWT 2.1.0

pylint 2.9.6

pyOpenSSL 21.0.0

pyparsing 3.0.4

pyrsistent 0.18.0

PySocks 1.7.1

python-dateutil 2.8.2

python-jsonrpc-server 0.4.0

python-language-server 0.36.2

pythran 0.9.11

pytz 2021.3

pyzmq 20.0.0

requests 2.26.0

requests-oauthlib 1.3.0

rope 0.19.0

rsa 4.7.2

ruamel-yaml-conda 0.15.100

scikit-learn 1.0.1

scipy 1.7.1

seaborn 0.11.2

Send2Trash 1.8.0

setuptools 52.0.0.post20210825

six 1.15.0

sniffio 1.2.0

snowballstemmer 2.1.0

tensorboard 2.6.0

tensorboard-data-server 0.6.0

tensorboard-plugin-wit 1.6.0

tensorflow-datasets 4.4.0

tensorflow-estimator 2.7.0

tensorflow-macos 2.6.0

tensorflow-metadata 1.4.0

tensorflow-metal 0.2.0

Tensorforce 0.6.5

termcolor 1.1.0

terminado 0.9.4

testpath 0.5.0

threadpoolctl 3.0.0

toml 0.10.2

tomlkit 0.7.0

tornado 6.1

tqdm 4.62.3

traitlets 5.1.0

typing-extensions 3.7.4.3

ujson 4.0.2

urllib3 1.26.7

wcwidth 0.2.5

webencodings 0.5.1

Werkzeug 2.0.2

wheel 0.35.1

wrapt 1.12.1

yapf 0.31.0

yarl 1.6.3

zipp 3.6.0

0

Answer 4

mecil9 OP

Feb ’22

did u fix it mistake? i meet the same alert message。how to fix it？

`ImportError: dlopen(/Users/mecilmeng/miniforge3/envs/tf/lib/python3.9/site-packages/scipy/spatial/qhull.cpython-39-darwin.so, 0x0002): Library not loaded: @rpath/liblapack.3.dylib Referenced from: /Users/mecilmeng/miniforge3/envs/tf/lib/python3.9/site-packages/scipy/spatial/qhull.cpython-39-darwin.so Reason: tried: '/Users/mecilmeng/miniforge3/envs/tf/lib/liblapack.3.dylib' (no such file), '/Users/mecilmeng/miniforge3/envs/tf/lib/liblapack.3.dylib' (no such file), '/Users/mecilmeng/miniforge3/envs/tf/lib/python3.9/site-packages/scipy/spatial/../../../../liblapack.3.dylib' (no such file), '/Users/mecilmeng/miniforge3/envs/tf/lib/liblapack.3.dylib' (no such file), '/Users/mecilmeng/miniforge3/envs/tf/lib/liblapack.3.dylib' (no such file), '/Users/mecilmeng/miniforge3/envs/tf/lib/python3.9/site-packages/scipy/spatial/../../../../liblapack.3.dylib' (no such file), '/Users/mecilmeng/miniforge3/envs/tf/bin/../lib/liblapack.3.dylib' (no such file), '/Users/mecilmeng/miniforge3/envs/tf/bin/../lib/liblapack.3.dylib' (no such file), '/usr/local/lib/liblapack.3.dylib' (no such file), '/usr/lib/liblapack.3.dylib' (no such file)

0

Answer 5

enebin OP

Feb ’22

Hi. I've been suffering from the same problem but I finally found a simple solution which worked for me.

try this one:

pip install --upgrade --force-reinstall scikit-learn

I tested on Miniforge environment(macOS 12.2.1, M1 Air) and problem solved.

12

Answer 6

fasmb24 OP

Oct ’23

NOTE: This answer involves copying dilyb(s) that may not be compatible all versions of sklearn. I highly recommend trying to re-install the packages first. If you continue to fail and have reached a point where you are willing to try anything, the following steps may resolve the issue.

I was able to resolve this by manually copying over a copy of the necessary dylib(s) out of a homebrew installation. First: I ran brew install lapack [NOTE: For me, this installed version (3.11), you may end up with a different version after running the command] Then I identified all the relevant files in of /opt/homebrew/Cellar/lapack/3.11/lib/*:

liblapacke.dylib
libblas.3.11.0.dylib
libblas.3.dylib
libblas.dylib
liblapack.3.11.0.dylib
liblapack.3.dylib
liblapack.dylib
liblapacke.3.11.0.dylib
liblapacke.3.dylib

Identify a search path that I all my python packages would search. I'm using miniconda for my python environment management, for me it was: ~/miniconda3/envs/maxEnt/lib/.

Step 1: Delete all existing files with a matching name in this lib folder.

Step 2: Copy the files referenced above.

Step 3: Update all the symlinks (aliases) in these files [.dylib points to 3.11.0.dylib; .3.dylib points to 3.11.0.dylib].

0