Databricks Runtime 10.0 untuk ML (EoS)
Catatan
Dukungan untuk versi Databricks Runtime ini telah berakhir. Untuk tanggal akhir dukungan, lihat Riwayat akhir dukungan. Untuk semua versi Runtime Databricks yang didukung, lihat catatan rilis versi dan kompatibilitas Databricks Runtime.
Databricks Runtime 10.0 untuk Pembelajaran Mesin menyediakan lingkungan siap pakai untuk pembelajaran mesin dan ilmu data berdasarkan Databricks Runtime 10.0 (EoS). Databricks Runtime ML berisi banyak pustaka pembelajaran mesin yang populer, termasuk TensorFlow, PyTorch, dan XGBoost. Ini juga mendukung pelatihan pembelajaran mendalam terdistribusi menggunakan Horovod.
Untuk informasi selengkapnya, termasuk instruksi untuk membuat kluster ML Runtime Databricks, lihat AI dan pembelajaran mesin di Databricks.
Fitur dan peningkatan baru
Databricks Runtime 10.0 ML dibangun di atas Databricks Runtime 10.0. Untuk informasi tentang apa yang baru di Databricks Runtime 10.0, termasuk Apache Spark MLlib dan SparkR, lihat catatan rilis Databricks Runtime 10.0 (EoS).
Peramalan deret waktu dengan AutoML
AutoML sekarang mendukung peramalan deret waktu. untuk informasi selengkapnya, lihat dokumentasi AutoML.
Perubahan besar pada lingkungan Python ML Databricks Runtime
Paket Python yang ditambahkan
- databricks-automl-runtime 0.1.0
- imbalanced-learn 0.8.0
- Transformer 4.9.2
Lingkungan sistem
Kami telah memperbarui versi RStudio Server Open Source yang disertakan ke v1.4.
Lingkungan sistem di Databricks Runtime10.0 ML berbeda dari Databricks Runtime 10.0 sebagai berikut:
-
DBUtils: Databricks Runtime ML tidak menyertakan utilitas pustaka (dbutils.library) (versi lama).
Gunakan perintah
%pipsaja. Lihat Pustaka Python dalam lingkup buku catatan. - Untuk kluster GPU, Databricks Runtime ML menyertakan pustaka GPU NVIDIA berikut:
- CUDA 11.0
- cuDNN 8.0.5.39
- NCCL 2.10.3
- TensorRT 7.2.2
Pustaka
Bagian berikut mencantumkan pustaka yang disertakan dalam Databricks Runtime 10.0 ML yang berbeda dari yang termasuk dalam Databricks Runtime 10.0.
Di bagian ini:
Pustaka tingkat atas
Databricks Runtime 10.0 ML mencakup pustaka unggulan berikut:
- GraphFrames
- Horovod dan HorovodRunner
- MLflow
- PyTorch
- spark-tensorflow-connector
- TensorFlow
- TensorBoard
Pustaka Python
Databricks Runtime 10.0 ML menggunakan Virtualenv untuk manajemen paket Python dan mencakup banyak paket ML populer.
Selain paket yang ditentukan di bagian berikut, Databricks Runtime 10.0 ML juga mencakup paket berikut:
- hyperopt 0.2.5.db2
- sparkdl 2.2.0_db3
- feature_store 0.3.4
- automl 1.3.1
Perpustakaan Python pada kluster CPU
| Perpustakaan | Versi | Perpustakaan | Versi | Perpustakaan | Versi |
|---|---|---|---|---|---|
| absl-py | 0.11.0 | Antergos Linux | 2015.10 (ISO-Rolling) | direktori aplikasi | 1.4.4 |
| argon2-cffi | 20.1.0 | astor | 0.8.1 | astunparse | 1.6.3 |
| generator asinkron | 1.10 | atribut | 20.3.0 | panggilan balik | 0.2.0 |
| bcrypt | 3.2.0 | pemutih | 3.3.0 | kebahagiaan | 0.7.4 |
| boto3 | 1.16.7 | botocore | 1.19.7 | Penyempitan | 1.3.2 |
| cachetools | 4.2.2 | katalog | 2.0.6 | certifi | 5.12.2020 |
| cffi | 1.14.5 | chardet | 4.0.0 | Clang | 5.0 |
| klik | 7.1.2 | cloudpickle | 1.6.0 | cmdstanpy | 0.9.68 |
| configparser | 5.0.1 | convertdate | 2.3.2 | kriptografi | 3.4.7 |
| cycler | 0.10.0 | cymem | 2.0.5 | Cython | 0.29.23 |
| databricks-automl-runtime | 0.1.0 | databricks-cli | 0.14.3 | dbus-python | 1.2.16 |
| dekorator | 5.0.6 | defusedxml | 0.7.1 | dill | 0.3.2 |
| diskcache | 5.2.1 | distlib | 0.3.2 | distro-info | 0.23ubuntu1 |
| titik masuk | 0,3 | ephem | 4.0.0.2 | Ikhtisar Fitur | 1.0.0 |
| filelock | 3.0.12 | Flask | 1.1.2 | flatbuffers | 1.12 |
| fsspec | 0.9.0 | masa depan | 0.18.2 | gast | 0.4.0 |
| gitdb | 4.0.7 | GitPython | 3.1.12 | google-auth | 1.22.1 |
| google-auth-oauthlib | 0.4.2 | google-pasta | 0.2.0 | grpcio | 1.39.0 |
| gunicorn | 20.0.4 | h5py | 3.1.0 | hijri-converter | 2.2.1 |
| hari libur | 0.11.2 | Horovod | 0.22.1 | htmlmin | 0.1.12 |
| huggingface-hub | 0.0.12 | idna | 2.10 | ImageHash | 4.2.1 |
| imbalanced-learn (perangkat lunak Python untuk menangani dataset tidak seimbang) | 0.8.0 | importlib-metadata | 3.10.0 | ipykernel | 5.3.4 |
| ipython | 7.22.0 | ipython-genutils | 0.2.0 | ipywidgets | 7.6.3 |
| isodate | 0.6.0 | pustaka Python 'itsdangerous' | 1.1.0 | jedi | 0.17.2 |
| Jinja2 | 2.11.3 | jmespath | 0.10.0 | joblib | 1.0.1 |
| joblibspark | 0.3.0 | jsonschema | 3.2.0 | jupyter-client | 6.1.12 |
| jupyter-core | 4.7.1 | jupyterlab-pygments | 0.1.2 | jupyterlab-widgets | 1.0.0 |
| keras | 2.6.0 | Keras-Preprocessing | 1.1.2 | kiwisolver | 1.3.1 |
| koalas | 1.8.1 | kalender bulan Korea | 0.2.1 | lightgbm | 3.1.1 |
| llvmlite | 0.37.0 | Kalender Bulan | 0.0.9 | Mako | 1.1.3 |
| Markdown | 3.3.3 | MarkupSafe | 1.1.1 | matplotlib | 3.4.2 |
| missingno | 0.5.0 | mistune | 0.8.4 | mleap | 0.17.0 |
| mlflow-skinny | 1.20.2 | multimethod | 1.4 | murmurhash | 1.0.5 |
| nbclient | 0.5.3 | nbconvert | 6.0.7 | nbformat | 5.1.3 |
| nest-asyncio | 1.5.1 | networkx | 2.5 | nltk | 3.6.1 |
| buku catatan | 6.3.0 | numba | 0.54.0 | numpy | 1.19.2 |
| oauthlib | 3.1.0 | opt-einsum | 3.3.0 | kemasan | 20.9 |
| pandas | 1.2.4 | pandas-profiling | 3.0.0 | pandocfilters | 1.4.3 |
| paramiko | 2.7.2 | parso | 0.7.0 | pathy | 0.6.0 |
| kambing hitam | 0.5.1 | petastorm | 0.11.2 | pexpect=4.8.0 | 4.8.0 |
| phik | 0.12.0 | pickleshare | 0.7.5 | Bantal | 8.2.0 |
| pip | 21.0.1 | plotly | 5.1.0 | Without context, it is difficult to improve the translation accurately. If "preshed" refers to a specific term or concept, further details are needed for a precise translation. | 3.0.5 |
| prometheus-client | 0.10.1 | prompt-toolkit | 3.0.17 | nabi | 1.0.1 |
| protobuf | 3.17.2 | psutil | 5.8.0 | psycopg2 | 2.8.5 |
| ptyprocess | 0.7.0 | pyarrow | 4.0.0 | pyasn1 | 0.4.8 |
| pyasn1-modul | 0.2.8 | pycparser | 2,20 | pydantic | 1.8.2 |
| Pygments | 2.8.1 | PyGObject | 3.36.0 | PyMeeus | 0.5.11 |
| PyNaCl | 1.3.0 | pyodbc | 4.0.30 | pyparsing | 2.4.7 |
| pyrsistent | 0.17.3 | pystan | 2.19.1.1 | python-apt | 2.0.0+ubuntu0.20.4.6 |
| python-dateutil | 2.8.1 | editor Python | 1.0.4 | pytz | 2020.5 |
| PyWavelets | 1.1.1 | PyYAML | 5.4.1 | pyzmq | 20.0.0 |
| regex | 2021.4.4 | permintaan | 2.25.1 | requests-oauthlib | 1.3.0 |
| requests-unixsocket | 0.2.0 | rsa | 4.7.2 | s3transfer | 0.3.7 |
| sacremoses | 0.0.45 | scikit-learn | 0.24.1 | scipy | 1.6.2 |
| seaborn | 0.11.1 | Send2Trash | 1.5.0 | setuptools | 52.0.0 |
| setuptools-git | 1.2 | shap | 0.39.0 | simplejson | 3.17.2 |
| six | 1.15.0 | Pemotong | 0.0.7 | smart-open | 5.2.0 |
| smmap | 3.0.5 | tidak terfokus | 3.1.2 | spacy-legacy | 3.0.8 |
| spark-tensorflow-distributor | 1.0.0 | sqlparse | 0.4.1 | srsly | 2.4.1 |
| ssh-import-id | 5.10 | statsmodels | 0.12.2 | menyusun tabel | 0.8.7 |
| Tersangkut-dalam-unicode | 0.1.0 | keteguhan | 6.2.0 | tensorboard | 2.6.0 |
| tensorboard-data-server | 0.6.1 | tensorboard-plugin-wit | 1.8.0 | tensorflow-cpu | 2.6.0 |
| tensorflow-estimator | 2.6.0 | termcolor | 1.1.0 | selesai | 0.9.4 |
| testpath | 0.4.4 | thinc | 8.0.8 | threadpoolctl | 2.1.0 |
| tokenizers | 0.10.3 | obor/senter | 1.9.0+cpu | torchvision | 0.10.0+cpu |
| tornado | 6.1 | tqdm | 4.59.0 | traitlets=5.0.5 | 5.0.5 |
| Transformer | 4.9.2 | typer | 0.3.2 | typing-extensions | 3.7.4.3 |
| ujson | 4.0.2 | pembaruan-otomatis | 0.1 | urllib3 | 1.25.11 |
| virtualenv | 20.4.1 | visions | 0.7.1 | wasabi | 0.8.2 |
| wcwidth | 0.2.5 | webencodings=0.5.1 | 0.5.1 | websocket-client | 0.57.0 |
| Werkzeug | 1.0.1 | roda | 0.36.2 | widgetsnbextension | 3.5.1 |
| terbungkus | 1.12.1 | xgboost | 1.4.2 | zipp | 3.4.1 |
Pustaka Phyton di kluster GPU
| Perpustakaan | Versi | Perpustakaan | Versi | Perpustakaan | Versi |
|---|---|---|---|---|---|
| absl-py | 0.11.0 | Antergos Linux | 2015.10 (ISO-Rolling) | appdirs | 1.4.4 |
| argon2-cffi | 20.1.0 | astor | 0.8.1 | astunparse | 1.6.3 |
| async-generator | 1.10 | atribut | 20.3.0 | panggilan balik | 0.2.0 |
| bcrypt | 3.2.0 | pemutih | 3.3.0 | kebahagiaan | 0.7.4 |
| boto3 | 1.16.7 | botocore | 1.19.7 | Penyempitan | 1.3.2 |
| cachetools | 4.2.2 | katalog | 2.0.6 | sertifikat | 5.12.2020 |
| cffi | 1.14.5 | chardet | 4.0.0 | Clang | 5.0 |
| klik | 7.1.2 | cloudpickle | 1.6.0 | cmdstanpy | 0.9.68 |
| configparser | 5.0.1 | convertdate | 2.3.2 | kriptografi | 3.4.7 |
| cycler | 0.10.0 | cymem | 2.0.5 | Cython | 0.29.23 |
| databricks-automl-runtime | 0.1.0 | databricks-cli | 0.14.3 | dbus-python | 1.2.16 |
| Dekorator | 5.0.6 | defusedxml | 0.7.1 | adas | 0.3.2 |
| diskcache | 5.2.1 | distlib | 0.3.2 | distro-info | 0.23ubuntu1 |
| titik masuk | 0,3 | ephem | 4.0.0.2 | Ikhtisar-aspek | 1.0.0 |
| filelock | 3.0.12 | Flask | 1.1.2 | flatbuffers | 1.12 |
| fsspec | 0.9.0 | masa depan | 0.18.2 | gast | 0.4.0 |
| gitdb | 4.0.7 | GitPython | 3.1.12 | google-auth | 1.22.1 |
| google-auth-oauthlib | 0.4.2 | Google-pasta | 0.2.0 | grpcio | 1.39.0 |
| gunicorn | 20.0.4 | h5py | 3.1.0 | hijri-converter | 2.2.1 |
| hari libur | 0.11.2 | Horovod | 0.22.1 | htmlmin | 0.1.12 |
| huggingface-hub | 0.0.12 | idna | 2.10 | Pengkodean Gambar | 4.2.1 |
| imbalanced-learn | 0.8.0 | importlib-metadata | 3.10.0 | ipykernel | 5.3.4 |
| ipython | 7.22.0 | ipython-genutils | 0.2.0 | ipywidgets | 7.6.3 |
| isodate | 0.6.0 | berbahaya | 1.1.0 | jedi | 0.17.2 |
| Jinja2 | 2.11.3 | jmespath | 0.10.0 | joblib | 1.0.1 |
| joblibspark | 0.3.0 | jsonschema | 3.2.0 | jupyter-client | 6.1.12 |
| jupyter-core | 4.7.1 | jupyterlab-pygments | 0.1.2 | jupyterlab-widgets | 1.0.0 |
| keras | 2.6.0 | Keras-Preprocessing | 1.1.2 | kiwisolver | 1.3.1 |
| koalas | 1.8.1 | kalender-lunar-korea | 0.2.1 | lightgbm | 3.1.1 |
| llvmlite | 0.37.0 | Kalender Lunar | 0.0.9 | Mako | 1.1.3 |
| Markdown | 3.3.3 | MarkupSafe | 1.1.1 | matplotlib | 3.4.2 |
| missingno | 0.5.0 | Mistune | 0.8.4 | mleap | 0.17.0 |
| mlflow-skinny | 1.20.2 | multimetode | 1.4 | murmurhash | 1.0.5 |
| nbclient | 0.5.3 | nbconvert | 6.0.7 | nbformat | 5.1.3 |
| nest-asyncio | 1.5.1 | networkx | 2.5 | nltk | 3.6.1 |
| buku catatan | 6.3.0 | numba | 0.54.0 | numpy | 1.19.2 |
| oauthlib | 3.1.0 | opt-einsum | 3.3.0 | kemasan | 20.9 |
| pandas | 1.2.4 | pandas-profiling | 3.0.0 | pandocfilters | 1.4.3 |
| paramiko | 2.7.2 | parso | 0.7.0 | -patia | 0.6.0 |
| kambing hitam | 0.5.1 | petastorm | 0.11.2 | pexpect=4.8.0 | 4.8.0 |
| phik | 0.12.0 | pickleshare | 0.7.5 | Bantal | 8.2.0 |
| pip | 21.0.1 | plotly | 5.1.0 | preshed | 3.0.5 |
| prompt-toolkit | 3.0.17 | nabi | 1.0.1 | protobuf | 3.17.2 |
| psutil | 5.8.0 | psycopg2 | 2.8.5 | ptyprocess | 0.7.0 |
| pyarrow | 4.0.0 | pyasn1 | 0.4.8 | pyasn1-modul | 0.2.8 |
| pycparser | 2,20 | pydantic | 1.8.2 | Pygments | 2.8.1 |
| PyGObject | 3.36.0 | PyMeeus | 0.5.11 | PyNaCl | 1.3.0 |
| pyodbc | 4.0.30 | pyparsing | 2.4.7 | pyrsistent | 0.17.3 |
| pystan | 2.19.1.1 | python-apt | 2.0.0+ubuntu0.20.4.6 | python-dateutil | 2.8.1 |
| python-editor | 1.0.4 | pytz | 2020.5 | PyWavelets | 1.1.1 |
| PyYAML | 5.4.1 | pyzmq | 20.0.0 | regex | 2021.4.4 |
| permintaan | 2.25.1 | requests-oauthlib | 1.3.0 | requests-unixsocket | 0.2.0 |
| rsa | 4.7.2 | s3transfer | 0.3.7 | sacremoses | 0.0.45 |
| scikit-learn | 0.24.1 | scipy | 1.6.2 | seaborn | 0.11.1 |
| Send2Trash | 1.5.0 | setuptools | 52.0.0 | setuptools-git | 1.2 |
| shap | 0.39.0 | simplejson | 3.17.2 | six | 1.15.0 |
| pengiris | 0.0.7 | smart-open | 5.2.0 | smmap | 3.0.5 |
| linglung | 3.1.2 | spacy-legacy | 3.0.8 | spark-tensorflow-distributor | 1.0.0 |
| sqlparse (pustaka Python untuk mem-parsing kueri SQL) | 0.4.1 | srsly | 2.4.1 | ssh-import-id | 5.10 |
| statsmodels | 0.12.2 | menyusun tabel | 0.8.7 | terjebak-dalam-unicode | 0.1.0 |
| kegigihan | 6.2.0 | tensorboard | 2.6.0 | tensorboard-data-server | 0.6.1 |
| tensorboard-plugin-wit | 1.8.0 | tensorflow | 2.6.0 | tensorflow-estimator | 2.6.0 |
| termcolor | 1.1.0 | selesai | 0.9.4 | testpath | 0.4.4 |
| thinc | 8.0.8 | threadpoolctl | 2.1.0 | penghitung token | 0.10.3 |
| obor | 1.9.0+cu111 | torchvision | 0.10.0+cu111 | tornado | 6.1 |
| tqdm | 4.59.0 | Traitlets versi 5.0.5 | 5.0.5 | Transformer | 4.9.2 |
| typer | 0.3.2 | typing-extensions | 3.7.4.3 | ujson | 4.0.2 |
| pembaruan-otomatis | 0.1 | urllib3 | 1.25.11 | virtualenv | 20.4.1 |
| visions | 0.7.1 | wasabi | 0.8.2 | wcwidth | 0.2.5 |
| webencodings | 0.5.1 | websocket-client | 0.57.0 | Werkzeug | 1.0.1 |
| roda | 0.36.2 | widgetsnbextension | 3.5.1 | dibalut | 1.12.1 |
| xgboost | 1.4.2 | zipp | 3.4.1 |
Paket Spark yang berisi modul Python
| Paket Spark | Modul Python | Versi |
|---|---|---|
| graphframes | graphframes | 0.8.1-db6-spark3.2 |
Perpustakaan R
Pustaka R identik dengan Pustaka R di Databricks Runtime 10.0.
Pustaka Java dan Scala (Kluster Scala 2.12)
Selain pustaka Java dan Scala di Databricks Runtime 10.0, Databricks Runtime 10.0 ML berisi JAR berikut:
Kluster CPU
| ID Grup | ID Artefak | Versi |
|---|---|---|
| com.typesafe.akka | akka-actor_2.12 | 2.5.23 |
| ml.combust.mleap | mleap-databricks-runtime_2.12 | 0.17.0-4882dc3 |
| ml.dmlc | xgboost4j-spark_2.12 | 1.4.1 |
| ml.dmlc | xgboost4j_2.12 | 1.4.1 |
| org.graphframes | graphframes_2.12 | 0.8.1-db6-spark3.2 |
| org.mlflow | mlflow-client | 1.20.2 |
| org.mlflow | mlflow-spark | 1.20.2 |
| org.scala-lang.modules | scala-java8-compat_2.12 | 0.8.0 |
| org.tensorflow | spark-tensorflow-connector_2.12 | 1.15.0 |
Kluster GPU
| ID Grup | ID Artefak | Versi |
|---|---|---|
| com.typesafe.akka | akka-actor_2.12 | 2.5.23 |
| ml.combust.mleap | mleap-databricks-runtime_2.12 | 0.17.0-4882dc3 |
| ml.dmlc | xgboost4j-gpu_2.12 | 1.4.1 |
| ml.dmlc | xgboost4j-spark-gpu_2.12 | 1.4.1 |
| org.graphframes | graphframes_2.12 | 0.8.1-db6-spark3.2 |
| org.mlflow | mlflow-client | 1.20.2 |
| org.mlflow | mlflow-spark | 1.20.2 |
| org.scala-lang.modules | scala-java8-compat_2.12 | 0.8.0 |
| org.tensorflow | spark-tensorflow-connector_2.12 | 1.15.0 |