Databricks Runtime 11.0 untuk Pembelajaran Mesin (EoS)
Catatan
Dukungan untuk versi Databricks Runtime ini telah berakhir. Untuk tanggal akhir dukungan, lihat Riwayat akhir dukungan. Untuk semua versi Runtime Databricks yang didukung, lihat catatan rilis versi dan kompatibilitas Databricks Runtime.
Databricks Runtime 11.0 untuk Pembelajaran Mesin menyediakan lingkungan siap pakai untuk pembelajaran mesin dan ilmu data berdasarkan Databricks Runtime 11.0 (EoS). Databricks Runtime ML berisi banyak pustaka pembelajaran mesin yang populer, termasuk TensorFlow, PyTorch, dan XGBoost. Databricks Runtime ML mencakup AutoML, alat untuk secara otomatis melatih alur pembelajaran mesin. Databricks Runtime ML juga mendukung pelatihan pembelajaran mendalam menggunakan Horovod.
Untuk informasi selengkapnya, termasuk instruksi untuk membuat kluster ML Runtime Databricks, lihat AI dan pembelajaran mesin di Databricks.
Fitur dan peningkatan baru
Databricks Runtime 11.0 ML dibangun di atas Databricks Runtime 11.0. Untuk informasi tentang apa yang baru di Databricks Runtime 11.0, termasuk Apache Spark MLlib dan SparkR, lihat catatan rilis Databricks Runtime 11.0 (EoS).
Peningkatan AutoML
AutoML sekarang mendukung pecahan pengambilan sampel yang lebih tinggi untuk performa yang lebih baik pada himpunan data besar. Untuk detailnya, lihat Mengambil sampel himpunan data besar.
Lingkungan sistem
Lingkungan sistem di Databricks Runtime 11.0 ML berbeda dari Databricks Runtime 11.0 sebagai berikut:
-
DBUtils: Databricks Runtime ML tidak menyertakan utility perpustakaan (dbutils.library) (versi lawas).
Gunakan perintah
%pipsaja. Lihat Pustaka Python cakupan notebook. - Untuk kluster GPU, Databricks Runtime ML menyertakan pustaka GPU NVIDIA berikut:
- CUDA 11.3
- cuDNN 8.0.5.39
- NCCL 2.9.9
- TensorRT 7.2.2
Pustaka
Bagian berikut mencantumkan pustaka yang disertakan dalam Databricks Runtime 11.0 ML yang berbeda dari yang disertakan dalam Databricks Runtime 11.0.
Di bagian ini:
Pustaka tingkat atas
Databricks Runtime 11.0 ML menyertakan pustaka tingkat atas berikut:
- GraphFrames
- Horovod dan HorovodRunner
- MLflow
- PyTorch
- spark-tensorflow-connector
- TensorFlow
- TensorBoard
Pustaka Python
Databricks Runtime 11.0 ML menggunakan Virtualenv untuk manajemen paket Python dan mencakup banyak paket ML populer.
Selain paket yang ditentukan di bagian berikut, Databricks Runtime 11.0 ML juga mencakup paket berikut:
- hyperopt 0.2.7.db1
- sparkdl 2.2.0-db6
- feature_store 0.4.1
- automl 1.10
Pustaka Python pada kluster CPU
| Perpustakaan | Versi | Pustaka | Versi | Perpustakaan | Versi |
|---|---|---|---|---|---|
| absl-py | 1.0.0 | Antergos Linux | 2015.10 (ISO-Rolling) | argon2-cffi | 20.1.0 |
| astor | 0.8.1 | astunparse | 1.6.3 | async-generator | 1.10 |
| atribut | 21.2.0 | panggilan balasan | 0.2.0 | backports.entry-points-selectable | 1.1.1 |
| bcrypt | 3.2.2 | pemutih | 4.0.0 | blis | 0.7.7 |
| boto3 | 1.21.18 | botocore | 1.24.18 | cachetools | 5.1.0 |
| katalog | 2.0.7 | sertifikat | 2021.10.8 | cffi | 1.14.6 |
| chardet | 4.0.0 | charset-normalizer | 2.0.4 | klik | 8.0.3 |
| cloudpickle (perpustakaan Python untuk serialisasi objek) | 2.0.0 | cmdstanpy | 0.9.68 | configparser | 5.2.0 |
| convertdate | 2.4.0 | kriptografi | 3.4.8 | cycler | 0.10.0 |
| cymem | 2.0.6 | Cython | 0.29.24 | databricks-automl-runtime | 0.2.8 |
| databricks-cli | 0.16.4 | dbl-tempo | 0.1.12 | dbus-python | 1.2.16 |
| debugpy | 1.4.1 | penghias | 5.1.0 | defusedxml | 0.7.1 |
| dill | 0.3.4 | diskcache | 5.4.0 | distlib | 0.3.4 |
| distro-info | 0.23ubuntu1 | titik masuk | 0,3 | ephem | 4.1.3 |
| Ikhtisar Fitur | 1.0.0 | fasttext | 0.9.2 | filelock | 3.3.1 |
| Flask | 1.1.2 | flatbuffers | 2.0 | fsspec | 2021.8.1 |
| masa depan | 0.18.2 | gast | 0.5.3 | gitdb | 4.0.9 |
| GitPython | 3.1.27 | google-auth | 2.6.0 | google-auth-oauthlib | 0.4.6 |
| google-pasta | 0.2.0 | grpcio | 1.44.0 | gunicorn | 20.1.0 |
| gviz-api | 1.10.0 | h5py | 3.3.0 | hijri-converter | 2.2.3 |
| hari libur | 0,13 | Horovod | 0.24.3 | htmlmin | 0.1.12 |
| huggingface-hub | 0.6.0 | idna | 3.2 | ImageHash | 4.2.1 |
| imbalanced-learn | 0.8.1 | importlib-metadata | 4.8.1 | ipykernel | 6.12.1 |
| ipython | 7.32.0 | ipython-genutils | 0.2.0 | ipywidgets | 7.7.0 |
| isodate | 0.6.1 | itsdangerous | 2.0.1 | jedi | 0.18.0 |
| Jinja2 | 2.11.3 | jmespath | 0.10.0 | joblib | 1.0.1 |
| joblibspark | 0.5.0 | jsonschema | 3.2.0 | jupyter-client | 6.1.12 |
| jupyter-core | 4.8.1 | jupyterlab-pygments | 0.1.2 | jupyterlab-widgets | 1.0.0 |
| keras | 2.8.0 | Keras-Preprocessing | 1.1.2 | kiwisolver | 1.3.1 |
| kalender lunar Korea | 0.2.1 | kode bahasa | 3.3.0 | libclang | 14.0.1 |
| lightgbm | 3.3.2 | llvmlite | 0.38.0 | Kalender Lunar | 0.0.9 |
| Mako | 1.2.0 | Markdown | 3.3.6 | MarkupSafe | 2.0.1 |
| matplotlib | 3.4.3 | matplotlib-inline | 0.1.2 | missingno | 0.5.1 |
| menyetel salah | 0.8.4 | mleap | 0.20.0 | mlflow-skinny | 1,26,0 |
| multimetode | 1.8 | murmurhash | 1.0.7 | nbclient | 0.5.3 |
| nbconvert | 6.1.0 | nbformat | 5.1.3 | nest-asyncio | 1.5.1 |
| networkx | 2.6.3 | nltk | 3.6.5 | buku catatan | 6.4.5 |
| numba | 0.55.1 | numpy | 1.20.3 | oauthlib | 3.2.0 |
| opt-einsum | 3.3.0 | pengemasan | 21.0 | pandas | 1.3.4 |
| pandas-profiling | 3.1.0 | pandocfilters | 1.4.3 | paramiko | 2.9.2 |
| parso | 0.8.2 | pati | 0.6.1 | orang bodoh | 0.5.2 |
| petastorm | 0.11.4 | pexpect 4.8.0 | 4.8.0 | phik | 0.12.2 |
| pickleshare | 0.7.5 | Bantal | 8.4.0 | pip | 21.2.4 |
| platformdirs | 2.5.2 | plotly | 5.6.0 | pmdarima | 1.8.5 |
| preshed | 3.0.6 | prometheus-client | 0.11.0 | prompt-toolkit | 3.0.20 |
| nabi | 1.0.1 | protobuf | 3.19.4 | psutil | 5.8.0 |
| psycopg2 | 2.9.3 | ptyprocess | 0.7.0 | pyarrow | 7.0.0 |
| pyasn1 | 0.4.8 | pyasn1-modul | 0.2.8 | pybind11 | 2.9.2 |
| pycparser | 2,20 | pydantic | 1.8.2 | Pygments | 2.10.0 |
| PyGObject | 3.36.0 | PyMeeus | 0.5.11 | PyNaCl | 1.5.0 |
| pyodbc | 4.0.31 | pyparsing | 3.0.4 | pyrsistent | 0.18.0 |
| pystan | 2.19.1.1 | python-apt | 2.0.0+ubuntu0.20.4.7 | python-dateutil | 2.8.2 |
| python-editor | 1.0.4 | pytz | 2021.3 | PyWavelets | 1.1.1 |
| PyYAML | 6.0 | pyzmq | 22.2.1 | regex | 2021.8.3 |
| permintaan | 2.26.0 | requests-oauthlib | 1.3.1 | requests-unixsocket | 0.2.0 |
| rsa | 4.8 | s3transfer | 0.5.2 | sacremoses | 0.0.53 |
| scikit-learn | 0.24.2 | scipy | 1.7.1 | seaborn | 0.11.2 |
| Send2Trash | 1.8.0 | setuptools | 58.0.4 | setuptools-git | 1.2 |
| shap | 0.40.0 | simplejson | 3.17.6 | six | 1.16.0 |
| pemotong | 0.0.7 | smart-open | 5.2.1 | smmap | 5.0.0 |
| lapang | 3.2.3 | spacy-legacy | 3.0.9 | spacy-loggers | 1.0.2 |
| spark-tensorflow-distributor | 1.0.0 | sqlparse | 0.4.2 | srsly | 2.4.3 |
| ssh-import-id | 5.10 | statsmodels | 0.12.2 | tabulasi | 0.8.9 |
| terjerat-dalam-unicode | 0.1.0 | kegigihan | 8.0.1 | tensorboard | 2.8.0 |
| tensorboard-data-server | 0.6.1 | tensorboard-plugin-profile | 2.5.0 | tensorboard-plugin-wit | 1.8.1 |
| tensorflow-cpu | 2.8.0 | tensorflow-estimator | 2.8.0 | tensorflow-io-gcs-filesystem | 0.25.0 |
| termcolor | 1.1.0 | selesai | 0.9.4 | jalur pengujian | 0.5.0 |
| tf-estimator-nightly | 2.8.0.dev2021122109 | thinc | 8.0.15 | threadpoolctl | 2.2.0 |
| tokenizers | 0.12.1 | obor/senter | 1.11.0+cpu | torchvision | 0.12.0+cpu |
| tornado | 6.1 | tqdm | 4.62.3 | traitlets=5.0.5 | 5.1.0 |
| Transformer | 4.17.0 | typer | 0.4.1 | typing-extensions | 3.10.0.2 |
| ujson | 4.0.2 | pembaruan tanpa pengawasan | 0.1 | urllib3 | 1.26.7 |
| virtualenv | 20.8.0 | visions | 0.7.4 | wasabi | 0.9.1 |
| wcwidth | 0.2.5 | webencodings=0.5.1 | 0.5.1 | websocket-client | 1.3.1 |
| Werkzeug | 2.0.2 | roda | 0.37.0 | widgetsnbextension | 3.6.0 |
| terbungkus | 1.12.1 | xgboost | 1.5.2 | zipp | 3.6.0 |
Pustaka Phyton di kluster GPU
| Perpustakaan | Versi | Perpustakaan | Versi | Perpustakaan | Versi |
|---|---|---|---|---|---|
| absl-py | 1.0.0 | Antergos Linux | 2015.10 (ISO-Rolling) | argon2-cffi | 20.1.0 |
| astor | 0.8.1 | astunparse | 1.6.3 | generator asinkron | 1.10 |
| atribut | 21.2.0 | panggilan balik | 0.2.0 | backports.entry-points-selectable | 1.1.1 |
| bcrypt | 3.2.2 | pemutih | 4.0.0 | kebahagiaan | 0.7.7 |
| boto3 | 1.21.18 | botocore | 1.24.18 | cachetools | 5.1.0 |
| katalog | 2.0.7 | sertifikat | 2021.10.8 | cffi | 1.14.6 |
| chardet | 4.0.0 | charset-normalizer | 2.0.4 | klik | 8.0.3 |
| cloudpickle | 2.0.0 | cmdstanpy | 0.9.68 | configparser | 5.2.0 |
| konversi_tanggal | 2.4.0 | kriptografi | 3.4.8 | cycler | 0.10.0 |
| cymem | 2.0.6 | Cython | 0.29.24 | databricks-automl-runtime | 0.2.8 |
| databricks-cli | 0.16.4 | dbl-tempo | 0.1.12 | dbus-python | 1.2.16 |
| debugpy | 1.4.1 | dekorator | 5.1.0 | defusedxml | 0.7.1 |
| dill | 0.3.4 | diskcache | 5.4.0 | distlib | 0.3.4 |
| distro-info | 0.23ubuntu1 | titik masuk | 0,3 | ephem | 4.1.3 |
| ikhtisar-fasilitas | 1.0.0 | fasttext | 0.9.2 | filelock | 3.3.1 |
| Flask | 1.1.2 | flatbuffers | 2.0 | fsspec | 1 Agustus 2021 |
| masa depan | 0.18.2 | gast | 0.5.3 | gitdb | 4.0.9 |
| GitPython | 3.1.27 | google-auth | 2.6.0 | google-auth-oauthlib | 0.4.6 |
| google-pasta | 0.2.0 | grpcio | 1.44.0 | gunicorn | 20.1.0 |
| gviz-api | 1.10.0 | h5py | 3.3.0 | hijri-converter | 2.2.3 |
| hari libur | 0,13 | Horovod | 0.24.3 | htmlmin | 0.1.12 |
| huggingface-hub | 0.6.0 | idna | 3.2 | Hash Gambar | 4.2.1 |
| imbalanced-learn | 0.8.1 | importlib-metadata | 4.8.1 | ipykernel | 6.12.1 |
| ipython | 7.32.0 | ipython-genutils | 0.2.0 | ipywidgets | 7.7.0 |
| isodate | 0.6.1 | itu berbahaya | 2.0.1 | jedi | 0.18.0 |
| Jinja2 | 2.11.3 | jmespath | 0.10.0 | joblib | 1.0.1 |
| joblibspark | 0.5.0 | jsonschema | 3.2.0 | jupyter-client | 6.1.12 |
| jupyter-core | 4.8.1 | jupyterlab-pygments | 0.1.2 | jupyterlab-widgets | 1.0.0 |
| keras | 2.8.0 | Keras-Prapemrosesan | 1.1.2 | kiwisolver | 1.3.1 |
| kalender lunar Korea | 0.2.1 | kode bahasa | 3.3.0 | libclang | 14.0.1 |
| lightgbm | 3.3.2 | llvmlite | 0.38.0 | Kalender Lunar | 0.0.9 |
| Mako | 1.2.0 | Markdown | 3.3.6 | MarkupSafe | 2.0.1 |
| matplotlib | 3.4.3 | matplotlib-inline | 0.1.2 | missingno | 0.5.1 |
| penyetelan yang salah | 0.8.4 | mleap | 0.20.0 | mlflow-skinny | 1,26,0 |
| metode ganda | 1.8 | murmurhash | 1.0.7 | nbclient | 0.5.3 |
| nbconvert | 6.1.0 | nbformat | 5.1.3 | nest-asyncio | 1.5.1 |
| networkx | 2.6.3 | nltk | 3.6.5 | buku catatan | 6.4.5 |
| numba | 0.55.1 | numpy | 1.20.3 | oauthlib | 3.2.0 |
| opt-einsum | 3.3.0 | kemasan | 21.0 | pandas | 1.3.4 |
| pandas-profiling | 3.1.0 | pandocfilters | 1.4.3 | paramiko | 2.9.2 |
| parso | 0.8.2 | pathy | 0.6.1 | korban mudah | 0.5.2 |
| petastorm | 0.11.4 | pexpect | 4.8.0 | phik | 0.12.2 |
| pickleshare | 0.7.5 | Bantal | 8.4.0 | pip (penginstal paket untuk Python) | 21.2.4 |
| platformdirs | 2.5.2 | plotly | 5.6.0 | pmdarima | 1.8.5 |
| preshed | 3.0.6 | prompt-toolkit | 3.0.20 | nabi | 1.0.1 |
| protobuf | 3.19.4 | psutil | 5.8.0 | psycopg2 | 2.9.3 |
| ptyprocess | 0.7.0 | pyarrow | 7.0.0 | pyasn1 | 0.4.8 |
| pyasn1-modul | 0.2.8 | pybind11 | 2.9.2 | pycparser | 2,20 |
| pydantic | 1.8.2 | Pygments | 2.10.0 | PyGObject | 3.36.0 |
| PyMeeus | 0.5.11 | PyNaCl | 1.5.0 | pyodbc | 4.0.31 |
| pyparsing | 3.0.4 | pyrsistent | 0.18.0 | pystan | 2.19.1.1 |
| python-apt | 2.0.0+ubuntu0.20.4.7 | python-dateutil | 2.8.2 | editor Python | 1.0.4 |
| pytz | 2021.3 | PyWavelets | 1.1.1 | PyYAML | 6.0 |
| pyzmq | 22.2.1 | regex | 3 Agustus 2021 | permintaan | 2.26.0 |
| requests-oauthlib | 1.3.1 | requests-unixsocket | 0.2.0 | rsa | 4.8 |
| s3transfer | 0.5.2 | sacremoses | 0.0.53 | scikit-learn | 0.24.2 |
| scipy | 1.7.1 | seaborn | 0.11.2 | Send2Trash | 1.8.0 |
| setuptools | 58.0.4 | setuptools-git | 1.2 | shap | 0.40.0 |
| simplejson | 3.17.6 | six | 1.16.0 | pengiris | 0.0.7 |
| smart-open | 5.2.1 | smmap | 5.0.0 | spacy | 3.2.3 |
| spacy-legacy | 3.0.9 | spacy-loggers | 1.0.2 | spark-tensorflow-distributor | 1.0.0 |
| sqlparse | 0.4.2 | srsly | 2.4.3 | ssh-import-id | 5.10 |
| statsmodels | 0.12.2 | membuat tabel | 0.8.9 | terjerat-dalam-unicode | 0.1.0 |
| kegigihan | 8.0.1 | Tensorboard (alat visualisasi untuk pembelajaran mesin) | 2.8.0 | tensorboard-data-server | 0.6.1 |
| tensorboard-plugin-profile | 2.5.0 | tensorboard-plugin-wit | 1.8.1 | tensorflow | 2.8.0 |
| tensorflow-estimator | 2.8.0 | tensorflow-io-gcs-filesystem | 0.25.0 | termcolor | 1.1.0 |
| selesai | 0.9.4 | jalur uji | 0.5.0 | tf-estimator-nightly | 2.8.0.dev2021122109 |
| thinc | 8.0.15 | threadpoolctl | 2.2.0 | pemotong teks | 0.12.1 |
| obor | 1.11.0+cu113 | torchvision | 0.12.0+cu113 | tornado | 6.1 |
| tqdm | 4.62.3 | traitlets | 5.1.0 | Transformer | 4.17.0 |
| typer | 0.4.1 | typing-extensions | 3.10.0.2 | ujson | 4.0.2 |
| pembaruan otomatis | 0.1 | urllib3 | 1.26.7 | virtualenv | 20.8.0 |
| visions | 0.7.4 | wasabi | 0.9.1 | wcwidth | 0.2.5 |
| webencodings | 0.5.1 | websocket-client | 1.3.1 | Werkzeug | 2.0.2 |
| roda | 0.37.0 | widgetsnbextension | 3.6.0 | terbungkus | 1.12.1 |
| xgboost | 1.5.2 | zipp | 3.6.0 |
Paket Spark yang berisi modul Python
| Paket Spark | Modul Python | Versi |
|---|---|---|
| graphframes | graphframes | 0.8.2-db1-spark3.2 |
Pustaka R
Pustaka R identik dengan Pustaka R di Databricks Runtime 11.0.
Pustaka Java dan Scala (Kluster Scala 2.12)
Selain pustaka Java dan Scala di Databricks Runtime 11.0, Databricks Runtime 11.0 ML berisi JAR berikut:
Kluster CPU
| ID Kelompok | ID Artefak | Versi |
|---|---|---|
| com.typesafe.akka | akka-actor_2.12 | 2.5.23 |
| ml.combust.mleap | mleap-databricks-runtime_2.12 | 0.18.1-23eb1ef |
| ml.dmlc | xgboost4j-spark_2.12 | 1.5.2 |
| ml.dmlc | xgboost4j_2.12 | 1.5.2 |
| org.graphframes | graphframes_2.12 | 0.8.2-db1-spark3.2 |
| org.mlflow | mlflow-client | 1,26,0 |
| org.mlflow | mlflow-spark | 1,26,0 |
| org.scala-lang.modules | scala-java8-compat_2.12 | 0.8.0 |
| org.tensorflow | spark-tensorflow-connector_2.12 | 1.15.0 |
Kluster GPU
| ID Grup | Artefak ID | Versi |
|---|---|---|
| com.typesafe.akka | akka-actor_2.12 | 2.5.23 |
| ml.combust.mleap | mleap-databricks-runtime_2.12 | 0.18.1-23eb1ef |
| ml.dmlc | xgboost4j-spark_2.12 | 1.5.2 |
| ml.dmlc | xgboost4j_2.12 | 1.5.2 |
| org.graphframes | graphframes_2.12 | 0.8.2-db1-spark3.2 |
| org.mlflow | mlflow-client | 1,26,0 |
| org.mlflow | mlflow-spark | 1,26,0 |
| org.scala-lang.modules | scala-java8-compat_2.12 | 0.8.0 |
| org.tensorflow | spark-tensorflow-connector_2.12 | 1.15.0 |