Databricks Runtime 8.3 untuk ML (EoS)

Databricks merilis versi ini pada Juni 2021.

Databricks Runtime 8.3 untuk Pembelajaran Mesin menyediakan lingkungan siap pakai untuk pembelajaran mesin dan ilmu data berdasarkan Databricks Runtime 8.3 (EoS). Runtime Bahasa Umum Databricks ML berisi banyak pustaka pembelajaran mesin populer, termasuk TensorFlow, PyTorch, dan XGBoost. Ini juga mendukung pelatihan pembelajaran mendalam terdistribusi menggunakan Horovod.

Untuk informasi selengkapnya, termasuk instruksi untuk membuat kluster ML Runtime Databricks, lihat AI dan pembelajaran mesin di Databricks.

Fitur dan peningkatan baru

ML Databricks Runtime 8.3 dibangun di atas Databricks Runtime 8.3. Untuk informasi tentang apa yang baru dalam Databricks Runtime 8.3, termasuk Apache Spark MLlib dan SparkR, lihat catatan rilis Databricks Runtime 8.3 (EoS).

ML Databricks Runtime 8.3 juga mencakup paket baru berikut:

• Rekayasa fitur dan penyajian
• AutoML

Perubahan besar pada lingkungan Phyton ML Databricks Runtime

Lihat Databricks Runtime 8.3 (EoS) untuk perubahan besar pada lingkungan Databricks Runtime Python. Untuk daftar lengkap paket Python yang diinstal dan versinya, lihat pustaka Python .

Paket Phyton ditingkatkan

• koalas 1.7.0 -> 1.8.0
• mlflow 1.15.0 -> 1.17.0
• pandas 1.1.3 -> 1.1.5
• petastorm 0.9.8 -> 0.10.0
• xgboost 1.3.3 -> 1.4.1

Paket Python yang ditambahkan

• holidays: 0.10.5.2

Gunakan Shiny di dalam buku catatan R

Anda sekarang dapat mengembangkan, menghosting, dan berbagi aplikasi Shiny langsung dari notebook R Azure Databricks, mirip dengan RStudio yang dihosting. Untuk detailnya, lihat Shiny di Azure Databricks.

Penghentian

Lingkungan Conda, bersama dengan perintah %conda, sekarang tidak digunakan lagi yang mendukung pip dan virtualenv, serta akan dihapus dalam rilis besar yang akan datang. Selain itu, gambar kustom yang menggunakan lingkungan berbasis Conda dengan Layanan Kontainer Databricks akan tetap didukung, tetapi tidak akan memiliki kemampuan pustaka yang dicakup notebook. Databricks merekomendasikan penggunaan lingkungan berbasis virtualenv dengan Layanan Kontainer Databricks dan %pip untuk semua pustaka yang dicakup notebook.

Lingkungan sistem

Lingkungan sistem di ML Databricks Runtime 8.3 berbeda dari Databricks Runtime 8.3 sebagai berikut:

• DBUtils: Databricks Runtime ML tidak menyertakan utilitas Pustaka (dbutils.library) (warisan). Gunakan perintah %pip dan %conda sebagai gantinya. Lihat Pustaka Python cakupan buku catatan.
• Untuk kluster GPU, Databricks Runtime ML menyertakan pustaka GPU NVIDIA berikut:
  • CUDA 11.0
  • cuDNN 8.0.4.30
  • NCCL 2.7.8
  • TensorRT 7.1.3

Pustaka

Bagian-bagian berikut mencantumkan pustaka yang termasuk dalam Databricks Runtime 8.3 ML dan berbeda dari pustaka yang ada dalam Databricks Runtime 8.3.

Di bagian ini:

• Pustaka tingkat atas
• Pustaka Python
• Pustaka R
• Pustaka Java dan Scala (Kluster Scala 2.12)

Pustaka tingkat atas

ML Databricks Runtime 8.3 mencakup pustaka tingkat atas berikut:

• GraphFrames
• Horovod dan HorovodRunner
• MLflow
• PyTorch
• spark-tensorflow-connector
• TensorFlow 2.5.0
• TensorBoard

Pustaka Python

ML Databricks Runtime 8.3 menggunakan Conda untuk manajemen paket Phyton dan mencakup banyak paket ML populer.

Selain paket yang ditentukan di lingkungan Conda di bagian berikut, ML Databricks Runtime 8.3 juga mencakup paket berikut:

• hyperopt 0.2.5.db1
• sparkdl 2.1.0.db4
• feature_store 0.3.1
• automl 1.0.0

Pustaka Phyton di kluster CPU

name: databricks-ml
channels:
  - pytorch
  - defaults
dependencies:
  - _libgcc_mutex=0.1=main
  - absl-py=0.11.0=pyhd3eb1b0_1
  - aiohttp=3.7.4=py38h27cfd23_1
  - asn1crypto=1.4.0=py_0
  - astor=0.8.1=py38h06a4308_0
  - async-timeout=3.0.1=py38h06a4308_0
  - attrs=20.3.0=pyhd3eb1b0_0
  - backcall=0.2.0=pyhd3eb1b0_0
  - bcrypt=3.2.0=py38h7b6447c_0
  - blas=1.0=mkl
  - blinker=1.4=py38h06a4308_0
  - boto3=1.16.7=pyhd3eb1b0_0
  - botocore=1.19.7=pyhd3eb1b0_0
  - brotlipy=0.7.0=py38h27cfd23_1003
  - bzip2=1.0.8=h7b6447c_0
  - c-ares=1.17.1=h27cfd23_0
  - ca-certificates=2021.4.13=h06a4308_1
  - cachetools=4.2.2=pyhd3eb1b0_0
  - certifi=2020.12.5=py38h06a4308_0
  - cffi=1.14.3=py38h261ae71_2
  - chardet=3.0.4=py38h06a4308_1003
  - click=7.1.2=pyhd3eb1b0_0
  - cloudpickle=1.6.0=py_0
  - configparser=5.0.1=py_0
  - cpuonly=1.0=0
  - cryptography=3.1.1=py38h1ba5d50_0
  - cycler=0.10.0=py38_0
  - cython=0.29.21=py38h2531618_0
  - decorator=4.4.2=pyhd3eb1b0_0
  - dill=0.3.2=py_0
  - docutils=0.15.2=py38h06a4308_1
  - entrypoints=0.3=py38_0
  - ffmpeg=4.2.2=h20bf706_0
  - flask=1.1.2=pyhd3eb1b0_0
  - freetype=2.10.4=h5ab3b9f_0
  - fsspec=0.8.3=py_0
  - future=0.18.2=py38_1
  - gitdb=4.0.7=pyhd3eb1b0_0
  - gitpython=3.1.12=pyhd3eb1b0_1
  - gmp=6.1.2=h6c8ec71_1
  - gnutls=3.6.15=he1e5248_0
  - google-auth=1.22.1=py_0
  - google-auth-oauthlib=0.4.2=pyhd3eb1b0_2
  - google-pasta=0.2.0=py_0
  - gunicorn=20.0.4=py38h06a4308_0
  - h5py=2.10.0=py38h7918eee_0
  - hdf5=1.10.4=hb1b8bf9_0
  - icu=58.2=he6710b0_3
  - idna=2.10=pyhd3eb1b0_0
  - importlib-metadata=2.0.0=py_1
  - intel-openmp=2019.4=243
  - ipykernel=5.3.4=py38h5ca1d4c_0
  - ipython=7.19.0=py38hb070fc8_1
  - ipython_genutils=0.2.0=pyhd3eb1b0_1
  - isodate=0.6.0=py_1
  - itsdangerous=1.1.0=pyhd3eb1b0_0
  - jedi=0.17.2=py38h06a4308_1
  - jinja2=2.11.2=pyhd3eb1b0_0
  - jmespath=0.10.0=py_0
  - joblib=0.17.0=py_0
  - jpeg=9b=h024ee3a_2
  - jupyter_client=6.1.7=py_0
  - jupyter_core=4.6.3=py38_0
  - kiwisolver=1.3.0=py38h2531618_0
  - krb5=1.17.1=h173b8e3_0
  - lame=3.100=h7b6447c_0
  - lcms2=2.11=h396b838_0
  - ld_impl_linux-64=2.33.1=h53a641e_7
  - libedit=3.1.20191231=h14c3975_1
  - libffi=3.3=he6710b0_2
  - libgcc-ng=9.1.0=hdf63c60_0
  - libgfortran-ng=7.3.0=hdf63c60_0
  - libidn2=2.3.0=h27cfd23_0
  - libopus=1.3.1=h7b6447c_0
  - libpng=1.6.37=hbc83047_0
  - libpq=12.2=h20c2e04_0
  - libprotobuf=3.13.0.1=hd408876_0
  - libsodium=1.0.18=h7b6447c_0
  - libstdcxx-ng=9.1.0=hdf63c60_0
  - libtasn1=4.16.0=h27cfd23_0
  - libtiff=4.1.0=h2733197_1
  - libunistring=0.9.10=h27cfd23_0
  - libuv=1.40.0=h7b6447c_0
  - libvpx=1.7.0=h439df22_0
  - lightgbm=3.1.1=py38h2531618_0
  - lz4-c=1.9.2=heb0550a_3
  - mako=1.1.3=py_0
  - markdown=3.3.3=py38h06a4308_0
  - markupsafe=1.1.1=py38h7b6447c_0
  - matplotlib-base=3.2.2=py38hef1b27d_0
  - mkl=2019.4=243
  - mkl-service=2.3.0=py38he904b0f_0
  - mkl_fft=1.2.0=py38h23d657b_0
  - mkl_random=1.1.0=py38h962f231_0
  - more-itertools=8.6.0=pyhd3eb1b0_0
  - multidict=5.1.0=py38h27cfd23_2
  - ncurses=6.2=he6710b0_1
  - nettle=3.7.2=hbbd107a_1
  - networkx=2.5.1=pyhd3eb1b0_0
  - ninja=1.10.2=hff7bd54_1
  - nltk=3.5=py_0
  - numpy=1.19.2=py38h54aff64_0
  - numpy-base=1.19.2=py38hfa32c7d_0
  - oauthlib=3.1.0=py_0
  - olefile=0.46=py_0
  - openh264=2.1.0=hd408876_0
  - openssl=1.1.1k=h27cfd23_0
  - packaging=20.4=py_0
  - pandas=1.1.5=py38ha9443f7_0
  - paramiko=2.7.2=py_0
  - parso=0.7.0=py_0
  - patsy=0.5.1=py38_0
  - pexpect=4.8.0=pyhd3eb1b0_3
  - pickleshare=0.7.5=pyhd3eb1b0_1003
  - pillow=8.0.1=py38he98fc37_0
  - pip=20.2.4=py38h06a4308_0
  - plotly=4.14.3=pyhd3eb1b0_0
  - prompt-toolkit=3.0.8=py_0
  - prompt_toolkit=3.0.8=0
  - protobuf=3.13.0.1=py38he6710b0_1
  - psutil=5.7.2=py38h7b6447c_0
  - psycopg2=2.8.5=py38h3c74f83_1
  - ptyprocess=0.6.0=pyhd3eb1b0_2
  - pyasn1=0.4.8=py_0
  - pyasn1-modules=0.2.8=py_0
  - pycparser=2.20=py_2
  - pygments=2.7.2=pyhd3eb1b0_0
  - pyjwt=1.7.1=py38_0
  - pynacl=1.4.0=py38h7b6447c_1
  - pyodbc=4.0.30=py38he6710b0_0
  - pyopenssl=19.1.0=pyhd3eb1b0_1
  - pyparsing=2.4.7=pyhd3eb1b0_0
  - pysocks=1.7.1=py38h06a4308_0
  - python=3.8.8=hdb3f193_4
  - python-dateutil=2.8.1=pyhd3eb1b0_0
  - python-editor=1.0.4=py_0
  - pytorch=1.8.1=py3.8_cpu_0
  - pytz=2020.5=pyhd3eb1b0_0
  - pyzmq=19.0.2=py38he6710b0_1
  - readline=8.0=h7b6447c_0
  - regex=2020.10.15=py38h7b6447c_0
  - requests=2.24.0=py_0
  - requests-oauthlib=1.3.0=py_0
  - retrying=1.3.3=py_2
  - rsa=4.7.2=pyhd3eb1b0_1
  - s3transfer=0.3.6=pyhd3eb1b0_0
  - scikit-learn=0.23.2=py38h0573a6f_0
  - scipy=1.5.2=py38h0b6359f_0
  - setuptools=50.3.1=py38h06a4308_1
  - simplejson=3.17.2=py38h27cfd23_2
  - six=1.15.0=py38h06a4308_0
  - smmap=3.0.5=pyhd3eb1b0_0
  - sqlite=3.33.0=h62c20be_0
  - sqlparse=0.4.1=py_0
  - statsmodels=0.12.0=py38h7b6447c_0
  - tabulate=0.8.7=py38h06a4308_0
  - threadpoolctl=2.1.0=pyh5ca1d4c_0
  - tk=8.6.10=hbc83047_0
  - torchvision=0.9.1=py38_cpu
  - tornado=6.0.4=py38h7b6447c_1
  - tqdm=4.50.2=py_0
  - traitlets=5.0.5=pyhd3eb1b0_0
  - typing-extensions=3.7.4.3=hd3eb1b0_0
  - typing_extensions=3.7.4.3=pyh06a4308_0
  - unixodbc=2.3.9=h7b6447c_0
  - urllib3=1.25.11=py_0
  - wcwidth=0.2.5=py_0
  - websocket-client=0.57.0=py38_2
  - werkzeug=1.0.1=pyhd3eb1b0_0
  - wheel=0.35.1=pyhd3eb1b0_0
  - wrapt=1.12.1=py38h7b6447c_1
  - x264=1!157.20191217=h7b6447c_0
  - xz=5.2.5=h7b6447c_0
  - yarl=1.6.3=py38h27cfd23_0
  - zeromq=4.3.3=he6710b0_3
  - zipp=3.4.0=pyhd3eb1b0_0
  - zlib=1.2.11=h7b6447c_3
  - zstd=1.4.5=h9ceee32_0
  - pip:
      - argon2-cffi==20.1.0
      - astunparse==1.6.3
      - async-generator==1.10
      - azure-core==1.11.0
      - azure-storage-blob==12.7.1
      - bleach==3.3.0
      - confuse==1.4.0
      - convertdate==2.3.2
      - databricks-cli==0.14.3
      - defusedxml==0.7.1
      - diskcache==5.2.1
      - docker==4.4.4
      - facets-overview==1.0.0
      - flatbuffers==1.12
      - gast==0.3.3
      - grpcio==1.32.0
      - hijri-converter==2.1.1
      - holidays==0.10.5.2
      - horovod==0.21.3
      - htmlmin==0.1.12
      - imagehash==4.2.0
      - ipywidgets==7.6.3
      - joblibspark==0.3.0
      - jsonschema==3.2.0
      - jupyterlab-pygments==0.1.2
      - jupyterlab-widgets==1.0.0
      - keras-preprocessing==1.1.2
      - koalas==1.8.0
      - korean-lunar-calendar==0.2.1
      - llvmlite==0.36.0
      - missingno==0.4.2
      - mistune==0.8.4
      - mleap==0.16.1
      - mlflow-skinny==1.17.0
      - msrest==0.6.21
      - nbclient==0.5.3
      - nbconvert==6.0.7
      - nbformat==5.1.3
      - nest-asyncio==1.5.1
      - notebook==6.4.0
      - numba==0.53.1
      - opt-einsum==3.3.0
      - pandas-profiling==2.11.0
      - pandocfilters==1.4.3
      - petastorm==0.10.0
      - phik==0.11.2
      - prometheus-client==0.10.1
      - pyarrow==1.0.1
      - pymeeus==0.5.11
      - pyrsistent==0.17.3
      - pywavelets==1.1.1
      - pyyaml==5.4.1
      - querystring-parser==1.2.4
      - seaborn==0.10.0
      - send2trash==1.5.0
      - shap==0.39.0
      - slicer==0.0.7
      - spark-tensorflow-distributor==0.1.0
      - tangled-up-in-unicode==0.1.0
      - tensorboard==2.4.1
      - tensorboard-plugin-wit==1.8.0
      - tensorflow-cpu==2.4.1
      - tensorflow-estimator==2.4.0
      - termcolor==1.1.0
      - terminado==0.9.5
      - testpath==0.5.0
      - visions==0.6.0
      - webencodings==0.5.1
      - widgetsnbextension==3.5.1
      - xgboost==1.4.1
prefix: /databricks/conda/envs/databricks-ml

Pustaka Phyton di kluster GPU

name: databricks-ml-gpu
channels:
  - defaults
dependencies:
  - _libgcc_mutex=0.1=main
  - absl-py=0.11.0=pyhd3eb1b0_1
  - aiohttp=3.7.4=py38h27cfd23_1
  - asn1crypto=1.4.0=py_0
  - astor=0.8.1=py38h06a4308_0
  - async-timeout=3.0.1=py38h06a4308_0
  - attrs=20.3.0=pyhd3eb1b0_0
  - backcall=0.2.0=pyhd3eb1b0_0
  - bcrypt=3.2.0=py38h7b6447c_0
  - blas=1.0=mkl
  - blinker=1.4=py38h06a4308_0
  - boto3=1.16.7=pyhd3eb1b0_0
  - botocore=1.19.7=pyhd3eb1b0_0
  - brotlipy=0.7.0=py38h27cfd23_1003
  - c-ares=1.17.1=h27cfd23_0
  - ca-certificates=2021.4.13=h06a4308_1
  - cachetools=4.2.2=pyhd3eb1b0_0
  - certifi=2020.12.5=py38h06a4308_0
  - cffi=1.14.3=py38h261ae71_2
  - chardet=3.0.4=py38h06a4308_1003
  - click=7.1.2=pyhd3eb1b0_0
  - cloudpickle=1.6.0=py_0
  - configparser=5.0.1=py_0
  - cryptography=3.1.1=py38h1ba5d50_0
  - cycler=0.10.0=py38_0
  - cython=0.29.21=py38h2531618_0
  - decorator=4.4.2=pyhd3eb1b0_0
  - dill=0.3.2=py_0
  - docutils=0.15.2=py38h06a4308_1
  - entrypoints=0.3=py38_0
  - flask=1.1.2=pyhd3eb1b0_0
  - freetype=2.10.4=h5ab3b9f_0
  - fsspec=0.8.3=py_0
  - future=0.18.2=py38_1
  - gitdb=4.0.7=pyhd3eb1b0_0
  - gitpython=3.1.12=pyhd3eb1b0_1
  - google-auth=1.22.1=py_0
  - google-auth-oauthlib=0.4.2=pyhd3eb1b0_2
  - google-pasta=0.2.0=py_0
  - grpcio=1.31.0=py38hf8bcb03_0
  - gunicorn=20.0.4=py38h06a4308_0
  - h5py=2.10.0=py38h7918eee_0
  - hdf5=1.10.4=hb1b8bf9_0
  - icu=58.2=he6710b0_3
  - idna=2.10=pyhd3eb1b0_0
  - importlib-metadata=2.0.0=py_1
  - intel-openmp=2019.4=243
  - ipykernel=5.3.4=py38h5ca1d4c_0
  - ipython=7.19.0=py38hb070fc8_1
  - ipython_genutils=0.2.0=pyhd3eb1b0_1
  - isodate=0.6.0=py_1
  - itsdangerous=1.1.0=pyhd3eb1b0_0
  - jedi=0.17.2=py38h06a4308_1
  - jinja2=2.11.2=pyhd3eb1b0_0
  - jmespath=0.10.0=py_0
  - joblib=0.17.0=py_0
  - jpeg=9b=h024ee3a_2
  - jupyter_client=6.1.7=py_0
  - jupyter_core=4.6.3=py38_0
  - kiwisolver=1.3.0=py38h2531618_0
  - krb5=1.17.1=h173b8e3_0
  - lcms2=2.11=h396b838_0
  - ld_impl_linux-64=2.33.1=h53a641e_7
  - libedit=3.1.20191231=h14c3975_1
  - libffi=3.3=he6710b0_2
  - libgcc-ng=9.1.0=hdf63c60_0
  - libgfortran-ng=7.3.0=hdf63c60_0
  - libpng=1.6.37=hbc83047_0
  - libpq=12.2=h20c2e04_0
  - libprotobuf=3.13.0.1=hd408876_0
  - libsodium=1.0.18=h7b6447c_0
  - libstdcxx-ng=9.1.0=hdf63c60_0
  - libtiff=4.1.0=h2733197_1
  - lightgbm=3.1.1=py38h2531618_0
  - lz4-c=1.9.2=heb0550a_3
  - mako=1.1.3=py_0
  - markdown=3.3.3=py38h06a4308_0
  - markupsafe=1.1.1=py38h7b6447c_0
  - matplotlib-base=3.2.2=py38hef1b27d_0
  - mkl=2019.4=243
  - mkl-service=2.3.0=py38he904b0f_0
  - mkl_fft=1.2.0=py38h23d657b_0
  - mkl_random=1.1.0=py38h962f231_0
  - more-itertools=8.6.0=pyhd3eb1b0_0
  - multidict=5.1.0=py38h27cfd23_2
  - ncurses=6.2=he6710b0_1
  - networkx=2.5.1=pyhd3eb1b0_0
  - nltk=3.5=py_0
  - numpy=1.19.2=py38h54aff64_0
  - numpy-base=1.19.2=py38hfa32c7d_0
  - oauthlib=3.1.0=py_0
  - olefile=0.46=py_0
  - openssl=1.1.1k=h27cfd23_0
  - packaging=20.4=py_0
  - pandas=1.1.5=py38ha9443f7_0
  - paramiko=2.7.2=py_0
  - parso=0.7.0=py_0
  - patsy=0.5.1=py38_0
  - pexpect=4.8.0=pyhd3eb1b0_3
  - pickleshare=0.7.5=pyhd3eb1b0_1003
  - pillow=8.0.1=py38he98fc37_0
  - pip=20.2.4=py38h06a4308_0
  - plotly=4.14.3=pyhd3eb1b0_0
  - prompt-toolkit=3.0.8=py_0
  - prompt_toolkit=3.0.8=0
  - protobuf=3.13.0.1=py38he6710b0_1
  - psutil=5.7.2=py38h7b6447c_0
  - psycopg2=2.8.5=py38h3c74f83_1
  - ptyprocess=0.6.0=pyhd3eb1b0_2
  - pyasn1=0.4.8=py_0
  - pyasn1-modules=0.2.8=py_0
  - pycparser=2.20=py_2
  - pygments=2.7.2=pyhd3eb1b0_0
  - pyjwt=1.7.1=py38_0
  - pynacl=1.4.0=py38h7b6447c_1
  - pyodbc=4.0.30=py38he6710b0_0
  - pyopenssl=19.1.0=pyhd3eb1b0_1
  - pyparsing=2.4.7=pyhd3eb1b0_0
  - pysocks=1.7.1=py38h06a4308_0
  - python=3.8.8=hdb3f193_4
  - python-dateutil=2.8.1=pyhd3eb1b0_0
  - python-editor=1.0.4=py_0
  - pytz=2020.5=pyhd3eb1b0_0
  - pyzmq=19.0.2=py38he6710b0_1
  - readline=8.0=h7b6447c_0
  - regex=2020.10.15=py38h7b6447c_0
  - requests=2.24.0=py_0
  - requests-oauthlib=1.3.0=py_0
  - retrying=1.3.3=py_2
  - rsa=4.7.2=pyhd3eb1b0_1
  - s3transfer=0.3.6=pyhd3eb1b0_0
  - scikit-learn=0.23.2=py38h0573a6f_0
  - scipy=1.5.2=py38h0b6359f_0
  - setuptools=50.3.1=py38h06a4308_1
  - simplejson=3.17.2=py38h27cfd23_2
  - six=1.15.0=py38h06a4308_0
  - smmap=3.0.5=pyhd3eb1b0_0
  - sqlite=3.33.0=h62c20be_0
  - sqlparse=0.4.1=py_0
  - statsmodels=0.12.0=py38h7b6447c_0
  - tabulate=0.8.7=py38h06a4308_0
  - threadpoolctl=2.1.0=pyh5ca1d4c_0
  - tk=8.6.10=hbc83047_0
  - tornado=6.0.4=py38h7b6447c_1
  - tqdm=4.50.2=py_0
  - traitlets=5.0.5=pyhd3eb1b0_0
  - typing-extensions=3.7.4.3=hd3eb1b0_0
  - typing_extensions=3.7.4.3=pyh06a4308_0
  - unixodbc=2.3.9=h7b6447c_0
  - urllib3=1.25.11=py_0
  - wcwidth=0.2.5=py_0
  - websocket-client=0.57.0=py38_2
  - werkzeug=1.0.1=pyhd3eb1b0_0
  - wheel=0.35.1=pyhd3eb1b0_0
  - wrapt=1.12.1=py38h7b6447c_1
  - xz=5.2.5=h7b6447c_0
  - yarl=1.6.3=py38h27cfd23_0
  - zeromq=4.3.3=he6710b0_3
  - zipp=3.4.0=pyhd3eb1b0_0
  - zlib=1.2.11=h7b6447c_3
  - zstd=1.4.5=h9ceee32_0
  - pip:
      - argon2-cffi==20.1.0
      - astunparse==1.6.3
      - async-generator==1.10
      - azure-core==1.11.0
      - azure-storage-blob==12.7.1
      - bleach==3.3.0
      - confuse==1.4.0
      - convertdate==2.3.2
      - databricks-cli==0.14.3
      - defusedxml==0.7.1
      - diskcache==5.2.1
      - docker==4.4.4
      - facets-overview==1.0.0
      - flatbuffers==1.12
      - gast==0.3.3
      - hijri-converter==2.1.1
      - holidays==0.10.5.2
      - horovod==0.21.3
      - htmlmin==0.1.12
      - imagehash==4.2.0
      - ipywidgets==7.6.3
      - joblibspark==0.3.0
      - jsonschema==3.2.0
      - jupyterlab-pygments==0.1.2
      - jupyterlab-widgets==1.0.0
      - keras-preprocessing==1.1.2
      - koalas==1.8.0
      - korean-lunar-calendar==0.2.1
      - llvmlite==0.36.0
      - missingno==0.4.2
      - mistune==0.8.4
      - mleap==0.16.1
      - mlflow-skinny==1.17.0
      - msrest==0.6.21
      - nbclient==0.5.3
      - nbconvert==6.0.7
      - nbformat==5.1.3
      - nest-asyncio==1.5.1
      - notebook==6.4.0
      - numba==0.53.1
      - opt-einsum==3.3.0
      - pandas-profiling==2.11.0
      - pandocfilters==1.4.3
      - petastorm==0.10.0
      - phik==0.11.2
      - pyarrow==1.0.1
      - pymeeus==0.5.11
      - pyrsistent==0.17.3
      - pywavelets==1.1.1
      - pyyaml==5.4.1
      - querystring-parser==1.2.4
      - seaborn==0.10.0
      - send2trash==1.5.0
      - shap==0.39.0
      - slicer==0.0.7
      - spark-tensorflow-distributor==0.1.0
      - tangled-up-in-unicode==0.1.0
      - tensorboard==2.4.1
      - tensorboard-plugin-wit==1.8.0
      - tensorflow==2.4.1
      - tensorflow-estimator==2.4.0
      - termcolor==1.1.0
      - terminado==0.9.5
      - testpath==0.5.0
      - torch==1.8.1
      - torchvision==0.9.1
      - visions==0.6.0
      - webencodings==0.5.1
      - widgetsnbextension==3.5.1
      - xgboost==1.4.1
prefix: /databricks/conda/envs/databricks-ml-gpu

Paket Spark yang berisi modul Python

Pustaka R

Pustaka R identik dengan Pustaka R di Databricks Runtime 8.3.

Pustaka Java dan Scala (Kluster Scala 2.12)

Selain pustaka Java dan Scala di Databricks Runtime 8.3, ML Databricks Runtime 8.3 berisi JAR berikut:

Kluster CPU

Kluster GPU