Bagikan melalui

Databricks Runtime 7.6 untuk Pembelajaran Mesin (EoS)

  • Artikel

Catatan

Dukungan untuk versi Databricks Runtime ini telah berakhir. Untuk tanggal akhir dukungan, lihat Riwayat akhir dukungan. Untuk semua versi Runtime Databricks yang didukung, lihat Versi dan kompatibilitas catatan rilis Databricks Runtime.

Databricks merilis versi ini pada Februari 2021.

Databricks Runtime 7.6 untuk Pembelajaran Mesin menyediakan lingkungan siap pakai untuk pembelajaran mesin dan ilmu data berdasarkan Databricks Runtime 7.6 (EoS). Runtime Bahasa Umum Databricks ML berisi banyak pustaka pembelajaran mesin populer, termasuk TensorFlow, PyTorch, dan XGBoost. Ini juga mendukung pelatihan pembelajaran mendalam terdistribusi menggunakan Horovod.

Untuk informasi selengkapnya, termasuk instruksi untuk membuat kluster ML Runtime Databricks, lihat AI dan pembelajaran mesin di Databricks.

Untuk bantuan tentang migrasi dari Databricks Runtime 6.x, lihat Panduan migrasi Databricks Runtime 7.x (EoS).

Fitur baru dan perubahan besar

Runtime Bahasa Umum Databricks 7.6 ML dibangun di atas Runtime Bahasa Umum Databricks 7.6. Untuk informasi tentang apa yang baru di Databricks Runtime 7.6, termasuk Apache Spark MLlib dan SparkR, lihat catatan rilis Databricks Runtime 7.6 (EoS).

Penghentian

  • Tensoflow 1.x tidak akan didukung dalam rilis utama Runtime Bahasa Umum Databricks yang akan datang
  • Paket CUDA berikut tidak digunakan lagi dan akan dihapus dalam rilis utama Databricks Runtime yang akan datang:
    • Alat-baris-perintah-cuda
    • cuda-compiler
    • cuda-cudart-dev
    • cuda-cufft
    • cuda-cufft-dev
    • cuda-cuobjdump
    • cuda-cupti
    • cuda-curand
    • cuda-curand-dev
    • cuda-cusolver
    • cuda-cusolver-dev
    • cuda-cusparse
    • cuda-cusparse-dev
    • cuda-documentation
    • cuda-driver-dev
    • cuda-gdb
    • cuda-gpu-library-advisor
    • cuda-libraries-dev
    • cuda-license
    • cuda-memcheck
    • cuda-minimal-build
    • cuda-misc-headers
    • cuda-npp
    • cuda-npp-dev
    • cuda-nsight
    • cuda-nvcc
    • cuda-nvdisasm
    • cuda-nvgraph
    • cuda-nvgraph-dev
    • cuda-nvjpeg
    • cuda-nvjpeg-dev
    • cuda-nvml-dev
    • cuda-nvprune
    • cuda-nvrtc-dev
    • cuda-nvvp
    • cuda-samples
    • cuda-sanitizer-api
    • cuda-toolkit
    • cuda-tools
    • cuda-visual-tools
    • freeglut3
    • libcublas-dev
    • libcudnn7-dev
    • libdrm-dev
    • libegl1
    • libegm-mesa0
    • libgbl1-mesa-dev
    • libgbm1
    • libgles1
    • libgles2
    • libglu1-mesa
    • libglu1-mesa-dev
    • libnccl-dev
    • libnvinfer-dev
    • libnvinfer-plugin-dev
    • libopengl0
    • libwayland-server0
    • libx11-xcb-dev
    • libxcb-dri2-0-dev
    • libxcb-dri3-dev
    • libxcb-glx0-dev
    • libxcb-present-dev
    • libxcb-randr0
    • libxcb-randr0-dev
    • libxcb-render0-dev
    • libxcb-shape0-dev
    • libxcb-sync-dev
    • libxcb-xfixes0
    • libxcb-xfixes0-dev
    • libxdamage-dev
    • libxext-dev
    • libxfixes-dev
    • libxi-dev
    • libxmu-dev
    • libxmu-headers
    • libxshmfence-dev
    • libxxf86vm-dev
    • mesa-common-dev
    • nsight-compute
    • nsight-systems
    • x11proto-damage-dev
    • x11proto-fixes-dev
    • x11proto-input-dev
    • x11proto-xext-dev
    • x11proto-xf86vidmode-dev

Perubahan besar pada lingkungan Phyton ML Databricks Runtime

Lihat Databricks Runtime 7.6 (EoS) untuk perubahan besar pada lingkungan Databricks Runtime Python. Untuk daftar lengkap paket Python yang diinstal dan versinya, lihat Pustaka Python.

Paket Phyton ditingkatkan

  • databricks-cli 0.14.0 -> 0.14.1
  • koalas 1.4.0 -> 1.5.0
  • lightgbm 2.3.0 -> 3.1.1
  • mlflow 1.12.1 -> 1.13.1
  • plotly 4.12.0 -> 4.14.1
  • pytorch 1.7.0 -> 1.7.1
  • torchvision 0.8.1 -> 0.8.2
  • xgboost 1.2.1 -> 1.3.1

Penyempurnaan

Integrasi PySpark dari XGBoost (Pratinjau Umum)

Integrasi XGBoost dengan PySpark telah ditingkatkan. Paket sparkdl 2.1.0-db5 ini mencakup dua estimator pyspark ML baru, XgboostRegressor dan XgboostClassifier, yang memungkinkan pengguna untuk melatih model XGBoost di Alur PySpark ML.

Sebelum versi ini, XGBoost tidak terintegrasi dengan PySpark. Pengguna harus menggunakan xgboost4j-spark di Scala atau memecahkan Alur PySpark ML, mengumpulkan Spark DataFrame pada driver sebagai DataFrame panda, dan menggunakan paket Python xgboost. Lihat dokumentasi sparkdl API dan Menggunakan XGBoost di Azure Databricks untuk detail lebih lanjut.

Lingkungan sistem

Lingkungan sistem di Runtime Bahasa Umum Databricks 7.6 ML berbeda dari Runtime Bahasa Umum Databricks 7.6 sebagai berikut:

Pustaka

Bagian berikut mencantumkan pustaka yang disertakan dalam Runtime Bahasa Umum Databricks 7.6 ML yang berbeda dari yang termasuk dalam Runtime Bahasa Umum Databricks 7.6.

Di bagian ini:

Pustaka tingkat atas

Runtime Bahasa Umum Databricks 7.6 mencakup pustaka tingkat atas berikut:

Pustaka Python

Runtime Bahasa Umum Databricks 7.6 ML menggunakan Conda untuk manajemen paket Python dan mencakup banyak paket ML populer.

Selain paket yang ditentukan di lingkungan Conda di bagian berikut, Runtime Bahasa Umum Databricks 7.6 ML juga menginstal paket berikut:

  • hyperopt 0.2.5.db1
  • sparkdl 2.1.0-db5

Pustaka Phyton di kluster CPU

name: databricks-ml
channels:
  - pytorch
  - defaults
dependencies:
  - _libgcc_mutex=0.1=main
  - absl-py=0.9.0=py37_0
  - asn1crypto=1.3.0=py37_1
  - astor=0.8.0=py37_0
  - backcall=0.1.0=py37_0
  - backports=1.0=pyhd3eb1b0_2
  - bcrypt=3.2.0=py37h7b6447c_0
  - blas=1.0=mkl
  - blinker=1.4=py37_0
  - boto3=1.12.0=py_0
  - botocore=1.15.0=py_0
  - c-ares=1.17.1=h27cfd23_0
  - ca-certificates=2021.1.19=h06a4308_1 # (updated from h06a4308_0 in May 26, 2021 maintenance update)
  - cachetools=4.2.0=pyhd3eb1b0_0
  - certifi=2020.12.5=py37h06a4308_0
  - cffi=1.14.0=py37he30daa8_1 # (updated from py37h2e261b9_0 in May 26, 2021 maintenance update)
  - chardet=3.0.4=py37h06a4308_1003
  - click=7.0=py37_0
  - cloudpickle=1.4.1=py_0
  - configparser=3.7.4=py37_0
  - cpuonly=1.0=0
  - cryptography=2.8=py37h1ba5d50_0
  - cycler=0.10.0=py37_0
  - cython=0.29.15=py37he6710b0_0
  - decorator=4.4.1=py_0
  - dill=0.3.1.1=py37_1
  - docutils=0.15.2=py37_0
  - entrypoints=0.3=py37_0
  - flask=1.1.1=py_1
  - freetype=2.9.1=h8a8886c_1
  - future=0.18.2=py37_1
  - gast=0.3.3=py_0
  - gitdb=4.0.5=py_0
  - gitpython=3.1.0=py_0
  - google-auth=1.11.2=py_0
  - google-auth-oauthlib=0.4.1=py_2
  - google-pasta=0.2.0=py_0
  - grpcio=1.27.2=py37hf8bcb03_0
  - gunicorn=20.0.4=py37_0
  - h5py=2.10.0=py37h7918eee_0
  - hdf5=1.10.4=hb1b8bf9_0
  - icu=58.2=he6710b0_3
  - idna=2.8=py37_0
  - intel-openmp=2020.0=166
  - ipykernel=5.1.4=py37h39e3cac_0
  - ipython=7.12.0=py37h5ca1d4c_0
  - ipython_genutils=0.2.0=pyhd3eb1b0_1
  - isodate=0.6.0=py_1
  - itsdangerous=1.1.0=py37_0
  - jedi=0.17.2=py37h06a4308_1
  - jinja2=2.11.1=py_0
  - jmespath=0.10.0=py_0
  - joblib=0.14.1=py_0
  - jpeg=9b=h024ee3a_2
  - jupyter_client=5.3.4=py37_0
  - jupyter_core=4.6.1=py37_0
  - kiwisolver=1.1.0=py37he6710b0_0
  - krb5=1.17.1=h173b8e3_0 # (updated from 1.16.4 in May 26, 2021 maintenance update)
  - ld_impl_linux-64=2.33.1=h53a641e_7
  - libedit=3.1.20181209=hc058e9b_0
  - libffi=3.3=he6710b0_2 # (updated from 3.2.1 in May 26, 2021 maintenance update)
  - libgcc-ng=9.1.0=hdf63c60_0
  - libgfortran-ng=7.3.0=hdf63c60_0
  - libpng=1.6.37=hbc83047_0
  - libpq=12.2=h20c2e04_0 # (updated from 11.2 in May 26, 2021 maintenance update)
  - libprotobuf=3.11.4=hd408876_0
  - libsodium=1.0.16=h1bed415_0
  - libstdcxx-ng=9.1.0=hdf63c60_0
  - libtiff=4.1.0=h2733197_0
  - libuv=1.40.0=h7b6447c_0
  - lightgbm=3.1.1=py37h2531618_0
  - lz4-c=1.8.1.2=h14c3975_0
  - mako=1.1.2=py_0
  - markdown=3.1.1=py37_0
  - markupsafe=1.1.1=py37h14c3975_1
  - matplotlib-base=3.1.3=py37hef1b27d_0
  - mkl=2020.0=166
  - mkl-service=2.3.0=py37he8ac12f_0
  - mkl_fft=1.0.15=py37ha843d7b_0
  - mkl_random=1.1.0=py37hd6b4f25_0
  - ncurses=6.2=he6710b0_1
  - networkx=2.4=py_1
  - ninja=1.10.2=py37hff7bd54_0
  - nltk=3.4.5=py37_0
  - numpy=1.18.1=py37h4f9e942_0
  - numpy-base=1.18.1=py37hde5b4d6_1
  - oauthlib=3.1.0=py_0
  - olefile=0.46=py37_0
  - openssl=1.1.1k=h27cfd23_0 # (updated from 1.1.1i in May 26, 2021 maintenance update)
  - packaging=20.1=py_0
  - pandas=1.0.1=py37h0573a6f_0
  - paramiko=2.7.1=py_0
  - parso=0.7.0=py_0
  - patsy=0.5.1=py37_0
  - pexpect=4.8.0=pyhd3eb1b0_3
  - pickleshare=0.7.5=pyhd3eb1b0_1003
  - pillow=7.0.0=py37hb39fc2d_0
  - pip=20.0.2=py37_3
  - plotly=4.14.1=pyhd3eb1b0_0
  - prompt_toolkit=3.0.3=py_0
  - protobuf=3.11.4=py37he6710b0_0
  - psutil=5.6.7=py37h7b6447c_0
  - psycopg2=2.8.6=py37h3c74f83_1 # (updated from 2.8.4 in May 26, 2021 maintenance update)
  - ptyprocess=0.6.0=pyhd3eb1b0_2
  - pyasn1=0.4.8=py_0
  - pyasn1-modules=0.2.8=py_0
  - pycparser=2.19=py37_0
  - pygments=2.5.2=py_0
  - pyjwt=2.0.1=py37h06a4308_0
  - pynacl=1.3.0=py37h7b6447c_0
  - pyodbc=4.0.30=py37he6710b0_0
  - pyopenssl=19.1.0=pyhd3eb1b0_1
  - pyparsing=2.4.6=py_0
  - pysocks=1.7.1=py37_1
  - python=3.7.10=hdb3f193_0 # (updated from 3.7.6 in May 26, 2021 maintenance update)
  - python-dateutil=2.8.1=py_0
  - python-editor=1.0.4=py_0
  - pytorch=1.7.1=py3.7_cpu_0
  - pytz=2019.3=py_0
  - pyzmq=18.1.1=py37he6710b0_0
  - readline=8.1=h27cfd23_0 # (updated from 7.0 in May 26, 2021 maintenance update)
  - requests=2.22.0=py37_1
  - requests-oauthlib=1.3.0=py_0
  - retrying=1.3.3=py37_2
  - rsa=4.0=py_0
  - s3transfer=0.3.4=pyhd3eb1b0_0
  - scikit-learn=0.22.1=py37hd81dba3_0
  - scipy=1.4.1=py37h0b6359f_0
  - setuptools=45.2.0=py37_0
  - simplejson=3.17.0=py37h7b6447c_0
  - six=1.14.0=py37h06a4308_0
  - smmap=3.0.4=py_0
  - sqlite=3.35.4=hdfb4753_0 # (updated from 3.31.1 in May 26, 2021 maintenance update)
  - sqlparse=0.4.1=py_0
  - statsmodels=0.11.0=py37h7b6447c_0
  - tabulate=0.8.3=py37_0
  - tk=8.6.10=hbc83047_0 # (updated from 8.6.8 in May 26, 2021 maintenance update)
  - torchvision=0.8.2=py37_cpu
  - tornado=6.0.3=py37h7b6447c_3
  - tqdm=4.42.1=py_0
  - traitlets=4.3.3=py37_0
  - typing_extensions=3.7.4.3=py_0
  - unixodbc=2.3.7=h14c3975_0
  - urllib3=1.25.8=py37_0
  - wcwidth=0.1.8=py_0
  - websocket-client=0.56.0=py37_0
  - werkzeug=1.0.0=py_0
  - wheel=0.34.2=py37_0
  - wrapt=1.11.2=py37h7b6447c_0
  - xz=5.2.5=h7b6447c_0 # (updated from 5.2.4 in May 26, 2021 maintenance update)
  - zeromq=4.3.1=he6710b0_3
  - zlib=1.2.11=h7b6447c_3
  - zstd=1.3.7=h0b5b093_0
  - pip:
      - astunparse==1.6.3
      - azure-core==1.10.0
      - azure-storage-blob==12.7.0
      - databricks-cli==0.14.1
      - diskcache==5.1.0
      - docker==4.4.1
      - gorilla==0.3.0
      - horovod==0.20.3
      - joblibspark==0.3.0
      - keras-preprocessing==1.1.2
      - koalas==1.5.0
      - mleap==0.16.1
      - mlflow==1.13.1
      - msrest==0.6.19
      - opt-einsum==3.3.0
      - petastorm==0.9.7
      - pyarrow==1.0.1
      - pyyaml==5.4
      - querystring-parser==1.2.4
      - seaborn==0.10.0
      - spark-tensorflow-distributor==0.1.0
      - tensorboard==2.3.0
      - tensorboard-plugin-wit==1.8.0
      - tensorflow-cpu==2.3.1
      - tensorflow-estimator==2.3.0
      - termcolor==1.1.0
      - xgboost==1.3.1
prefix: /databricks/conda/envs/databricks-ml

Pustaka Phyton di kluster GPU

name: databricks-ml-gpu
channels:
  - pytorch
  - defaults
dependencies:
  - _libgcc_mutex=0.1=main
  - absl-py=0.9.0=py37_0
  - asn1crypto=1.3.0=py37_1
  - astor=0.8.0=py37_0
  - backcall=0.1.0=py37_0
  - backports=1.0=pyhd3eb1b0_2
  - bcrypt=3.2.0=py37h7b6447c_0
  - blas=1.0=mkl
  - blinker=1.4=py37_0
  - boto3=1.12.0=py_0
  - botocore=1.15.0=py_0
  - c-ares=1.17.1=h27cfd23_0
  - ca-certificates=2021.1.19=h06a4308_1 # (updated from h06a4308_0 in May 26, 2021 maintenance update)
  - cachetools=4.2.0=pyhd3eb1b0_0
  - certifi=2020.12.5=py37h06a4308_0
  - cffi=1.14.0=py37he30daa8_1 # (updated from py37h2e261b9_0 in May 26, 2021 maintenance update)
  - chardet=3.0.4=py37h06a4308_1003
  - click=7.0=py37_0
  - cloudpickle=1.4.1=py_0
  - configparser=3.7.4=py37_0
  - cryptography=2.8=py37h1ba5d50_0
  - cudatoolkit=10.1.243=h6bb024c_0
  - cycler=0.10.0=py37_0
  - cython=0.29.15=py37he6710b0_0
  - decorator=4.4.1=py_0
  - dill=0.3.1.1=py37_1
  - docutils=0.15.2=py37_0
  - entrypoints=0.3=py37_0
  - flask=1.1.1=py_1
  - freetype=2.9.1=h8a8886c_1
  - future=0.18.2=py37_1
  - gast=0.3.3=py_0
  - gitdb=4.0.5=py_0
  - gitpython=3.1.0=py_0
  - google-auth=1.11.2=py_0
  - google-auth-oauthlib=0.4.1=py_2
  - google-pasta=0.2.0=py_0
  - grpcio=1.27.2=py37hf8bcb03_0
  - gunicorn=20.0.4=py37_0
  - h5py=2.10.0=py37h7918eee_0
  - hdf5=1.10.4=hb1b8bf9_0
  - icu=58.2=he6710b0_3
  - idna=2.8=py37_0
  - intel-openmp=2020.0=166
  - ipykernel=5.1.4=py37h39e3cac_0
  - ipython=7.12.0=py37h5ca1d4c_0
  - ipython_genutils=0.2.0=pyhd3eb1b0_1
  - isodate=0.6.0=py_1
  - itsdangerous=1.1.0=py37_0
  - jedi=0.17.2=py37h06a4308_1
  - jinja2=2.11.1=py_0
  - jmespath=0.10.0=py_0
  - joblib=0.14.1=py_0
  - jpeg=9b=h024ee3a_2
  - jupyter_client=5.3.4=py37_0
  - jupyter_core=4.6.1=py37_0
  - kiwisolver=1.1.0=py37he6710b0_0
  - krb5=1.17.1=h173b8e3_0 # (updated from 1.16.4 in May 26, 2021 maintenance update)
  - ld_impl_linux-64=2.33.1=h53a641e_7
  - libedit=3.1.20181209=hc058e9b_0
  - libffi=3.3=he6710b0_2 # (updated from 3.2.1 in May 26, 2021 maintenance update)
  - libgcc-ng=9.1.0=hdf63c60_0
  - libgfortran-ng=7.3.0=hdf63c60_0
  - libpng=1.6.37=hbc83047_0
  - libpq=12.2=h20c2e04_0 # (updated from 11.2 in May 26, 2021 maintenance update)
  - libprotobuf=3.11.4=hd408876_0
  - libsodium=1.0.16=h1bed415_0
  - libstdcxx-ng=9.1.0=hdf63c60_0
  - libtiff=4.1.0=h2733197_0
  - libuv=1.40.0=h7b6447c_0
  - lightgbm=3.1.1=py37h2531618_0
  - lz4-c=1.8.1.2=h14c3975_0
  - mako=1.1.2=py_0
  - markdown=3.1.1=py37_0
  - markupsafe=1.1.1=py37h14c3975_1
  - matplotlib-base=3.1.3=py37hef1b27d_0
  - mkl=2020.0=166
  - mkl-service=2.3.0=py37he8ac12f_0
  - mkl_fft=1.0.15=py37ha843d7b_0
  - mkl_random=1.1.0=py37hd6b4f25_0
  - ncurses=6.2=he6710b0_1
  - networkx=2.4=py_1
  - ninja=1.10.2=py37hff7bd54_0
  - nltk=3.4.5=py37_0
  - numpy=1.18.1=py37h4f9e942_0
  - numpy-base=1.18.1=py37hde5b4d6_1
  - oauthlib=3.1.0=py_0
  - olefile=0.46=py37_0
  - openssl=1.1.1k=h27cfd23_0 # (updated from 1.1.1i in May 26, 2021 maintenance update)
  - packaging=20.1=py_0
  - pandas=1.0.1=py37h0573a6f_0
  - paramiko=2.7.1=py_0
  - parso=0.7.0=py_0
  - patsy=0.5.1=py37_0
  - pexpect=4.8.0=pyhd3eb1b0_3
  - pickleshare=0.7.5=pyhd3eb1b0_1003
  - pillow=7.0.0=py37hb39fc2d_0
  - pip=20.0.2=py37_3
  - plotly=4.14.1=pyhd3eb1b0_0
  - prompt_toolkit=3.0.3=py_0
  - protobuf=3.11.4=py37he6710b0_0
  - psutil=5.6.7=py37h7b6447c_0
  - psycopg2=2.8.6=py37h3c74f83_1 # (updated from 2.8.4 in May 26, 2021 maintenance update)
  - ptyprocess=0.6.0=pyhd3eb1b0_2
  - pyasn1=0.4.8=py_0
  - pyasn1-modules=0.2.8=py_0
  - pycparser=2.19=py37_0
  - pygments=2.5.2=py_0
  - pyjwt=2.0.1=py37h06a4308_0
  - pynacl=1.3.0=py37h7b6447c_0
  - pyodbc=4.0.30=py37he6710b0_0
  - pyopenssl=19.1.0=pyhd3eb1b0_1
  - pyparsing=2.4.6=py_0
  - pysocks=1.7.1=py37_1
  - python=3.7.10=hdb3f193_0 # (updated from 3.7.6 in May 26, 2021 maintenance update)
  - python-dateutil=2.8.1=py_0
  - python-editor=1.0.4=py_0
  - pytorch=1.7.1=py3.7_cuda10.1.243_cudnn7.6.3_0
  - pytz=2019.3=py_0
  - pyzmq=18.1.1=py37he6710b0_0
  - readline=8.1=h27cfd23_0 # (updated from 7.0 in May 26, 2021 maintenance update)
  - requests=2.22.0=py37_1
  - requests-oauthlib=1.3.0=py_0
  - retrying=1.3.3=py37_2
  - rsa=4.0=py_0
  - s3transfer=0.3.4=pyhd3eb1b0_0
  - scikit-learn=0.22.1=py37hd81dba3_0
  - scipy=1.4.1=py37h0b6359f_0
  - setuptools=45.2.0=py37_0
  - simplejson=3.17.0=py37h7b6447c_0
  - six=1.14.0=py37h06a4308_0
  - smmap=3.0.4=py_0
  - sqlite=3.35.4=hdfb4753_0 # (updated from 3.31.1 in May 26, 2021 maintenance update)
  - sqlparse=0.4.1=py_0
  - statsmodels=0.11.0=py37h7b6447c_0
  - tabulate=0.8.3=py37_0
  - tk=8.6.10=hbc83047_0 # (updated from 8.6.8 in May 26, 2021 maintenance update)
  - torchvision=0.8.2=py37_cu101
  - tornado=6.0.3=py37h7b6447c_3
  - tqdm=4.42.1=py_0
  - traitlets=4.3.3=py37_0
  - typing_extensions=3.7.4.3=py_0
  - unixodbc=2.3.7=h14c3975_0
  - urllib3=1.25.8=py37_0
  - wcwidth=0.1.8=py_0
  - websocket-client=0.56.0=py37_0
  - werkzeug=1.0.0=py_0
  - wheel=0.34.2=py37_0
  - wrapt=1.11.2=py37h7b6447c_0
  - xz=5.2.5=h7b6447c_0 # (updated from 5.2.4 in May 26, 2021 maintenance update)
  - zeromq=4.3.1=he6710b0_3
  - zlib=1.2.11=h7b6447c_3
  - zstd=1.3.7=h0b5b093_0
  - pip:
      - astunparse==1.6.3
      - azure-core==1.10.0
      - azure-storage-blob==12.7.0
      - databricks-cli==0.14.1
      - diskcache==5.1.0
      - docker==4.4.1
      - gorilla==0.3.0
      - horovod==0.20.3
      - joblibspark==0.3.0
      - keras-preprocessing==1.1.2
      - koalas==1.5.0
      - mleap==0.16.1
      - mlflow==1.13.1
      - msrest==0.6.19
      - opt-einsum==3.3.0
      - petastorm==0.9.7
      - pyarrow==1.0.1
      - pyyaml==5.4
      - querystring-parser==1.2.4
      - seaborn==0.10.0
      - spark-tensorflow-distributor==0.1.0
      - tensorboard==2.3.0
      - tensorboard-plugin-wit==1.8.0
      - tensorflow==2.3.1
      - tensorflow-estimator==2.3.0
      - termcolor==1.1.0
      - xgboost==1.3.1
prefix: /databricks/conda/envs/databricks-ml-gpu

Paket Spark yang berisi modul Python

Paket Spark Modul Python Versi
graphframes graphframes 0.8.1-db1-spark3.0

Pustaka R

Pustaka R identik dengan Pustaka R di Runtime Bahasa Umum Databricks 7.6.

Pustaka Java dan Scala (Kluster Scala 2.12)

Selain pustaka Java dan Scala di Runtime Bahasa Umum Databricks 7.6, Runtime Bahasa Umum Databricks 7.6 ML berisi JAR berikut:

Kluster CPU

ID Grup ID Artefak Versi
com.typesafe.akka akka-actor_2.12 2.5.23
ml.combust.mleap mleap-databricks-runtime_2.12 0.17.3-4882dc3
ml.dmlc xgboost4j-spark_2.12 1.2.0
ml.dmlc xgboost4j_2.12 1.2.0
org.mlflow mlflow-client 1.13.1
org.scala-lang.modules scala-java8-compat_2.12 0.8.0
org.tensorflow spark-tensorflow-connector_2.12 1.15.0

Kluster GPU

ID Grup ID Artefak Versi
com.typesafe.akka akka-actor_2.12 2.5.23
ml.combust.mleap mleap-databricks-runtime_2.12 0.17.3-4882dc3
ml.dmlc xgboost4j-spark-gpu_2.12 1.2.0
ml.dmlc xgboost4j-gpu_2.12 1.2.0
org.mlflow mlflow-client 1.13.1
org.scala-lang.modules scala-java8-compat_2.12 0.8.0
org.tensorflow spark-tensorflow-connector_2.12 1.15.0