Configure the Streaming runner for MySQL🔗

When using MySQL as a streaming data source, you must provide a MySQL driver for the Streaming runner to use. This document describes how to install the driver in your cloud infrastructure, and how to supply the Streaming runner with the environment variables needed to use the driver. The environment variables you need to set are dependent on whether you are deploying the Streaming runner on AWS or Azure.

Prerequisites🔗

Download an appropriate MySQL driver from MySQL Community Downloads. Version 8.0.33 or later is required.
Generate a SHA512 value, which the Streaming runner will use via an environment variable. To generate the value, follow the process given in Generating the SHA512 value, below.

AWS🔗

Upload the driver package to an S3 bucket that's in the same account space as the Streaming runner and accessible by the Streaming runner.
The Streaming runner needs access to the S3 bucket, especially the permission s3:GetObject to download the package.
Define the SHA512, S3 bucket, and S3 key values, using the following environment variables:
- MYSQL_DRIVER_SHA512
- MYSQL_DRIVER_S3_BUCKET
- MYSQL_DRIVER_S3_KEY

Example:

MYSQL_DRIVER_SHA512="3eabd70f9a947918f434a44923a8e3ff4c3fbc93e6c90f4992c94d804860ef3d09f09fb4fbf905d53a39b51aab965ecfd65f2ff2aa105387b3c8f49c18d7713c"
MYSQL_DRIVER_S3_BUCKET="bucket-name"
MYSQL_DRIVER_S3_KEY="path/to/driver.jar"

Note

The MYSQL_DRIVER_S3_BUCKET, and MYSQL_DRIVER_S3_KEY environment variables, and their values, need to be provided to launch the Streaming runner.
If you are using an AWS-managed MySQL-compatible database as a source for AWS, read the MySQL for AWS documentation in the AWS DMS (Database Migration Service) docs.

Azure🔗

Upload the driver package to an Azure Blob Storage location that's in the same resource group as the Streaming runner and accessible by the Streaming runner.
The Microsoft.Storage/storageAccounts/blobServices/containers/blobs/read permission is required for the Streaming runner to download the driver. For more information, read Get Blob.
Define the SHA512, account name, container name, and Azure Blob name values, using the following environment variables:
- MYSQL_DRIVER_SHA512
- MYSQL_DRIVER_ABS_ACCOUNT_NAME
- MYSQL_DRIVER_ABS_CONTAINER_NAME
- MYSQL_DRIVER_ABS_BLOB_NAME

Example:

MYSQL_DRIVER_SHA512="3eabd70f9a947918f434a44923a8e3ff4c3fbc93e6c90f4992c94d804860ef3d09f09fb4fbf905d53a39b51aab965ecfd65f2ff2aa105387b3c8f49c18d7713c"
MYSQL_DRIVER_ABS_ACCOUNT_NAME="storageaccount"
MYSQL_DRIVER_ABS_CONTAINER_NAME="storagecontainer"
MYSQL_DRIVER_ABS_BLOB_NAME="path/to/driver.jar"

Note

The MYSQL_DRIVER_ABS_ACCOUNT_NAME, MYSQL_DRIVER_ABS_CONTAINER_NAME, and MYSQL_DRIVER_ABS_BLOB_NAME environment variables, and their values, need to be provided to download the files. The Streaming runner will launch only if the files are successfully downloaded.
The installation template will create a managed identity that the Streaming runner uses to authenticate against required Azure resources. When adding a MySQL driver to a storage account, you must also grant the role Storage Blob Data Contributor to the created managed identity against the storage account that the driver has been placed into.

Generating the SHA512 value🔗

You must set either the MYSQL_DRIVER_SHA512 or the DISABLE_MYSQL_DRIVER_SHA512 environment variable, regardless of the chosen cloud provider. We strongly recommend using the MYSQL_DRIVER_SHA512 variable, and providing the SHA512 value of the driver .jar file you are using. For example:

MYSQL_DRIVER_SHA512="3eabd70f9a947918f434a44923a8e3ff4c3fbc93e6c90f4992c94d804860ef3d09f09fb4fbf905d53a39b51aab965ecfd65f2ff2aa105387b3c8f49c18d7713c"

Note

It's not recommended to set the DISABLE_MYSQL_DRIVER_SHA512 variable to "true", because this means the Streaming runner will no longer validate the hash of the driver file.
If a hash is provided in MYSQL_DRIVER_SHA512, and it doesn't match the hash of the downloaded file, then the Streaming runner won't start.

To generate the SHA512 value, open a terminal window and run the command appropriate to your operating system:

Operating system	Command
Windows	`certutil -hashfile "<path-to-file>" SHA512`
MacOS	`shasum -a 512 <path-to-file>`
Linux	`sha512sum <path-to-file>`

Where <path-to-file> is the full path and name of the .jar driver file.