Secrets overview🔗
Several functions in Maia require access to secrets stored in your cloud provider's secret manager. These include:
- Identification between installed agents and your Maia account.
- Storing database credentials to be used when configuring a pipeline's source.
- Storing source credentials for use in connectors.
Using the cloud provider's secret manager means that Maia never needs to store the values of your passwords or keys—it's all handled by your cloud provider. This provides an extra assurance of security for your credentials.
Secret manager flow in Maia🔗
- Users authenticate themselves to access Maia—where secrets, including customer references, are managed securely.
- Users invoke Designer to create and configure data pipelines and to define secrets for an existing cloud provider's secret within the corresponding secret definition.
- Task requests are sent to the Agent Gateway for direct communication with customer-hosted agents, and the scheduler coordinates pipeline executions based on schedules and triggers.
- A Matillion-hosted Maia Foundation runner securely stores and retrieves customer secrets from the Matillion-hosted vault as necessary for pipeline execution.
- Customer secret vaults securely store and retrieve customer secrets. A customer-hosted Maia Foundation runner provides the option to execute data pipelines on-premises or in the customer's cloud network.
Security Benefits🔗
- Maia never directly stores passwords or keys, relying instead on your cloud provider for secure storage.
- Your cloud provider's secret manager offers robust security measures for protecting credentials, ensuring the confidentiality and integrity of sensitive information.
Agents🔗
The Maia Foundation runner is responsible for processing pipeline tasks, which are individual units of work within a data integration workflow. These tasks handle data integration and transformation operations by securely connecting to data sources and targets.
The Maia Foundation runner can be configured in two ways:
- Full SaaS: Fully managed by Matillion and resides in Matillion's VPC.
- Hybrid SaaS: Runs inside a user's VPC.
Note
Maia Foundation runners can access stored secrets, which serve as the repository for all your secrets. In your projects, information is limited to the names of Maia Foundation runners and the secrets they can access. Maia doesn't provide direct access to the values of secrets. However, these secrets can be used within your projects to access your data services.
Using secrets🔗
In Maia, secrets and secret definitions are stored at the project level. To use a secret:
- Create a named secret in your cloud provider's secret manager.
- Add the secret name to the secrets stored in Maia. Doing this stores only the name and location of the secret, not the secret's actual value.
- Call the secret by name when you need to use the credentials—for example in a data source connector. The secret name is resolved at runtime to obtain the credentials stored in the secret key.
Note
If you need to store multiple passwords and keys, each should be in a separate, named secret.
Error
If an agent cannot access a password, it will result in an error says "The agent can't access the customer's secret manager".
Secret managers🔗
To learn more about your cloud provider's secret manager technology, read the corresponding documentation: