Use Azure KeyVault in the Interana import pipeline
When using Azure KeyVault, the Interana import pipeline connects to and downloads data from Azure Containers in the same way as any Azure pipeline. But in this case, the import pipeline must connect to KeyVault to retrieve the credentials to be able to connect to the Azure Container.
First, Interana connects to Azure AD (Active Directory) to retrieve an access token. Interana uses this access token to authenticate with the KeyVault and retrieve the credentials. Once Interana has the credentials, the import pipeline works the same as a normal Azure pipeline.
Pipeline Parameters
When creating an Azure KeyVault pipeline, you'll still use the data source type as azure_blob
and two of the same parameters, storage_account
and container
.
Common Parameters
The data source type is azure_blob
.
Pipeline parameters
These parameters are required, as with any Azure pipeline.
Parameter | Required? | Description |
---|---|---|
storage_account | Yes | The name of the Azure Storage account. |
container | Yes | The name of the container to search and download files from. |
Azure KeyVault parameters
We added the following parameters to support Azure KeyVault.
Parameter | Required? | Description |
---|---|---|
certificate | Yes | The name of private key, as stored in the secret table. |
client_id | Yes | The ID of the application connecting to the blob store. |
thumbprint | Yes | The thumbprint of the private key stored in the secret table and used to authenticate with Azure AD. |
tenant | Yes | The instance of the Azure AD endpoint used to authenticate with. |
vault | Yes | The name of KeyVault to retrieve the credentials from. |
secret_id | Yes | The ID of the Azure Blob credentials stored in the KeyVault. |
secret_version | Yes | The version of the secret, which is the "friendly" name of the secret. This is the name from the Get Secret API. |
api_version | No | The parameter for the get secret request; default is 2016-10-01, which is the current version of the API. |
resource | No | The resource we will authenticate to Azure AD to use. We recommend that you set this to https://vault.azure.net. |
Certificate parameters
Rather than including the certificate file on every import node, we created a new table, secret
, to store the contents of the certificate. The table only has the following columns:
Parameter | Required? | Description |
---|---|---|
name | Yes | The name provided by the user to identify the certificate. |
secret | Yes | The contents of the certificate. |
Add a certificate to the table
Use the following command to add a new secret:
./insert_cert_to_db.py --add <certificate name> <path to certificate>
For example:
./insert_cert_to_db.py --add my_secret my_certificate.pem
Remove a certificate from the table
Use the following command to remove a secret:
./insert_cert_to_db.py --remove <certificate name>
For example:
./insert_cert_to_db.py --remove my_secret
Example of setting up an Azure KeyVault pipeline
Here is an example of how to set up an Azure KeyVault pipeline.
First, use the insert_cert_to_db.py
script to add the certificate (my_cert.pem
) to the secret
table:
./insert_cert_to_db.py --add test_cert my_cert.pem
Next, use a config file to create a table and pipeline:
ia table create config-file azure_keyvault_test_table.txt
With the table and pipeline created, we can now create a job:
ia job create azure_test_pipeline onetime 2014-01-01 2014-01-02
Sample Config File
{ "table": { "name": "azure_test", "type": "Event", "time_column": "time", "time_column_format": "milliseconds", "shard_keys": ["shard_key"] }, "ingest": [ { "name": "azure_test_pipeline", "data_source_type": "azure_blob", "table_name": "azure_test", "data_source_parameters": { "file_pattern": "{year:04d}/{month:02d}/{day:02d}/", "storage_account": "interanadevfiles", "container": "integration-test-files", "certificate": "test_cert", "client_id": "7h35und4y5-1u5h-41vv4y5-5501f411c3", "thumbprint": "A4B0E7EDE645A3C7C248FE7C61703EA043D3D0CD", "tenant": "https://login.online.com/", "vault": "firstvault2", "secret_id": "2447209f9a998230343ea48b14f2f34620f8", "secret_version": "dev-storagekey" } } ] }