Use the Transformer Library to transform data before ingest

Step 1: Identify the transformations you want to apply

It's important to have a complete understanding of the problems I want to solve with my transformations. Since I want the mobile data to integrate seamlessly with my desktop data, I'll want to make the mobile events as similar as possible to the desktop events, while retaining the ability to separate mobile and desktop usage when I need to.

Given these requirements, here are the transformations I'd like to apply:

1. My mobile timestamp field has a different name and than the desktop timestamp field. Since the time key on our events table is the timestamp column, I'll need to resolve these differences.
2. We don't pass a source field in our mobile events. I know that we'll want to use the source field to be able to differentiate between application sources, so I'd like to add that field to our mobile events.
3. The desktop event attributes message_id and is_private are contained in one mobile event attribute: message_info. I'll want to separate these two pieces of information into separate columns, just as in our desktop events.
4. Since these mobile events are being generated from a new code path, I'll want to be able to easily debug errors in these events. Since we're using a client timestamp for our mobile events, there's no guarantee that we'll be able to track down the file an erroneous event came from in S3. I'll make sure to add the file name that contained the event so that we can diagnose issues faster.
5. Our mobile events are gzipped to save storage costs, so I'll need to decompress these files before import.

Now that I understand the transformers required, I can start developing my transformer configuration.

Step 2: Write the transformations

To implement transformers, we use the import server. This example writes transformations from scratch. Don't worry, it's not too tough!

An Interana Transformer Library configuration file is composed of a list of transformer "steps." Every file (and event within that file) will pass through these steps on their way to an output file.

I start by constructing a very basic configuration that will decompress, decode, and load the JSON event into a python dictionary (for the actual transformations), and dump the event back into JSON. We do this in a list of four steps, which are lists themselves:

[
["gunzip"],
["decode"],
["json_load"],
["json_dump"],
]

Now that I have that done, it's time to start implementing the actual data transformations. I'll start with the first transformation, the timestamp.

Step 3: Assemble and test the transformations

Now that we've written our transformers, it's time to test them! First, assemble your different transformers between your json_load and json_dump steps:

[
["gunzip"],
["decode"],
["json_load"],
["time_convert", {"read_formats": ["%Y-%m-%dT%H:%M:%S.%fZ"], "write_format": "%Y-%m-%d %H:%M:%S.%f", "column": "client_timestamp"}],
["rename", {"column": "client_timestamp", "new_name": "timestamp"}],
["add_label", {"column": "source", "label": "mobile"}],
["regex_extract", {"column": "message_info", "output_columns": ["message_id", "is_private"], "regex": "(?:(\d*)/([a-z]*)|.*)"}],
["omit", {"columns": ["message_info"]}],
["add_filename", {"column": "filename"}],
["json_dump"],
]

Remember that events pass through the transformer configuration sequentially, so the order is important! For instance, you would not want to omit the field message_info before you extract its contents.

Now it's time to test our your configuration. Go ahead and place some test events in a file on your import node, along with your configuration file. I'll put my test event into file test.gz and my configuration in config.

Next, run generators.py to test your transformers. Use this command to test config on test.gz and place output in out:

/opt/interana/backend/import_server/generators.py -i test.gz -o out -c config

You'll see a message reporting the success or failure of your transformer, with specific error messages if something went wrong. For instance, if we misspell the column parameter in the rename step:

["rename", {"colum": "client_timestamp", "new_name": "timestamp"}],

We'll get the following error message:

Once you see that your transformation has completed successfully, inspect the output and ensure that everything turned out all right:

Step 4: Apply the transformer configuration to the pipeline

Now that we're happy with the results of our transformer configuration, it's time to apply it to our import pipeline so that every file we import is transformed. If creating a new pipeline, specify that you want to apply the transformer configuration as follows:

/opt/interana/backend/import_server/setup_pipeline.py -t cooltech_events -p mobile_events --transformers ~/config

You can also apply the config configuration to an existing pipeline mobile_events for table cooltech_events using the same command above.

The transformer configuration will be stored in the Interana configuration database in the data_sources.transformer table, if you ever need to retrieve it.

At this point, we're done! I can now create an import job with the updated pipeline and have both mobile and desktop events in the same table, integrated despite different formats for seamless querying.