Skip to main content
Interania

How to apply time conversions during ingest

0votes
15updates
148views

This document explains Interana's default handling of time formats during ingest, and how to manually apply time conversions during ingest.

 

Interana time detection

When Interana discovers a new column in the raw data, it applies auto-detection heuristics to determine the data type. During this process, Interana looks for time columns as numbers (as Epoch time) and as strings (using time format strings).

Time as numbers

The following table provides examples of time number formats.

Raw Data Detected As
{"foo" : 1487181646} Time (seconds)
{"foo" : 1487181646000} Time (milliseconds)
{"foo" : 1487181646000000} Time (microseconds)

Time as strings 

The following table provides a couple of examples of time string formats.

Raw Data Detected As
{"foo" : "2017-08-09T11:22:24 +07:00"} Time (%Y-%m-%dT%H:%M:%S%Z)
{"foo" : "2017-07-16"} Time (%Y-%m-%d)

Complete list of format strings Interana uses for time detection

The following table provides a complete list of time formats Interana attempts to match when it discovers a new data column. These formats are applied using the C++ strptime() semantics.

Format String
%Y%m%d %Y-%m-%dT%H:%M:%S.%fZ %d/%m/%y  %m/%d/%Y %I:%M %p
%Y%m%d %H:%M:%S %Y-%m-%dT%H:%M:%SZ %d/%m/%y %H:%M %m/%d/%Y %I:%M:%S %p
%Y%m%d%H%M%S %d-%m-%Y %d/%m/%y %H:%M:%S %m/%d/%y
%Y%m%dT%H%M%S %d-%m-%y %d/%m/%y %I:%M %p %m/%d/%y %H:%M
%Y-%d-%m %d/%b/%Y:%H:%M:%S %d/%m/%y %I:%M:%S %p %m/%d/%y %H:%M:%S
%Y-%m-%d %d/%m/%Y %m-%d-%Y %m/%d/%y %I:%M %p 
%Y-%m-%d %H:%M %d/%m/%Y %H:%M %m-%d-%y %m/%d/%y %I:%M:%S %p
%Y-%m-%d %H:%M:%S %d/%m/%Y %H:%M:%S %m/%d/%Y %y-%d-%m
%Y-%m-%d %H:%M:%S.%f %d/%m/%Y %I:%M %p %m/%d/%Y %H:%M  %y-%m-%d
%Y-%m-%dT%H:%M:%S%Z %d/%m/%Y %I:%M:%S %p %m/%d/%Y %H:%M:%S  

Applying time conversions

This section explains the options for manually configuring a time format:

Setting a single time format

If your time column has a single, consistent time format that is not in Interana's default list, you can set it explicitly in the Ingest Wizard.

To set a single time format in the Ingest Wizard, do the following:
  1. Open the Ingest Wizard, and click the Pencil icon next to the data type of your target column.
  2. Choose a Time Format of Custom.
  3. Type in the desired time format. In the following example, the time format is applied using the C++ strptime() semantics.
  4. Click Save.

Consolidating multiple time formats

If your data has more than one time format in the same column, you can consolidate these formats using the Transformer Library.

To consolidate multiple time formats in the same column, do the following:
  1. Open the Transformer Library and click Manage Transformations. 
  2. Enable the time_convert transformer. 
  3. Pass a list of time formats to try.

These time formats will be applied using the Python3 strptime() semantics.

Verifying and modifying time settings

When Interana is parsing times correctly, the Ingest Wizard provides a preview in epoch milliseconds, the default Interana storage format. You can view the type of time conversion Interana performed, and modify it as necessary. 

To view and modify the Interana internal time conversion, do the following:
  1. Open the Ingest Wizard.
  2. Click the Pencil icon to see the details.
  3. Select a different Time Format from the drop-down list.
  4. Click Save.

  • Was this article helpful?