Skip to main content
Interania

Handling time formats during ingest

0votes
5updates
64views

This document explains Interana's default handling of time formats during ingest, and also two approaches for manually applying time conversions during ingest.

Default Time Detection

When Interana discovers a new column in the raw data, it applies some auto-detection heuristics to determine its data type. During this process, Interana looks for time columns as numbers (as Epoch time) and as strings (using time format strings).

Time As Numbers

Raw Data Detected As
{"foo" : 1487181646} Time (seconds)
{"foo" : 1487181646000} Time (milliseconds)
{"foo" : 1487181646000000} Time (microseconds)

Time As Strings (Just A Couple Examples)

Raw Data Detected As
{"foo" : "2017-08-09T11:22:24 +07:00"} Time (%Y-%m-%dT%H:%M:%S%Z)
{"foo" : "2017-07-16"} Time (%Y-%m-%d)

All Format Strings Used For Time Detection

Here is the full list of time formats Interana attempts to match when it discovers a new column in the data. These formats are applied using the C++ strptime() semantics.

Format String
%Y%m%d %Y-%m-%dT%H:%M:%S.%fZ %d/%m/%y  %m/%d/%Y %I:%M %p
%Y%m%d %H:%M:%S %Y-%m-%dT%H:%M:%SZ %d/%m/%y %H:%M %m/%d/%Y %I:%M:%S %p
%Y%m%d%H%M%S %d-%m-%Y %d/%m/%y %H:%M:%S %m/%d/%y
%Y%m%dT%H%M%S %d-%m-%y %d/%m/%y %I:%M %p %m/%d/%y %H:%M
%Y-%d-%m %d/%b/%Y:%H:%M:%S %d/%m/%y %I:%M:%S %p %m/%d/%y %H:%M:%S
%Y-%m-%d %d/%m/%Y %m-%d-%Y %m/%d/%y %I:%M %p 
%Y-%m-%d %H:%M %d/%m/%Y %H:%M %m-%d-%y %m/%d/%y %I:%M:%S %p
%Y-%m-%d %H:%M:%S %d/%m/%Y %H:%M:%S %m/%d/%Y %y-%d-%m
%Y-%m-%d %H:%M:%S.%f %d/%m/%Y %I:%M %p %m/%d/%Y %H:%M  %y-%m-%d
%Y-%m-%dT%H:%M:%S%Z %d/%m/%Y %I:%M:%S %p %m/%d/%Y %H:%M:%S  

Manual Time Transformation

There are two options for manual configuration of your time format.

Single, consistent time format

If your time column has a single, consistent time format (that happens not to be in Interana's default list) then you can set it explicitly in the Ingest Wizard. Click the Pencil icon next to the data type of your target column, choose a Time Format of Custom, and type in the desired time format. This time format will be applied using the C++ strptime() semantics.

Variable (Multiple) Time Formats

If your data has more than one time format in the same column, and you need to consolidate these formats, then you can use the Transformer Library. Click Manage Transformations and enable the time_convert transformer, passing it a list of time formats to try. These time formats will be applied using the Python3 strptime() semantics.

Are My Times Being Parsed Correctly?

If Interana is parsing your times correctly, the Ingest Wizard will preview them as epoch milliseconds (our internal storage format). 

If you'd like to see the conversion Interana is using internally, click the Pencil icon to see the details.

  • Was this article helpful?