Skip to main content
Interania

Interana structured logs reference

1votes
44updates
130views
This applies tov2.24

You can ingest Interana structured logs to analyze Interana usage (query usage) and Interana performance (ingest monitoring). This document is a reference for the parameters contained in Interana structured logs. This reference is organized as follows:

Ingest monitoring structured logs

This section covers ingest monitoring structured logs, and includes the following topics:

Common properties of ingest structured log events

  • Events from the import-pipeline can be queried with:
    • process = /opt/interana/backend/import_server/import_pipeline.py
  • Events from the purifier can be queried with:
    • process = purifier

The following tables list the common properties of structured log events for ingest pipeline and purifier.

Ingest pipeline
 

Each structured log event emitted by the ingest pipeline has the following properties:

  • pipeline_id
  • job_id
  • inst_id
  • table_id
  • table_name
  • continuous - 1 = forever, 0 = one-time

For file related events, each event also includes:

  • batch_id
  • original_filename
  • remote_filename_md5
  • file_size - transformed size
  • original_size - raw size
  • line_count - lines after transformation
  • lines_dropped - lines lost in the transformation phase; Note: this is only valid for pipelines using the transformer library (generators)
  • lines_total - total number of lines in the file: lines successfully transformed and lines that were dropped
  • iteration_date
  • concat_filename
Purifier

Purifier common properties include the following:

  • batch_id
  • pipeline_id
  • job_id
  • inst_id
  • table_id
  • purifier_filename - same as the concat_filename in the ingest pipeline

Key events

Unless otherwise noted, event types are found in the "event_name" field.

Ingest pipeline
 

File-Based Events

Look for the events concerning the file's lifecycle through the ingest pipeline. For a healthy ingest, you should see one of each of the following events per file:

  • uploaded_by_customer—represents the time the file was made available to Interana. Interana uses the modtime of the file.
  • detected_by_interana—time when Interana first scans the iteration date. Every file found in a given iteration date has the same detected_by_interana time
  • get_request—file downloaded.
  • list_request—whenever we make a list request to an S3 bucket; S3 only
  • found_files—emitted at end of each iteration date scan; "file_count" = number of files found overall, not new files to import

Stream-Based Events 

These events refer to a set of records that is flushed into the next stage of the ingest pipeline:

  • batch_closed—emitted when a batch of records from our internal bus is flushed to the transformer stage of the import pipeline

Generic Events—Apply to File-Based and Stream-Based Ingest

These events refer to files, but they also apply to streaming ingest. Each time Interana flush3w a batch of records, that batch is treated as a temp file and fires the following events:

  • file_transformation_start– about to start transforming the file
  • file_transformation_complete– transformation of the file has finished
  • purification_start– about to run the purifier on the file
  • purification_end– purifier has completed, so the import of the file has finished.  When we see this event, we generally consider the file to successfully imported.

Errors

Interana has the following events for errors within the ingest pipeline:

  • transformer_failure - error in the transformation phase, for both generators and classic transformers; might be some extra info in the "result" column
  • purifier_failure - error in the purification phase; "error_code" column has the purifier's return code
  • error_processing_file - general file import error -- emitted in both transformer_failure and purifier_failure cases, but will catch any other errors within the file's import pipeline lifecycle; slightly more detail in the "result" column
  • insufficient_disk_space - import job cannot proceed due to not enough disk space 
  • insufficient_disk_percent - import job cannot proceed due to not enough free disk percent

Misc.

  • terminate_called - when we set the exit flag, either pausing the job or shutting down the import-pipeline service
  • uncaught_exception - error we couldn't recover from, so the job crashed; "exception" contains exception

 

Purifier
 

Purifier ingest

Purifier ingest events:

  • purifier_start—start of the Purifier ingest.
  • purifier_finish—end of the Purifier ingest.
  • parseJsonChunk (activity_name field)—emitted when a chunk of the file has finished parsing and includes the number of lines read (lines_read) and successfully parsed (lines_parsed).
  • detectNewColumns (activity_name field)—new columns detected.

Errors

These events contain an "error_count" field that shows the number of occurrences of these errors.

Time column errors to look for:

  • count_invalid_timestamp
  • count_timestamp_far_in_future
  • count_parse_error_or_exception

Conversion Function Errors

  • import_conversion_failed

Each of the events contains column information and the number of failures:

  • table_id
  • column_name
  • column_type
  • conversion_function
  • conversion_function_params
  • conversion_failure_count

Useful named expressions

Metrics

The following table lists useful named expressions for metrics.

Metrics—named expressions

Files Successfully Imported

  • Aggregator
    • Count Unique: original_filename
  • Filter
    • event_name: purification_end
  • Divided By
    • Aggregator
      • Count Unique: original_filename
    • Filter
      • event_name: detected_by_interana

Lines Transformed

  • Aggregator
    • Sum: line_count
  • Filter
    • event_name: purification_start

New Columns Added

  • Aggregator
    • Count Events
  • Filters
    • process: purifier
  • activity_name: detectNewColumns

Purifier - Lines Parsed

  • Aggregator
    • Sum: lines_parsed
  • Filters
    • process: purifier
    • activity_name: parseJson_chunk

Purifier - Lines Read

  • Aggregator
    • Sum: lines_read
  • Filters
    • process: purifier
    • activity_name: parseJson_chunk

S3 List Request Cost

  • Aggregator
    • Count Events
  • Filter
    • event_name: list_request
    • Divided By
      • Aggregator
        • Maximum: __s3_list_request_cost_denominator__


Derived Columns

The following table lists useful named expressions for derived columns.

Derived columns—named expressions

__s3_list_request_cost_denominator__

long get_s3_list_request_cost_denominator() {

return 200000;

}

 

Dashboards

The following table lists useful named expressions for dashboards.

Dashboards—named experssions
Customer import health

Import Heartbeat

  • View: Time
  • Measure
    • Count Events
  • Compare
    • table_id
    • table_name
    • pipeline_id
  • Filters
    • process: /opt/interana/backend/import_server/import_pipeline.py
    • event_name: purification_end

Files Successfully Imported

  • View: Bar
  • Measure
    • Count Events
  • Compare
    • event_name
  • Filter
    • process: import_pipeline
    • event_name: purification_end, detected_by_interana

Percentage of Files Successfully Imported

  • View: Time
  • Measure
    • Files Successfully Imported
  • Filter
    • process: import_pipeline

Lines Processed

  • View: Time
  • Measure
    • Lines Transformed
    • Purifier - Lines Parsed
    • Purifier - Lines Read
  • Filter
    • process: import_pipeline, purifier

New Columns Added By Table

  • View: Time
  • Measure
    • New Columns Added
  • Compare
    • table_id
  • Filter
    • process: purifier

Conversion Failures

  • View: Stacked Area Time
  • Measure
    • Sum: conversion_failure_count
  • Compare
    • table_id
    • column_name
  • Filter
    • process: purifier
    • event_name: import_conversion_failed

Conversion Failures by Column, Type, Conversion Function

  • View: table
  • Measure
    • Sum: conversion_failure_count
  • Compare
    • table_id
    • column_name
    • column_type
    • conversion_function
    • conversion_function_params
  • Filter
    • process: purifier
    • event_name: import_conversion_failed

Time Column Errors

  • View: Time
  • Measure
    • Sum: error_count
  • Compare

    • table_id, event_name

  • Filter
    • process: purifier
    • event_name: count_invalid_timestamp, count_timestamp_far_in_future, count_parse_error_or_exception

S3 List Requests - Last 2 Days

  • View: Number
  • Measure
    • Count Events
  • Filter
    • process: import_pipeline
    • event_name: list_request

S3 List Request Cost - Last 2 Days

  • View: Number
  • Measure
    • S3 List Request Cost
  • Filter
    • process: import_pipeline
    • event_name: list_request
Global import stats

Lines Transformed

  • View: Stacked Area Time
  • Measure
    • Lines Transformed
  • Compare
    • customer
  • Filters
    • process: import_pipeline

Files Processed

  • View: Stacked Area Time
  • Measure
    • Count Unique: original_filename
  • Compare
    • customer
  • Filters
    • process: import_pipeline
    • event_name: purification_end

Purifier - Lines Parsed

  • View: Stacked Area Time
  • Measure
    • Purifier - Lines Parsed
  • Compare
    • customer
  • Filters
    • process: purifier

Purifier - Lines Read

  • View: Stacked Area Time
  • Measure
    • Purifier - Lines Read
  • Compare
    • customer
  • Filters
    • process: purifier

Files Imported By Customer - Raw Size

  • View: Stacked Area Time
  • Measure
    • Sum: original_size
  • Compare
    • customer
  • Filters
    • process: import_pipeline

Files Processed By Customer - Raw Size

  • View: Stacked Area Time
  • Measure
    • Purifier - Lines Read
  • Compare
    • customer
  • Filters
    • process: import_pipeline
Columns to set to groupable
  • table_id
  • pipeline_id
  • job_id
  • customer_id
  • iteration_date

 

Query usage structured logs

This section covers ingest monitoring structured logs, and includes the following topics:

Global attributes: event types

The following table lists the attributes of all event types.

Attribute  Description  Example
event_name Name of the event create_named_expression, dashboard_chart_update, uploaded_by_customer, purifier_finished, activity_start, activity_end
event_type Type of event import, query, etc.
event_class Class in which the event belongs query, typeahead, background, import, interana_request,
sys_util, set_blob, progress_bar, add_counters, cancel_query,
precacher, cardinality_monitor, login,saml_sso, users,
config_action
process Managed process query-api-server, purifier, import-pipeline, etc.
hostname Name of the cluster interana1
__org__ Organization of the event creation Interana
username Username for the user who created the event event_creator@interana.com
severity Severity of the event INFO, WARNING, ERROR, EXCEPTION, FATAL
query_api_id Query ID number  1214833815374918
user_id User ID for the cluster 15
ip_address Cluster IP address 10.10.10.10
__name__ Username prefix example, admin
_t Time of the event in human readable time format Mon Jul 18 21:18:35 2017
__time__ Timestamp of the query in Unix Epoch (POSIX) time (milliseconds) 1468876731793

 

Shard keys

The following table lists the attributes used for shard keys.

Attribute  Description  Example `
shardkey Tracks one user or one import pipeline user_id
transactionid    

Named expression: create_named expression

The following table lists the attributes that are used when a named expression is created.

Attribute  Description  Example
filters Set of filters in the named expression   
pnp_id   0, 8, 1642
creator_uid UID for the creator of the event 15
start Start time in Unix epoch time (milliseconds) 1468271920000
end End time in Unix epoch time (milliseconds) 1468876720000
description Description of the named event This cohort calculates......
condition.qualifier For cohorts: translation of the choices 'exactly', 'at most', 'at least' >=, <=, =
condition.num For cohorts: number input of the qualifier measure 1, 500
entity_column Shard key used in the named expression userid
table_name Table where the shard key is located example_events
time_offset Offset time for the named expression 0

Query processing: processing_results

The following table lists the attributes used in query processing.

Attribute  Description  Example 
is_dashboard   false, true
dashboard_requestor   null, precacher, None
dashboard_requestor_id   null, dashboard-example-1, dashboard-newuser-0, dashboard-newsample-2
wait_and_run_time   0.08718085289001465
lifetime_started   723
priority   hipri, lowpri, low, high, 10, 0, 1
slots_needed   16

Run a query: drillstate 

When a query is run click GO or Enter the Explorer.

This event is NOT logged for queries using the Interana external API. You can use the "get_request_log" event with "endpoint" = "api_view" to count those queries.

The following table lists the attributes used when a query is run.

Attribute  Description  Example 
drillstate_agg_arg_types The measure used in the query. The prefix denotes the shard key, and the suffix denotes the aggregator. user_id.Has Authenticated with example_ADMIN
drillstate_hash Unique event identifier de03b1b45755f7unique75501d5537example
view View in Explorer time, table, number, bar, pie, hist
drillstate_group_by Group by's used (set) ["user_id.email"], ["properties.InputType","userId.traits.dg-user-type"]
drillstate_start Time frame start date/time, Unix epoch time 1468448486000
drillstate_end Time frame end date/time, Unix epoch time 1469053286778
drillstate_time_zone_offset Time zone offset, Unix epoch time 288000000
drillstate_do_not_sample Query was not sampled true
drillstate_filter_text Filters used ["(`user_id.email` != \"*null*\")"]
drillstate_filter_type Filter types text, null, None
drillstate_show_all_others Show all others used in queries false, true
token_key Token key used false, vVABPnKsAWxstcExample, BW4c2MYExamplewCExample, e5rMV41mRLAcin1R+Example
dashboard_dashboard_id Dashboard ID None, dashboard-example-1, dashboard-newsample-0

Select a typeahead: timed_request_top_values

The following table lists the attributes used when a typeahead is selected.

Attribute  Description  Example `
column Column selected in the typeahead user_id.email, event_type, feature, browser
event_class Event class used typeahead

Group chart: group_chart_log

The following table lists the attributes used for group charts.

 

Attribute  Description  Example 
is_dashboard    
line_number    
explanation   The full data set was examined, with only 0 matches. Cannot rule out sample bias, delta distribution. All 3 matches localized in one shard out of 6, with only 3 matches. Cannot rule out sample bias.
matches   [0,0,0,0,0,21], [0,0,3,0,0,0], [2744403,2662227]
token_key   false, vVABPnExampleKey9k0000
scale_confidence   0.111111, 0.888888, 0.991223, 1
dashboard_owner   admin, newuser, admin.lam
message_template   null
dashboard_count   None
shard_scale   1.0
dashboard_dashboard_id   null, dashboard-example-1, dashboard-newuser-0, dashboard-newsample-2
dashboard_version   None, 2.19-4f353e1, 2.18-97c9d16, 2.20-bef72dd
external_api   false, true
dashboard_requester   precacher, None
event_class   query
total_events_matched   1, 2, 35, 60953

Interana logging data dictionary

For information about the Interana logs that show your organization's use of the Interana platform, see the Interana logging data dictionary.

  • Was this article helpful?