Interana structured logs reference: Ingest monitoring
Use the ingest logs to analyze the performance of your data ingestion processes. This document is a reference for ingest monitoring structured logs.
Global attributes: event types
The following table lists the attributes of all event types.
Attribute | Description | Example |
---|---|---|
event_name | Name of the event | create_named_expression, dashboard_chart_update, uploaded_by_customer, purifier_finished, activity_start, activity_end |
event_type | Type of event | import, query, etc. |
event_class | Class in which the event belongs | query, typeahead, background, import, interana_request, sys_util, set_blob, progress_bar, add_counters, cancel_query, precacher, cardinality_monitor, login,saml_sso, users, config_action |
process | Managed process | query-api-server, purifier, import-pipeline, etc. |
hostname | Name of the cluster | interana1 |
__org__ | Organization of the event creation | Interana |
username | Username for the user who created the event | event_creator@interana.com |
severity | Severity of the event | INFO, WARNING, ERROR, EXCEPTION, FATAL |
query_api_id | Query ID number | 1214833815374918 |
user_id | User ID for the cluster | 15 |
ip_address | Cluster IP address | 10.10.10.10 |
__name__ | Username prefix | example, admin |
_t | Time of the event in human readable time format | Mon Jul 18 21:18:35 2017 |
__time__ | Timestamp of the query in Unix Epoch (POSIX) time (milliseconds) | 1468876731793 |
Common properties of ingest structured log events
- Events from the import-pipeline can be queried with:
- process = /opt/interana/backend/import_server/import_pipeline.py
- Events from the purifier can be queried with:
- process = purifier
The following tables list the common properties of structured log events for ingest pipeline and purifier.
Ingest pipeline |
|
---|---|
Each structured log event emitted by the ingest pipeline has the following properties:
|
For file related events, each event also includes:
|
Purifier |
---|
Purifier common properties include the following:
|
Key events
Unless otherwise noted, event types are found in the "event_name" field.
Ingest pipeline |
|
---|---|
File-Based Events Look for the events concerning the file's lifecycle through the ingest pipeline. For a healthy ingest, you should see one of each of the following events per file:
|
Stream-Based Events These events refer to a set of records that is flushed into the next stage of the ingest pipeline:
|
Generic Events—Apply to File-Based and Stream-Based Ingest These events refer to files, but they also apply to streaming ingest. Each time Interana flushes a batch of records, that batch is treated as a temp file and fires the following events:
|
|
Errors Interana has the following events for errors within the ingest pipeline:
|
Misc.
|
Purifier |
|
---|---|
Purifier ingest Purifier ingest events:
|
|
Errors These events contain an "error_count" field that shows the number of occurrences of these errors. Time column errors to look for:
|
Conversion Function Errors
Each of the events contains column information and the number of failures:
|
Useful named expressions
- Metrics
- Derived columns
- Dashboards
- Customer import health
- Global ingest stats
- Columns to set to groupable
Metrics
The following table lists useful named expressions for metrics.
Metrics—named expressions |
---|
Files Successfully Imported
|
Lines Transformed
|
New Columns Added
|
Purifier - Lines Parsed
|
Purifier - Lines Read
|
S3 List Request Cost
|
Derived Columns
The following table lists useful named expressions for derived columns.
Derived columns—named expressions |
---|
__s3_list_request_cost_denominator__
|
Dashboards
The following table lists useful named expressions for dashboards.
Dashboards—named experssions |
---|
Customer import health |
Import Heartbeat
|
Files Successfully Imported
|
Percentage of Files Successfully Imported
|
Lines Processed
|
New Columns Added By Table
|
Conversion Failures
|
Conversion Failures by Column, Type, Conversion Function
|
Time Column Errors
|
S3 List Requests - Last 2 Days
|
S3 List Request Cost - Last 2 Days
|
Global import stats |
Lines Transformed
|
Files Processed
|
Purifier - Lines Parsed
|
Purifier - Lines Read
|
Files Imported By Customer - Raw Size
|
Files Processed By Customer - Raw Size
|
Columns to set to groupable |
|