Skip to main content
Interania

Interana structured logs reference: Query usage

0votes
16updates
128views

Use the query usage logs to analyze how your organization is using Interana to run queries. This lets you get insight into the types and frequency of query activities being performed by your users. 

This document is a reference for query usage structured logs.

Global attributes: event types

The following table lists the attributes of all event types.

Attribute  Description  Example
event_name Name of the event create_named_expression, dashboard_chart_update, uploaded_by_customer, purifier_finished, activity_start, activity_end
event_type Type of event import, query, etc.
event_class Class in which the event belongs query, typeahead, background, import, interana_request,
sys_util, set_blob, progress_bar, add_counters, cancel_query,
precacher, cardinality_monitor, login,saml_sso, users,
config_action
process Managed process query-api-server, purifier, import-pipeline, etc.
hostname Name of the cluster interana1
__org__ Organization of the event creation Interana
username Username for the user who created the event event_creator@interana.com
severity Severity of the event INFO, WARNING, ERROR, EXCEPTION, FATAL
query_api_id Query ID number  1214833815374918
user_id User ID for the cluster 15
ip_address Cluster IP address 10.10.10.10
__name__ Username prefix example, admin
_t Time of the event in human readable time format Mon Jul 18 21:18:35 2017
__time__ Timestamp of the query in Unix Epoch (POSIX) time (milliseconds) 1468876731793

Loading a dashboard

When an Interana user navigates to a dashboard, the following sequence of events occurs:

Event  What's Useful Here Key Fields Logged 
page load A user is viewing a dashboard {
    "username": "jeff@interana.com",
    "event_name": "add_counters",
    "user_action": "page-load",
    "page_name": "dashboard",
    "current_page": "/?dashboard=dashboard-jeff-1&name=Jeff Dashboard",
}
render dashboard charts The dashboard has 3 charts {
    "username": "jeff@interana.com",
    "event_name": "add_counters",
    "action_target": "render-dashboard-charts",
    "page": "dashboard",
    "dashboard_total_chart_count": 3,
    "current_page": "/?dashboard=dashboard-jeff-1&name=Jeff Dashboard",
}
get the dashboard definition from the server It took 74 milliseconds to load the dashboard metadata from the server (this doesn't include the actual dashboard results, just the overall structure) {
    "username": "jeff@interana.com",
    "event_name": "add_counters",
    "request_url": "/api/blob/get",
    "latency": 74,
    "method": "GET",
    "status": 200,
    "current_page": "/?dashboard=dashboard-jeff-1&name=Jeff Dashboard",
}
for each dashboard panel, prepare to run the query This is an unsampled bar view query against the query_loopback table {
    "username": "jeff@interana.com",
    "event_name": "drillstate",
    "client_qid": 8370391849101,
    "is_dashboard": true,
    "table": "query_loopback",
    "view": "bar"
    "drillstate_agg_arg_types": ["COUNT_STAR"],
    "drillstate_filter_text": "",
    "drillstate_do_not_sample": true,
    "drillstate_group_by": ["event_name"],
}
see if the query result is already cached We are able to use the cached result. (A cache "hit" means we are using the cache, and a "miss" means we need to run the query from scratch.) {
    "username": "jeff@interana.com",
    "event_name": "query_cache_hit",
    "client_qid": 8370391849101,
    "is_dashboard": true,
    "table": "query_loopback",
    "cache_key": "aCyVmBvUxuVh3b99PQ4UVp",
}
query completed Even more detailed information about the definition of the query that was run (or loaded from the cache) to populate this chart {
    "username": "jeff@interana.com",
    "event_name": "query_end",
    "is_dashboard": true,
    "dashboard_chart_id": "chartM1510780912509L66",
    "table_name": "query_loopback",
    "priority": "high",
    "start": 1531764797804,
    "now": 1531768397000,
    "global_filters": [],
    "time_offsets": [0],
    "orderby": [{
        "text": "Count Events",
        "direction": "desc",
        "type": "COUNT_STAR",
        "key": 0,
        "arg": null
    }],
    "processed_end_time": 1531768397804,
    "max_cache_age_seconds": 604800,
    "sample_limit": 20,
    "where": [""],
    "duration": 52,
    "do_not_sample": true,
    "num_groups": 10,
    "end": 1531768397804,
    "start_time_ms": 1531768397851,
    "time_windows": [{
        "num": "1",
        "unit": "auto"
    }],
    "view_type": "time",
    "processed_start_time": 1531764797804,
    "groups": [],
    "aggs": [{
        "unique_key": null,
        "type": "COUNT_STAR",
        "arg": ""
    }],
    "time": 1531768397
}
dashboard chart loaded Performance stats about the query, and types of objects referenced by the query {
    "username": "jeff@interana.com",
    "event_name": "add_counters",
    "counter_table": "chart_load",
    "current_page": "/?dashboard=dashboard-jeff-1&name=Jeff Dashboard",
    "query_cpu_ms": 75,
    "query_js_ms": 91,
    "query_python_ms": 1220,
    "sampled": true,
    "query_sample_rate": 0.08,
    "overall_sampling_quality": "sufficient",
    "table": "query_loopback",
    "query_match_percent": "100.0",
    "aggregator": "COUNT_STAR",
    "start": 1522566000001,
    "end": 1531768397727,
    "end_minus_start": 9202397726,
    "query_count": 1,
    "view": "time",
    "compare_type": null,
    "measures_count": 1,
    "filter_groups_count": 0,
    "columns_count": 0,
    "metrics_count": 0,
    "custom_measures_count": 0,
    "filter_type": "inputs",
    "max_groups": 0,
    "times_count": 1,
    "query_rows_examined": 107421,
    "query_rows_matched": 107421,
    "query_slow_leaf_ms": 33,
    "query_num_machines": "3",
    "query_slow_process_ms": 32,
    "query_median_leaf_ms": 27,
}

Named expression 

This section covers the structured logs for named expressions.

create_named expression

Logs the "name" being operating upon, not the full context with user-created details. The following table lists the attributes that are used when a named expression is created.

Attribute  Description  Example
filters Set of filters in the named expression   
pnp_id   0, 8, 1642
creator_uid UID for the creator of the event 15
start Start time in Unix epoch time (milliseconds) 1468271920000
end End time in Unix epoch time (milliseconds) 1468876720000
description Description of the named event This cohort calculates......
condition.qualifier For cohorts: translation of the choices 'exactly', 'at most', 'at least' >=, <=, =
condition.num For cohorts: number input of the qualifier measure 1, 500
entity_column Shard key used in the named expression userid
table_name Table where the shard key is located example_events
time_offset Offset time for the named expression 0

named_expression_delete

Logs the "name" being operating upon, not the full context with user-created details. See the previous create_named_expression table, for attributes that can be associated with a named expression.

named_expression_update

Logs the "name" being operating upon, not the full context with user-created details. See the previous create_named_expression table, for attributes that can be associated with a named expression.

Query processing: processing_results

The following table lists the attributes used in query processing.

Attribute  Description  Example 
is_dashboard   false, true
dashboard_requestor   null, precacher, None
dashboard_requestor_id   null, dashboard-example-1, dashboard-newuser-0, dashboard-newsample-2
wait_and_run_time   0.08718085289001465
lifetime_started   723
priority   hipri, lowpri, low, high, 10, 0, 1
slots_needed   16

Run a query: drillstate 

These are logged when a user runs a query using the Go button, or when a user enters the Explorer page.

This event is NOT logged for queries using the Interana external API. You can use the "get_request_log" event with "endpoint" = "api_view" to count those queries.

The following table lists the attributes used when a query is run.

Attribute  Description  Example 
drillstate_agg_arg_types The measure used in the query. The prefix denotes the shard key, and the suffix denotes the aggregator. user_id.Has Authenticated with example_ADMIN
drillstate_hash Unique event identifier de03b1b45755f7unique75501d5537example
view View in Explorer time, table, number, bar, pie, hist
drillstate_group_by Group by's used (set) ["user_id.email"], ["properties.InputType","userId.traits.dg-user-type"]
drillstate_start Time frame start date/time, Unix epoch time 1468448486000
drillstate_end Time frame end date/time, Unix epoch time 1469053286778
drillstate_time_zone_offset Time zone offset, Unix epoch time 288000000
drillstate_do_not_sample Query was not sampled true
drillstate_filter_text Filters used ["(`user_id.email` != \"*null*\")"]
drillstate_filter_type Filter types text, null, None
drillstate_show_all_others Show all others used in queries false, true
token_key Token key used false, vVABPnKsAWxstcExample, BW4c2MYExamplewCExample, e5rMV41mRLAcin1R+Example
dashboard_dashboard_id Dashboard ID None, dashboard-example-1, dashboard-newsample-0
sending_query This event has been deprecated  

Select a typeahead: timed_request_top_values

The following table lists the attributes used when a typeahead is selected.

Attribute  Description  Example `
column Column selected in the typeahead user_id.email, event_type, feature, browser
event_class Event class used typeahead

Group chart: group_chart_log

The following table lists the attributes used for group charts.

Attribute  Description  Example 
is_dashboard    
line_number    
explanation   The full data set was examined, with only 0 matches. Cannot rule out sample bias, delta distribution. All 3 matches localized in one shard out of 6, with only 3 matches. Cannot rule out sample bias.
matches   [0,0,0,0,0,21], [0,0,3,0,0,0], [2744403,2662227]
token_key   false, vVABPnExampleKey9k0000
scale_confidence   0.111111, 0.888888, 0.991223, 1
dashboard_owner   admin, newuser, admin.lam
message_template   null
dashboard_count   None
shard_scale   1.0
dashboard_dashboard_id   null, dashboard-example-1, dashboard-newuser-0, dashboard-newsample-2
dashboard_version   None, 2.19-4f353e1, 2.18-97c9d16, 2.20-bef72dd
external_api   false, true
dashboard_requester   precacher, None
event_class   query
total_events_matched   1, 2, 35, 60953