Interana structured logs reference: Query usage
Use the query usage logs to analyze how your organization is using Interana to run queries. This lets you get insight into the types and frequency of query activities being performed by your users.
This document is a reference for query usage structured logs.
Global attributes: event types
The following table lists the attributes of all event types.
| Attribute | Description | Example |
|---|---|---|
| event_name | Name of the event | create_named_expression, dashboard_chart_update, uploaded_by_customer, purifier_finished, activity_start, activity_end |
| event_type | Type of event | import, query, etc. |
| event_class | Class in which the event belongs | query, typeahead, background, import, interana_request, sys_util, set_blob, progress_bar, add_counters, cancel_query, precacher, cardinality_monitor, login,saml_sso, users, config_action |
| process | Managed process | query-api-server, purifier, import-pipeline, etc. |
| hostname | Name of the cluster | interana1 |
| __org__ | Organization of the event creation | Interana |
| username | Username for the user who created the event | event_creator@interana.com |
| severity | Severity of the event | INFO, WARNING, ERROR, EXCEPTION, FATAL |
| query_api_id | Query ID number | 1214833815374918 |
| user_id | User ID for the cluster | 15 |
| ip_address | Cluster IP address | 10.10.10.10 |
| __name__ | Username prefix | example, admin |
| _t | Time of the event in human readable time format | Mon Jul 18 21:18:35 2017 |
| __time__ | Timestamp of the query in Unix Epoch (POSIX) time (milliseconds) | 1468876731793 |
Loading a dashboard
When an Interana user navigates to a dashboard, the following sequence of events occurs:
| Event | What's Useful Here | Key Fields Logged |
|---|---|---|
| page load | A user is viewing a dashboard | { "username": "jeff@interana.com", "event_name": "add_counters", "user_action": "page-load", "page_name": "dashboard", "current_page": "/?dashboard=dashboard-jeff-1&name=Jeff Dashboard", } |
| render dashboard charts | The dashboard has 3 charts | { "username": "jeff@interana.com", "event_name": "add_counters", "action_target": "render-dashboard-charts", "page": "dashboard", "dashboard_total_chart_count": 3, "current_page": "/?dashboard=dashboard-jeff-1&name=Jeff Dashboard", } |
| get the dashboard definition from the server | It took 74 milliseconds to load the dashboard metadata from the server (this doesn't include the actual dashboard results, just the overall structure) | { "username": "jeff@interana.com", "event_name": "add_counters", "request_url": "/api/blob/get", "latency": 74, "method": "GET", "status": 200, "current_page": "/?dashboard=dashboard-jeff-1&name=Jeff Dashboard", } |
| for each dashboard panel, prepare to run the query | This is an unsampled bar view query against the query_loopback table | { "username": "jeff@interana.com", "event_name": "drillstate", "client_qid": 8370391849101, "is_dashboard": true, "table": "query_loopback", "view": "bar" "drillstate_agg_arg_types": ["COUNT_STAR"], "drillstate_filter_text": "", "drillstate_do_not_sample": true, "drillstate_group_by": ["event_name"], } |
| see if the query result is already cached | We are able to use the cached result. (A cache "hit" means we are using the cache, and a "miss" means we need to run the query from scratch.) | { "username": "jeff@interana.com", "event_name": "query_cache_hit", "client_qid": 8370391849101, "is_dashboard": true, "table": "query_loopback", "cache_key": "aCyVmBvUxuVh3b99PQ4UVp", } |
| query completed | Even more detailed information about the definition of the query that was run (or loaded from the cache) to populate this chart | { "username": "jeff@interana.com", "event_name": "query_end", "is_dashboard": true, "dashboard_chart_id": "chartM1510780912509L66", "table_name": "query_loopback", "priority": "high", "start": 1531764797804, "now": 1531768397000, "global_filters": [], "time_offsets": [0], "orderby": [{ "text": "Count Events", "direction": "desc", "type": "COUNT_STAR", "key": 0, "arg": null }], "processed_end_time": 1531768397804, "max_cache_age_seconds": 604800, "sample_limit": 20, "where": [""], "duration": 52, "do_not_sample": true, "num_groups": 10, "end": 1531768397804, "start_time_ms": 1531768397851, "time_windows": [{ "num": "1", "unit": "auto" }], "view_type": "time", "processed_start_time": 1531764797804, "groups": [], "aggs": [{ "unique_key": null, "type": "COUNT_STAR", "arg": "" }], "time": 1531768397 } |
| dashboard chart loaded | Performance stats about the query, and types of objects referenced by the query | { "username": "jeff@interana.com", "event_name": "add_counters", "counter_table": "chart_load", "current_page": "/?dashboard=dashboard-jeff-1&name=Jeff Dashboard", "query_cpu_ms": 75, "query_js_ms": 91, "query_python_ms": 1220, "sampled": true, "query_sample_rate": 0.08, "overall_sampling_quality": "sufficient", "table": "query_loopback", "query_match_percent": "100.0", "aggregator": "COUNT_STAR", "start": 1522566000001, "end": 1531768397727, "end_minus_start": 9202397726, "query_count": 1, "view": "time", "compare_type": null, "measures_count": 1, "filter_groups_count": 0, "columns_count": 0, "metrics_count": 0, "custom_measures_count": 0, "filter_type": "inputs", "max_groups": 0, "times_count": 1, "query_rows_examined": 107421, "query_rows_matched": 107421, "query_slow_leaf_ms": 33, "query_num_machines": "3", "query_slow_process_ms": 32, "query_median_leaf_ms": 27, } |
Named expression
This section covers the structured logs for named expressions.
create_named expression
Logs the "name" being operating upon, not the full context with user-created details. The following table lists the attributes that are used when a named expression is created.
| Attribute | Description | Example |
|---|---|---|
| filters | Set of filters in the named expression | |
| pnp_id | 0, 8, 1642 | |
| creator_uid | UID for the creator of the event | 15 |
| start | Start time in Unix epoch time (milliseconds) | 1468271920000 |
| end | End time in Unix epoch time (milliseconds) | 1468876720000 |
| description | Description of the named event | This cohort calculates...... |
| condition.qualifier | For cohorts: translation of the choices 'exactly', 'at most', 'at least' | >=, <=, = |
| condition.num | For cohorts: number input of the qualifier measure | 1, 500 |
| entity_column | Shard key used in the named expression | userid |
| table_name | Table where the shard key is located | example_events |
| time_offset | Offset time for the named expression | 0 |
named_expression_delete
Logs the "name" being operating upon, not the full context with user-created details. See the previous create_named_expression table, for attributes that can be associated with a named expression.
named_expression_update
Logs the "name" being operating upon, not the full context with user-created details. See the previous create_named_expression table, for attributes that can be associated with a named expression.
Query processing: processing_results
The following table lists the attributes used in query processing.
| Attribute | Description | Example |
|---|---|---|
| is_dashboard | false, true | |
| dashboard_requestor | null, precacher, None | |
| dashboard_requestor_id | null, dashboard-example-1, dashboard-newuser-0, dashboard-newsample-2 | |
| wait_and_run_time | 0.08718085289001465 | |
| lifetime_started | 723 | |
| priority | hipri, lowpri, low, high, 10, 0, 1 | |
| slots_needed | 16 |
Run a query: drillstate
These are logged when a user runs a query using the Go button, or when a user enters the Explorer page.
This event is NOT logged for queries using the Interana external API. You can use the "get_request_log" event with "endpoint" = "api_view" to count those queries.
The following table lists the attributes used when a query is run.
| Attribute | Description | Example |
|---|---|---|
| drillstate_agg_arg_types | The measure used in the query. The prefix denotes the shard key, and the suffix denotes the aggregator. | user_id.Has Authenticated with example_ADMIN |
| drillstate_hash | Unique event identifier | de03b1b45755f7unique75501d5537example |
| view | View in Explorer | time, table, number, bar, pie, hist |
| drillstate_group_by | Group by's used (set) | ["user_id.email"], ["properties.InputType","userId.traits.dg-user-type"] |
| drillstate_start | Time frame start date/time, Unix epoch time | 1468448486000 |
| drillstate_end | Time frame end date/time, Unix epoch time | 1469053286778 |
| drillstate_time_zone_offset | Time zone offset, Unix epoch time | 288000000 |
| drillstate_do_not_sample | Query was not sampled | true |
| drillstate_filter_text | Filters used | ["(`user_id.email` != \"*null*\")"] |
| drillstate_filter_type | Filter types | text, null, None |
| drillstate_show_all_others | Show all others used in queries | false, true |
| token_key | Token key used | false, vVABPnKsAWxstcExample, BW4c2MYExamplewCExample, e5rMV41mRLAcin1R+Example |
| dashboard_dashboard_id | Dashboard ID | None, dashboard-example-1, dashboard-newsample-0 |
| sending_query | This event has been deprecated |
Select a typeahead: timed_request_top_values
The following table lists the attributes used when a typeahead is selected.
| Attribute | Description | Example ` |
|---|---|---|
| column | Column selected in the typeahead | user_id.email, event_type, feature, browser |
| event_class | Event class used | typeahead |
Group chart: group_chart_log
The following table lists the attributes used for group charts.
| Attribute | Description | Example |
|---|---|---|
| is_dashboard | ||
| line_number | ||
| explanation | The full data set was examined, with only 0 matches. Cannot rule out sample bias, delta distribution. All 3 matches localized in one shard out of 6, with only 3 matches. Cannot rule out sample bias. | |
| matches | [0,0,0,0,0,21], [0,0,3,0,0,0], [2744403,2662227] | |
| token_key | false, vVABPnExampleKey9k0000 | |
| scale_confidence | 0.111111, 0.888888, 0.991223, 1 | |
| dashboard_owner | admin, newuser, admin.lam | |
| message_template | null | |
| dashboard_count | None | |
| shard_scale | 1.0 | |
| dashboard_dashboard_id | null, dashboard-example-1, dashboard-newuser-0, dashboard-newsample-2 | |
| dashboard_version | None, 2.19-4f353e1, 2.18-97c9d16, 2.20-bef72dd | |
| external_api | false, true | |
| dashboard_requester | precacher, None | |
| event_class | query | |
| total_events_matched | 1, 2, 35, 60953 |