Skip to main content
Interania

Pipeline creation and common functions

0votes
6updates
155views

This topic describes the user flow for creating a pipeline and common functions.

Requirements

Log into your cluster:

$ ssh ubuntu@<ip address>

Create a table for the pipeline to work on: 

$ /opt/interana/backend/import_server/tools/create_table.py -t 'Table Name' /path/to/dataset/data_set_file.json

Create your Pipeline

Navigate to /opt/interana/backend/import_server. This is where the pipeline tool scripts exist.

$ cd /opt/interana/backend/import_server

Use the 'Table Name' created previously and create a new pipeline with it:

$ ./setup_pipeline.py -t 'Table Name' -p 'Pipeline Name' -d file_system

  • Set a pattern: ~/path/to/datasets/*.json
  • Follow prompts if YES to advance settings

The "Wait Seconds" interval will run your pipeline in that interval if you decide to setup a Forever job later on.

Start your pipeline by creating a job

Create a one-time job

$ ./pipe_jobs.py -c <pipeline_id> 2016-09-13 2016-09-14

$ ./pipe_jobs.py -c <pipeline_id> yesterday tomorrow

Forever job (runs forever according to the wait interval set)

$ ./pipe_jobs.py -f <pipeline_id> yesterday today

Check to see your import was successful by tailing the log 

$ sudo tail -f /var/log/interana/import-pipeline.log

Refresh your server and check your data.

Edit your pipeline || job

Add / edit / remove transformers 

$ ./setup_pipeline.py -t 'Table Name' -p 'Pipeline Name'

View all of your jobs | pipelines

$ ./pipe_jobs.py -s all

Pause / resume / delete jobs

$ ./pipe_jobs.py --pause job_id

$ ./pipe_jobs.py --resume job_id

$ ./pipe_jobs.py --delete job_id

$ ./pipe_jobs.py --help

  • Was this article helpful?