Skip to main content
Interania

How to list and read S3 bucket files

0votes
7updates
20views

Amazon Simple Storage Service (S3) is a web service offered by Amazon Web Services. If you use Amazon's AWS service, you upload your data to an S3 bucket from which they are imported into Interana. 

This document demonstrates how to list and read files in an S3 bucket. In the event that you encounter an import error, this can help you troubleshoot the problem.

List and read S3 bucket files

Listing and reading the files in your S3  bucket can help you discover a potential timestamp error in a file. This could be the cause of the error, when you can't find an error in the import pipeline. 

To list and read s3 bucket files, do the following:
  1. Log in to the config node of the Interana cluster.
  2. Check for the running jobs in the table with the error.
ia job list -s running --unsafe | grep "<table_name>"
  1. Find S3 bucket for the given pipeline.
ia pipeline show <pipeline_name> --unsafe
  1. Display a list of files in the S3 bucket.
aws s3 ls s3://<s3_bucket_value>/<file_pattern>/<year>/<month>/<day>/<hour>

Example: aws s3 ls s3://asana-logs/api-perf-logs/2017/10/18/14
  1. Check to make sure there is enough space on the node to download the latest date-specified folder. If there's not enough space, download to a local system.
  2. Copy the latest date-specified folder.
aws s3 cp s3://<s3_bucket_value>/<file_pattern>/<year>/<month>/<day>/<hour> .
  1. Read the compressed file. The following example uses zless to read the compressed file.  The zless utility allows you to view the contents of a compressed or plain text, one screen at a time. 
Example: zless api-perf-logs-1-2017-10-18-14-59-48-7f4820d4-34e9-4264-ba21-examplefiledb.gz
  1. Check for the timestamp key from the table and search for the same in the downloaded file.
  2. Convert to human readable timestamp (https://www.epochconverter.com/) and compare, checking to make sure all the data is posted.

What's Next

This is what was achieved and what was omitted in this howto.

  • Was this article helpful?