Skip to main content
Interania

How to delete data from an overloaded string tier

0votes
6updates
21views

There may be times when the number of processes running on the string tier is so high that an import becomes blocked. When this happens, it appears that data is not coming in, or is coming in very slowly. If you look in the import-pipeline log, you will see the "String tier is overloaded, back off" error message.

It's important that you schedule downtime for the cluster to resolve this issue.

Deleting data from an overloaded string tier

If the string tier is overloaded because of one or more very high-cardinality columns, you can try to delete the columns with the ia column delete command. However, the command will most likely time out. The solution is to stop the string tier, let the threads die, and then restart it. This section walks you through this process.

To delete data when the string tier is overloaded, do the following:

  1. Log into the Interana config node on the cluster, and pause all jobs with the following command.
ia job -- all pause 
  1. Log in to each import node on the cluster and run the following command.
iactl stop string-server
  1. Log in to the first string node and run the following command.
iactl stop string-server-leaf
  1. Before going on to the next string node, run the following command to verify that the strings have stopped. The results should be zero.
pgrep string | wc -l 
  1. Log in to the other string nodes and run the following command.
iactl stop string-server-leaf

Now that all the services are stopped, clearing the string tier from overloaded processes, you can start the services up again.

  1. Log in to each import node and run the following command.
iactl start string-server
  1. Log in to each string node and run the following command.
iactl start string-server-leaf
  1. Verify the status of the string servers and leaf processes with the following command. The servers and processes should be running on all nodes.
iactl status
  1. You can now log in to the string tier and delete the desired column with the following command, substituting the actual table and column names for the variables.
ia column delete <table_name> <column_name> --run