net.opentsdb.core

Class Internal

java.lang.Object

net.opentsdb.core.Internal

public final class Internal
extends Object

This class is not part of the public API.

 ,____________________________,
 | This class is reserved for |
 | OpenTSDB's internal usage! |
 `----------------------------'
       \                   / \  //\
        \    |\___/|      /   \//  \\
             /0  0  \__  /    //  | \ \
            /     /  \/_/    //   |  \  \

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	Internal.Cell Represents a single data point in a row.
static class	Internal.KeyValueComparator Helper to sort a row with a mixture of millisecond and second data points.

Field Summary

Fields

Modifier and Type	Field and Description
static short	FLAG_BITS
static short	FLAGS_MASK
static short	LENGTH_MASK

Method Summary

Methods

Modifier and Type	Method and Description
static long	baseTime(TSDB tsdb, byte[] row) Extracts the timestamp from a row key.
static byte[]	buildQualifier(long timestamp, short flags) Returns a 2 or 4 byte qualifier based on the timestamp and the flags.
static int	compareQualifiers(byte[] a, int offset_a, byte[] b, int offset_b) Compares two data point byte arrays with offsets.
static org.hbase.async.KeyValue	complexCompact(org.hbase.async.KeyValue kv)
static ArrayList<Internal.Cell>	extractDataPoints(ArrayList<org.hbase.async.KeyValue> row, int estimated_nvalues) Breaks down all the values in a row into individual Internal.Cells sorted on the qualifier.
static ArrayList<Internal.Cell>	extractDataPoints(org.hbase.async.KeyValue column) Extracts the data points from a single column.
static double	extractFloatingPointValue(byte[] values, int value_idx, byte flags)
static long	extractIntegerValue(byte[] values, int value_idx, byte flags)
static byte[]	extractQualifier(byte[] qualifier, int offset) Extracts the 2 or 4 byte qualifier from a compacted byte array
static byte[]	fixFloatingPointValue(byte flags, byte[] value) Returns a corrected value if this is a floating point value to fix.
static byte	fixQualifierFlags(byte flags, int val_len) Fix the flags inside the last byte of a qualifier.
static boolean	floatingPointValueToFix(byte flags, byte[] value) Returns whether or not this is a floating value that needs to be fixed.
static short	getFlagsFromQualifier(byte[] qualifier) Parses the flag bits from the qualifier
static short	getFlagsFromQualifier(byte[] qualifier, int offset) Parses the flag bits from the qualifier
static int	getOffsetFromQualifier(byte[] qualifier) Returns the offset in milliseconds from the row base timestamp from a data point qualifier
static int	getOffsetFromQualifier(byte[] qualifier, int offset) Returns the offset in milliseconds from the row base timestamp from a data point qualifier at the given offset (for compacted columns)
static short	getQualifierLength(byte[] qualifier) Returns the length, in bytes, of the qualifier: 2 or 4 bytes
static short	getQualifierLength(byte[] qualifier, int offset) Returns the length, in bytes, of the qualifier: 2 or 4 bytes
static org.hbase.async.Scanner	getScanner(Query query)
static Map<String,String>	getTags(TSDB tsdb, byte[] row)
static long	getTimestampFromQualifier(byte[] qualifier, long base_time) Returns the absolute timestamp of a data point qualifier in milliseconds
static long	getTimestampFromQualifier(byte[] qualifier, long base_time, int offset) Returns the absolute timestamp of a data point qualifier in milliseconds
static byte	getValueLengthFromQualifier(byte[] qualifier) Returns the length of the value, in bytes, parsed from the qualifier
static byte	getValueLengthFromQualifier(byte[] qualifier, int offset) Returns the length of the value, in bytes, parsed from the qualifier
static boolean	inMilliseconds(byte qualifier) Determines if the qualifier is in milliseconds or not
static boolean	inMilliseconds(byte[] qualifier) Determines if the qualifier is in milliseconds or not
static boolean	inMilliseconds(byte[] qualifier, byte offset) Determines if the qualifier is in milliseconds or not
static String	metricName(TSDB tsdb, byte[] id)
static short	metricWidth(TSDB tsdb)
static Internal.Cell	parseSingleValue(org.hbase.async.KeyValue column) Extracts a Cell from a single data point, fixing potential errors with the qualifier flags

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

FLAG_BITS

public static final short FLAG_BITS

See Also:

Const.FLAG_BITS, Constant Field Values

LENGTH_MASK

public static final short LENGTH_MASK

See Also:

Const.LENGTH_MASK, Constant Field Values

FLAGS_MASK

public static final short FLAGS_MASK

See Also:

Const.FLAGS_MASK, Constant Field Values

Method Detail

getScanner

public static org.hbase.async.Scanner getScanner(Query query)

See Also:

TsdbQuery.getScanner()

metricName

public static String metricName(TSDB tsdb,
                byte[] id)

See Also:

RowKey.metricName(net.opentsdb.core.TSDB, byte[])

baseTime

public static long baseTime(TSDB tsdb,
            byte[] row)

Extracts the timestamp from a row key.

getTags

public static Map<String,String> getTags(TSDB tsdb,
                         byte[] row)

See Also:

Tags.getTags(net.opentsdb.core.TSDB, byte[])

extractIntegerValue

public static long extractIntegerValue(byte[] values,
                       int value_idx,
                       byte flags)

See Also:

RowSeq.extractIntegerValue(byte[], int, byte)

extractFloatingPointValue

public static double extractFloatingPointValue(byte[] values,
                               int value_idx,
                               byte flags)

See Also:

RowSeq.extractFloatingPointValue(byte[], int, byte)

metricWidth

public static short metricWidth(TSDB tsdb)

See Also:

TSDB.metrics_width()

complexCompact

public static org.hbase.async.KeyValue complexCompact(org.hbase.async.KeyValue kv)

See Also:

CompactionQueue.complexCompact(java.util.ArrayList, int, boolean)

parseSingleValue

public static Internal.Cell parseSingleValue(org.hbase.async.KeyValue column)

Extracts a Cell from a single data point, fixing potential errors with the qualifier flags

Parameters:

column - The column to parse

Returns:

A Cell if successful, null if the column did not contain a data point (i.e. it was meta data) or failed to parse

Throws:

IllegalDataException - if the qualifier was not 2 bytes long or it wasn't a millisecond qualifier

Since:

2.0

extractDataPoints

public static ArrayList<Internal.Cell> extractDataPoints(org.hbase.async.KeyValue column)

Extracts the data points from a single column. While it's meant for use on a compacted column, you can pass any other type of column and it will be returned. If the column represents a data point, a single cell will be returned. If the column contains an annotation or other object, the result will be an empty array list. Compacted columns will be split into individual data points. Note: This method does not account for duplicate timestamps in qualifiers.

Parameters:

column - The column to parse

Returns:

An array list of data point Internal.Cell objects. The list may be empty if the column did not contain a data point.

Throws:

IllegalDataException - if one of the cells cannot be read because it's corrupted or in a format we don't understand.

Since:

2.0

extractDataPoints

public static ArrayList<Internal.Cell> extractDataPoints(ArrayList<org.hbase.async.KeyValue> row,
                                         int estimated_nvalues)

Breaks down all the values in a row into individual Internal.Cells sorted on the qualifier. Columns with non data-point data will be discarded. Note: This method does not account for duplicate timestamps in qualifiers.

Parameters:

row - An array of data row columns to parse

estimated_nvalues - Estimate of the number of values to compact. Used to pre-allocate a collection of the right size, so it's better to overshoot a bit to avoid re-allocations.

Returns:

An array list of data point Internal.Cell objects. The list may be empty if the row did not contain a data point.

Throws:

IllegalDataException - if one of the cells cannot be read because it's corrupted or in a format we don't understand.

Since:

2.0

compareQualifiers

public static int compareQualifiers(byte[] a,
                    int offset_a,
                    byte[] b,
                    int offset_b)

Compares two data point byte arrays with offsets. Can be used on:

• Single data point columns
• Compacted columns

Warning: Does not work on Annotation or other columns

Parameters:

a - The first byte array to compare

offset_a - An offset for a

b - The second byte array

offset_b - An offset for b

Returns:

0 if they have the same timestamp, -1 if a is less than b, 1 otherwise.

Since:

2.0

fixQualifierFlags

public static byte fixQualifierFlags(byte flags,
                     int val_len)

Fix the flags inside the last byte of a qualifier.

OpenTSDB used to not rely on the size recorded in the flags being correct, and so for a long time it was setting the wrong size for floating point values (pretending they were encoded on 8 bytes when in fact they were on 4). So overwrite these bits here to make sure they're correct now, because once they're compacted it's going to be quite hard to tell if the flags are right or wrong, and we need them to be correct to easily decode the values.

Parameters:

flags - The least significant byte of a qualifier.

val_len - The number of bytes in the value of this qualifier.

Returns:

The least significant byte of the qualifier with correct flags.

floatingPointValueToFix

public static boolean floatingPointValueToFix(byte flags,
                              byte[] value)

Returns whether or not this is a floating value that needs to be fixed.

OpenTSDB used to encode all floating point values as `float' (4 bytes) but actually store them on 8 bytes, with 4 leading 0 bytes, and flags correctly stating the value was on 4 bytes. (from CompactionQueue)

Parameters:

flags - The least significant byte of a qualifier.

value - The value that may need to be corrected.

fixFloatingPointValue

public static byte[] fixFloatingPointValue(byte flags,
                           byte[] value)

Returns a corrected value if this is a floating point value to fix.

OpenTSDB used to encode all floating point values as `float' (4 bytes) but actually store them on 8 bytes, with 4 leading 0 bytes, and flags correctly stating the value was on 4 bytes.

This function detects such values and returns a corrected value, without the 4 leading zeros. Otherwise it returns the value unchanged. (from CompactionQueue)

Parameters:

flags - The least significant byte of a qualifier.

value - The value that may need to be corrected.

Throws:

IllegalDataException - if the value is malformed.

inMilliseconds

public static boolean inMilliseconds(byte[] qualifier,
                     byte offset)

Determines if the qualifier is in milliseconds or not

Parameters:

qualifier - The qualifier to parse

offset - An offset from the start of the byte array

Returns:

True if the qualifier is in milliseconds, false if not

Since:

2.0

inMilliseconds

public static boolean inMilliseconds(byte[] qualifier)

Determines if the qualifier is in milliseconds or not

Parameters:

qualifier - The qualifier to parse

Returns:

True if the qualifier is in milliseconds, false if not

Since:

2.0

inMilliseconds

public static boolean inMilliseconds(byte qualifier)

Determines if the qualifier is in milliseconds or not

Parameters:

qualifier - The first byte of a qualifier

Returns:

True if the qualifier is in milliseconds, false if not

Since:

2.0

getOffsetFromQualifier

public static int getOffsetFromQualifier(byte[] qualifier)

Returns the offset in milliseconds from the row base timestamp from a data point qualifier

Parameters:

qualifier - The qualifier to parse

Returns:

The offset in milliseconds from the base time

Throws:

IllegalArgument - if the qualifier is null or empty

Since:

2.0

getOffsetFromQualifier

public static int getOffsetFromQualifier(byte[] qualifier,
                         int offset)

Returns the offset in milliseconds from the row base timestamp from a data point qualifier at the given offset (for compacted columns)

Parameters:

qualifier - The qualifier to parse

offset - An offset within the byte array

Returns:

The offset in milliseconds from the base time

Throws:

IllegalDataException - if the qualifier is null or the offset falls outside of the qualifier array

Since:

2.0

getValueLengthFromQualifier

public static byte getValueLengthFromQualifier(byte[] qualifier)

Returns the length of the value, in bytes, parsed from the qualifier

Parameters:

qualifier - The qualifier to parse

Returns:

The length of the value in bytes, from 1 to 8.

Throws:

IllegalArgument - if the qualifier is null or empty

Since:

2.0

getValueLengthFromQualifier

public static byte getValueLengthFromQualifier(byte[] qualifier,
                               int offset)

Returns the length of the value, in bytes, parsed from the qualifier

Parameters:

qualifier - The qualifier to parse

offset - An offset within the byte array

Returns:

The length of the value in bytes, from 1 to 8.

Throws:

IllegalArgument - if the qualifier is null or the offset falls outside of the qualifier array

Since:

2.0

getQualifierLength

public static short getQualifierLength(byte[] qualifier)

Returns the length, in bytes, of the qualifier: 2 or 4 bytes

Parameters:

qualifier - The qualifier to parse

Returns:

The length of the qualifier in bytes

Throws:

IllegalArgument - if the qualifier is null or empty

Since:

2.0

getQualifierLength

public static short getQualifierLength(byte[] qualifier,
                       int offset)

Returns the length, in bytes, of the qualifier: 2 or 4 bytes

Parameters:

qualifier - The qualifier to parse

offset - An offset within the byte array

Returns:

The length of the qualifier in bytes

Throws:

IllegalArgument - if the qualifier is null or the offset falls outside of the qualifier array

Since:

2.0

getTimestampFromQualifier

public static long getTimestampFromQualifier(byte[] qualifier,
                             long base_time)

Returns the absolute timestamp of a data point qualifier in milliseconds

Parameters:

qualifier - The qualifier to parse

base_time - The base time, in seconds, from the row key

Returns:

The absolute timestamp in milliseconds

Throws:

IllegalArgument - if the qualifier is null or empty

Since:

2.0

getTimestampFromQualifier

public static long getTimestampFromQualifier(byte[] qualifier,
                             long base_time,
                             int offset)

Returns the absolute timestamp of a data point qualifier in milliseconds

Parameters:

qualifier - The qualifier to parse

base_time - The base time, in seconds, from the row key

offset - An offset within the byte array

Returns:

The absolute timestamp in milliseconds

Throws:

IllegalArgument - if the qualifier is null or the offset falls outside of the qualifier array

Since:

2.0

getFlagsFromQualifier

public static short getFlagsFromQualifier(byte[] qualifier)

Parses the flag bits from the qualifier

Parameters:

qualifier - The qualifier to parse

Returns:

A short representing the last 4 bits of the qualifier

Throws:

IllegalArgument - if the qualifier is null or empty

Since:

2.0

getFlagsFromQualifier

public static short getFlagsFromQualifier(byte[] qualifier,
                          int offset)

Parses the flag bits from the qualifier

Parameters:

qualifier - The qualifier to parse

offset - An offset within the byte array

Returns:

A short representing the last 4 bits of the qualifier

Throws:

IllegalArgument - if the qualifier is null or the offset falls outside of the qualifier array

Since:

2.0

extractQualifier

public static byte[] extractQualifier(byte[] qualifier,
                      int offset)

Extracts the 2 or 4 byte qualifier from a compacted byte array

Parameters:

qualifier - The qualifier to parse

offset - An offset within the byte array

Returns:

A byte array with only the requested qualifier

Throws:

IllegalArgument - if the qualifier is null or the offset falls outside of the qualifier array

Since:

2.0

buildQualifier

public static byte[] buildQualifier(long timestamp,
                    short flags)

Returns a 2 or 4 byte qualifier based on the timestamp and the flags. If the timestamp is in seconds, this returns a 2 byte qualifier. If it's in milliseconds, returns a 4 byte qualifier

Parameters:

timestamp - A Unix epoch timestamp in seconds or milliseconds

flags - Flags to set on the qualifier (length &| float)

Returns:

A 2 or 4 byte qualifier for storage in column or compacted column

Since:

2.0