org.apache.hadoop.hive.common.io

Interface DataCache

public interface DataCache

An abstract data cache that IO formats can use to retrieve and cache data.

Nested Class Summary

Nested Classes

Modifier and Type	Interface and Description
static class	DataCache.BooleanRef
static interface	DataCache.DiskRangeListFactory Disk range factory used during cache retrieval.

Method Summary

Modifier and Type	Method and Description
Allocator	getAllocator() Gets the allocator associated with this DataCache.
DiskRangeList	getFileData(Object fileKey, DiskRangeList range, long baseOffset, DataCache.DiskRangeListFactory factory, DataCache.BooleanRef gotAllData) Gets file data for particular offsets.
long[]	putFileData(Object fileKey, DiskRange[] ranges, MemoryBuffer[] data, long baseOffset) Puts file data into cache, or gets older data in case of collisions.
void	releaseBuffer(MemoryBuffer buffer) Releases the buffer returned by getFileData/provided to putFileData back to cache.
void	reuseBuffer(MemoryBuffer buffer) Notifies the cache that the buffer returned from getFileData/provided to putFileData will be used by another consumer and therefore released multiple times (one more time per call).

Method Detail

getFileData

DiskRangeList getFileData(Object fileKey,
                          DiskRangeList range,
                          long baseOffset,
                          DataCache.DiskRangeListFactory factory,
                          DataCache.BooleanRef gotAllData)

Gets file data for particular offsets. The range list is modified in place; it is then returned (since the list head could have changed). Ranges are replaced with cached ranges. Any such buffer is locked in cache to prevent eviction, and must therefore be released back to cache via a corresponding call (releaseBuffer) when the caller is done with it. In case of partial overlap with cached data, full cache blocks are always returned; there's no capacity for partial matches in return type. The rules are as follows: 1) If the requested range starts in the middle of a cached range, that cached range will not be returned by default (e.g. if [100,200) and [200,300) are cached, the request for [150,300) will only return [200,300) from cache). This may be configurable in impls. This is because we assume well-known range start offsets are used (rg/stripe offsets), so a request from the middle of the start doesn't make sense. 2) If the requested range ends in the middle of a cached range, that entire cached range will be returned (e.g. if [100,200) and [200,300) are cached, the request for [100,250) will return both ranges). It should really be same as #1, however currently ORC uses estimated end offsets; if we don't return the end block, the caller may read it from disk needlessly.

Parameters:

fileKey - Unique ID of the target file on the file system.

range - A set of DiskRange-s (linked list) that is to be retrieved. May be modified.

baseOffset - base offset for the ranges (stripe/stream offset in case of ORC).

factory - A factory to produce DiskRangeList-s out of cached MemoryBuffer-s.

gotAllData - An out param - whether all the requested data was found in cache.

Returns:

The new or modified list of DiskRange-s, where some ranges may contain cached data.

putFileData

long[] putFileData(Object fileKey,
                   DiskRange[] ranges,
                   MemoryBuffer[] data,
                   long baseOffset)

Puts file data into cache, or gets older data in case of collisions. The memory buffers provided MUST be allocated via an allocator returned by getAllocator method, to allow cache implementations that evict and then de-allocate the buffer. It is assumed that the caller will use the data immediately, therefore any buffers provided to putFileData (or returned due to cache collision) are locked in cache to prevent eviction, and must therefore be released back to cache via a corresponding call (releaseBuffer) when the caller is done with it. Buffers rejected due to conflict will neither be locked, nor automatically deallocated. The caller must take care to discard these buffers.

Parameters:

fileKey - Unique ID of the target file on the file system.

ranges - The ranges for which the data is being cached. These objects will not be stored.

data - The data for the corresponding ranges.

baseOffset - base offset for the ranges (stripe/stream offset in case of ORC).

Returns:

null if all data was put; bitmask indicating which chunks were not put otherwise; the replacement chunks from cache are updated directly in the array.

releaseBuffer

void releaseBuffer(MemoryBuffer buffer)

Releases the buffer returned by getFileData/provided to putFileData back to cache. See respective javadocs for details.

reuseBuffer

void reuseBuffer(MemoryBuffer buffer)

Notifies the cache that the buffer returned from getFileData/provided to putFileData will be used by another consumer and therefore released multiple times (one more time per call).

getAllocator

Allocator getAllocator()

Gets the allocator associated with this DataCache.