Hive Configuration

Table properties

Tables stored as ORC files use table properties to control their behavior. By using table properties, the table owner ensures that all clients store data with the same options.

Key Default Notes
orc.compress ZLIB high level compression = {NONE, ZLIB, SNAPPY}
orc.compress.size 262,144 compression chunk size
orc.stripe.size 67,108,864 memory buffer in bytes for writing
orc.row.index.stride 10,000 number of rows between index entries
orc.create.index true whether the ORC writer create indexes as part of the file or not
orc.bloom.filter.columns ”” comma separated list of column names
orc.bloom.filter.fpp 0.05 bloom filter false positive rate

For example, to create an ORC table without high level compression:

  name STRING,
  color STRING

Configuration properties

There are many Hive configuration properties related to ORC files:

Key Default Notes
hive.default.fileformat TextFile This is the default file format for new tables. If it is set to ORC, new tables will default to ORC.
hive.stats.gather.num.threads 10 Number of threads used by partialscan/noscan analyze command for partitioned tables. This is applicable only for file formats that implement the StatsProvidingRecordReader interface (like ORC).
hive.exec.orc.memory.pool 0.5 Maximum fraction of heap that can be used by ORC file writers.
hive.exec.orc.write.format NULL Define the version of the file to write. Possible values are 0.11 and 0.12. If this parameter is not defined, ORC will use the latest version.
hive.exec.orc.default.stripe.size 67,108,864 Define the default size of ORC writer buffers in bytes.
hive.exec.orc.default.block.size 268,435,456 Define the default file system block size for ORC files.
hive.exec.orc.dictionary.key.size.threshold 0.8 If the number of keys in a dictionary is greater than this fraction of the total number of non-null rows, turn off dictionary encoding. Use 1.0 to always use dictionary encoding.
hive.exec.orc.default.row.index.stride 10,000 Define the default number of rows between row index entries.
hive.exec.orc.default.buffer.size 262,144 Define the default ORC buffer size, in bytes.
hive.exec.orc.default.block.padding true Should ORC file writers pad stripes to minimize stripes that cross HDFS block boundaries.
hive.exec.orc.block.padding.tolerance 0.05 Define the tolerance for block padding as a decimal fraction of stripe size (for example, the default value 0.05 is 5% of the stripe size). For the defaults of 64Mb ORC stripe and 256Mb HDFS blocks, a maximum of 3.2Mb will be reserved for padding within the 256Mb block with the default hive.exec.orc.block.padding.tolerance. In that case, if the available size within the block is more than 3.2Mb, a new smaller stripe will be inserted to fit within that space. This will make sure that no stripe written will cross block boundaries and cause remote reads within a node local task.
hive.exec.orc.default.compress ZLIB Define the default compression codec for ORC file.
hive.exec.orc.encoding.strategy SPEED Define the encoding strategy to use while writing data. Changing this will only affect the light weight encoding for integers. This flag will not change the compression level of higher level compression codec (like ZLIB). Possible options are SPEED and COMPRESSION.
hive.orc.splits.include.file.footer false If turned on, splits generated by ORC will include metadata about the stripes in the file. This data is read remotely (from the client or HiveServer2 machine) and sent to all the tasks.
hive.orc.cache.stripe.details.size 10,000 Cache size for keeping meta information about ORC splits cached in the client.
hive.orc.compute.splits.num.threads 10 How many threads ORC should use to create splits in parallel. false If ORC reader encounters corrupt data, this value will be used to determine whether to skip the corrupt data or throw an exception. The default behavior is to throw an exception.
hive.exec.orc.zerocopy false Use zerocopy reads with ORC. (This requires Hadoop 2.3 or later.)
hive.merge.orcfile.stripe.level true When hive.merge.mapfiles, hive.merge.mapredfiles or hive.merge.tezfiles is enabled while writing a table with ORC file format, enabling this configuration property will do stripe-level fast merge for small ORC files. Note that enabling this configuration property will not honor the padding tolerance configuration (hive.exec.orc.block.padding.tolerance).
hive.orc.row.index.stride.dictionary.check true If enabled dictionary check will happen after first row index stride (default 10000 rows) else dictionary check will happen before writing first stripe. In both cases, the decision to use dictionary or not will be retained thereafter.
hive.exec.orc.compression.strategy SPEED Define the compression strategy to use while writing data. This changes the compression level of higher level compression codec. Value can be SPEED or COMPRESSION.
orc.write.variable.length.blocks false Should the ORC writer use HDFS variable length blocks, if they are available? If the new stripe would straddle a block, Hadoop is ≥ 2.7, and this is enabled, it will end the block before the new stripe.