Package org.apache.orc.impl

Class PhysicalFsWriter

java.lang.Object

org.apache.orc.impl.PhysicalFsWriter

All Implemented Interfaces:

PhysicalWriter

public class PhysicalFsWriter
extends Object
implements PhysicalWriter

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
protected static class	PhysicalFsWriter.VariantTracker	Record the information about each column encryption variant.

Nested classes/interfaces inherited from interface org.apache.orc.PhysicalWriter

PhysicalWriter.OutputReceiver

Constructor Summary

Constructors

Constructor	Description
PhysicalFsWriter(FileSystem fs, Path path, OrcFile.WriterOptions opts)
PhysicalFsWriter(FileSystem fs, Path path, OrcFile.WriterOptions opts, WriterEncryptionVariant[] encryption)
PhysicalFsWriter(FSDataOutputStream outputStream, OrcFile.WriterOptions opts, WriterEncryptionVariant[] encryption)

Method Summary

Modifier and Type	Method	Description
void	appendRawStripe(ByteBuffer buffer, OrcProto.StripeInformation.Builder dirEntry)	Appends raw stripe data (e.g.
void	close()	Closes the writer.
org.apache.orc.impl.PhysicalFsWriter.BufferedStream	createDataStream(StreamName name)	Create an OutputReceiver for the given name.
protected OutputStream	createIndexStream(StreamName name)
void	finalizeStripe(OrcProto.StripeFooter.Builder footerBuilder, OrcProto.StripeInformation.Builder dirEntry)	Flushes the data in all the streams, spills them to disk, write out stripe footer.
void	flush()	Flushes the writer so that readers can see the preceding postscripts.
long	getFileBytes(int column, WriterEncryptionVariant variant)	Get the number of bytes for a file in a given column by finding all the streams (not suppressed) for a given column and returning the sum of their sizes.
StreamOptions	getStreamOptions()	Get the unencrypted stream options for this file.
String	toString()
void	writeBloomFilter(StreamName name, OrcProto.BloomFilterIndex.Builder bloom)	Write a bloom filter index in the given stream name.
void	writeFileFooter(OrcProto.Footer.Builder builder)	Writes out the file footer.
void	writeFileMetadata(OrcProto.Metadata.Builder builder)	Writes out the file metadata.
void	writeHeader()	Writes the header of the file, which consists of the magic "ORC" bytes.
void	writeIndex(StreamName name, OrcProto.RowIndex.Builder index)	Write an index in the given stream name.
long	writePostScript(OrcProto.PostScript.Builder builder)	Writes out the postscript (including the size byte if needed).
void	writeStatistics(StreamName name, OrcProto.ColumnStatistics.Builder statistics)	Write a stripe or file statistics to the file.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Details

PhysicalFsWriter

public PhysicalFsWriter(FileSystem fs,
 Path path,
 OrcFile.WriterOptions opts)
                 throws IOException

Throws:

IOException

PhysicalFsWriter

public PhysicalFsWriter(FileSystem fs,
 Path path,
 OrcFile.WriterOptions opts,
 WriterEncryptionVariant[] encryption)
                 throws IOException

Throws:

IOException

PhysicalFsWriter

public PhysicalFsWriter(FSDataOutputStream outputStream,
 OrcFile.WriterOptions opts,
 WriterEncryptionVariant[] encryption)
                 throws IOException

Throws:

IOException

Method Details

getFileBytes

public long getFileBytes(int column,
 WriterEncryptionVariant variant)

Get the number of bytes for a file in a given column by finding all the streams (not suppressed) for a given column and returning the sum of their sizes. excludes index

Specified by:

getFileBytes in interface PhysicalWriter

Parameters:

column - column from which to get file size

variant - the encryption variant to check

Returns:

number of bytes for the given column

getStreamOptions

public StreamOptions getStreamOptions()

Description copied from interface: PhysicalWriter

Get the unencrypted stream options for this file. This class needs the stream options to write the indexes and footers. Additionally, the LLAP CacheWriter wants to disable the generic compression.

Specified by:

getStreamOptions in interface PhysicalWriter

writeFileMetadata

public void writeFileMetadata(OrcProto.Metadata.Builder builder)
                       throws IOException

Description copied from interface: PhysicalWriter

Writes out the file metadata.

Specified by:

writeFileMetadata in interface PhysicalWriter

Parameters:

builder - Metadata builder to finalize and write.

Throws:

IOException

writeFileFooter

public void writeFileFooter(OrcProto.Footer.Builder builder)
                     throws IOException

Description copied from interface: PhysicalWriter

Writes out the file footer.

Specified by:

writeFileFooter in interface PhysicalWriter

Parameters:

builder - Footer builder to finalize and write.

Throws:

IOException

writePostScript

public long writePostScript(OrcProto.PostScript.Builder builder)
                     throws IOException

Description copied from interface: PhysicalWriter

Writes out the postscript (including the size byte if needed).

Specified by:

writePostScript in interface PhysicalWriter

Parameters:

builder - Postscript builder to finalize and write.

Throws:

IOException

close

public void close()
           throws IOException

Description copied from interface: PhysicalWriter

Closes the writer.

Specified by:

close in interface PhysicalWriter

Throws:

IOException

flush

public void flush()
           throws IOException

Description copied from interface: PhysicalWriter

Flushes the writer so that readers can see the preceding postscripts.

Specified by:

flush in interface PhysicalWriter

Throws:

IOException

appendRawStripe

public void appendRawStripe(ByteBuffer buffer,
 OrcProto.StripeInformation.Builder dirEntry)
                     throws IOException

Description copied from interface: PhysicalWriter

Appends raw stripe data (e.g. for file merger).

Specified by:

appendRawStripe in interface PhysicalWriter

Parameters:

buffer - Stripe data buffer.

dirEntry - File metadata entry for the stripe, to be updated with relevant data.

Throws:

IOException

finalizeStripe

public void finalizeStripe(OrcProto.StripeFooter.Builder footerBuilder,
 OrcProto.StripeInformation.Builder dirEntry)
                    throws IOException

Description copied from interface: PhysicalWriter

Flushes the data in all the streams, spills them to disk, write out stripe footer.

Specified by:

finalizeStripe in interface PhysicalWriter

Parameters:

footerBuilder - Stripe footer to be updated with relevant data and written out.

dirEntry - File metadata entry for the stripe, to be updated with relevant data.

Throws:

IOException

writeHeader

public void writeHeader()
                 throws IOException

Description copied from interface: PhysicalWriter

Writes the header of the file, which consists of the magic "ORC" bytes.

Specified by:

writeHeader in interface PhysicalWriter

Throws:

IOException

createDataStream

public org.apache.orc.impl.PhysicalFsWriter.BufferedStream createDataStream(StreamName name)

Description copied from interface: PhysicalWriter

Create an OutputReceiver for the given name.

Specified by:

createDataStream in interface PhysicalWriter

Parameters:

name - the name of the stream

createIndexStream

protected OutputStream createIndexStream(StreamName name)

writeIndex

public void writeIndex(StreamName name,
 OrcProto.RowIndex.Builder index)
                throws IOException

Description copied from interface: PhysicalWriter

Write an index in the given stream name.

Specified by:

writeIndex in interface PhysicalWriter

Parameters:

name - the name of the stream

index - the bloom filter to write

Throws:

IOException

writeBloomFilter

public void writeBloomFilter(StreamName name,
 OrcProto.BloomFilterIndex.Builder bloom)
                      throws IOException

Description copied from interface: PhysicalWriter

Write a bloom filter index in the given stream name.

Specified by:

writeBloomFilter in interface PhysicalWriter

Parameters:

name - the name of the stream

bloom - the bloom filter to write

Throws:

IOException

writeStatistics

public void writeStatistics(StreamName name,
 OrcProto.ColumnStatistics.Builder statistics)

Description copied from interface: PhysicalWriter

Write a stripe or file statistics to the file.

Specified by:

writeStatistics in interface PhysicalWriter

Parameters:

name - the name of the stream

statistics - the statistics to write

toString

public String toString()

Overrides:

toString in class Object