LibraryToggle FramesPrintFeedback

The hdfs component enables you to read and write messages from/to an HDFS file system. HDFS is the distributed file system at the heart of Hadoop. It can only be built using JDK1.6 or later because this is a strict requirement for Hadoop itself. This component is hosted at http://github.com/dgreco/camel-hdfs. We decided to put it temporarily on this github because currently Camel is being built and tested using JDK1.5 and for this reason we couldn't put that component into the Camel official distribution. Hopefully, as soon Camel will allow to use JDK1.6 for building and testing we will put it into the trunk. This component is developed and tested using the latest Camel snapshot, but it should work seamlessly with the latest Camel GA version (at the time of writing 2.1.0)

Name Default Value Description
overwrite true The file can be overwritten
bufferSize 4096 The buffer size used by HDFS
replication 3 The HDFS replication factor
blockSize 67108864 The size of the HDFS blocks
fileType NORMAL_FILE

It can be SEQUENCE_FILE, MAP_FILE, ARRAY_FILE, or BLOOMMAP_FILE, see Hadoop

fileSystemType HDFS It can be LOCAL for local filesystem
keyType NULL

The type for the key in case of sequence or map files. See below.

valueType TEXT

The type for the key in case of sequence or map files. See below.

splitStrategy

A string describing the strategy on how to split the file based on different criteria. See below.

openedSuffix opened

When a file is opened for reading/ writing the file is renamed with this suffix to avoid to read it during the writing phase.

readSuffix read

Once the file has been read is renamed with this suffix to avoid to read it again.

initialDelay 0

For the consumer, how much to wait (milliseconds) before to start scanning the directory.

delay 0

The interval (milliseconds) between the directory scans.

pattern *

The pattern used for scanning the directory

chunkSize 4096

When reading a normal file, this is split into chunks producing a message per chunk

Comments powered by Disqus