Regarding destination file containing bytewritables
1 Answer(s)
Abhijit-Dezyre Support
Hi Venkatesh,
Please share the flume.conf with us to better understand the issue.
Here is the sample for flume.conf for local to hdfs:
-----------------------------------------------------------------
#Flume.conf
agent1.sources=src-1
agent1.channels=memory-channel
agent1.sinks=hdfs-sink
#describe and configure the sources
agent1.sources.src-1.type = spooldir
agent1.sources.src-1.spoolDir = /home/cloudera/flumeSpool
agent1.sources.src-1.fileHeader = true
agent1.sources.src-1.channels=memory-channel
#describe the channel
agent1.channels.memory-channel.type=memory
agent1.channels.memory-channel.capacity=10000
agent1.channels.memory-channel.transactionCapacity=100
#describe the sink properties
agent1.sinks.hdfs-sink.type=hdfs
agent1.sinks.hdfs-sink.channel=memory-channel
agent1.sinks.hdfs-sink.hdfs.path=/user/cloudera/flumeSpool
agent1.sinks.hdfs-sink.hdfs.fileType=DataStream
agent1.sinks.hdfs-sink.hdfs.writeFormat=Text
agent1.sinks.hdfs-sink.hdfs.rollCount=20
---------------------------------------------------------------------
Hope this helps
Thanks
Feb 21 2016 02:25 AM
Please share the flume.conf with us to better understand the issue.
Here is the sample for flume.conf for local to hdfs:
-----------------------------------------------------------------
#Flume.conf
agent1.sources=src-1
agent1.channels=memory-channel
agent1.sinks=hdfs-sink
#describe and configure the sources
agent1.sources.src-1.type = spooldir
agent1.sources.src-1.spoolDir = /home/cloudera/flumeSpool
agent1.sources.src-1.fileHeader = true
agent1.sources.src-1.channels=memory-channel
#describe the channel
agent1.channels.memory-channel.type=memory
agent1.channels.memory-channel.capacity=10000
agent1.channels.memory-channel.transactionCapacity=100
#describe the sink properties
agent1.sinks.hdfs-sink.type=hdfs
agent1.sinks.hdfs-sink.channel=memory-channel
agent1.sinks.hdfs-sink.hdfs.path=/user/cloudera/flumeSpool
agent1.sinks.hdfs-sink.hdfs.fileType=DataStream
agent1.sinks.hdfs-sink.hdfs.writeFormat=Text
agent1.sinks.hdfs-sink.hdfs.rollCount=20
---------------------------------------------------------------------
Hope this helps
Thanks