You can use SLF4J for logging in your App. Logging framework is not going to change as the program is in Java. If you want to capture dropped records/duplicate records , use the logging framework along with Counters() methods.

Thanks

But given the volumes of big data and various data processing steps, logs may be huge and I might wanna do logging in HDFS than on my local file system. SLF4J doesn't log into HDFS. This is especially useful if I am using Hadoop to process such logged data for further analysis. Is there any logging framework to write to HDFS directly?