Filesystem corruption shouldn't be something that a software engineer should have to account for, that's a lower level thing that would need to be handled by the underlying filesystem. HAMMER, ZFS, and BTRFS are the only filesystems I know of that actively checksum at the moment. BTRFS doesn't have very good performance, ZFS isn't natively supported on GNU/Linux and usage through FUSE will not yield the best performance either, and HAMMER, although a fast and powerful filesystem, is only available for DragonFlyBSD. I don't know about Hadoop support outside of GNU/Linux.

hi Zachary,

Please note that Hadoop uses the native file system (ext3, ext4, journal, etc) to store and retrieve files. Hadoop doesn't do checksums on the data stored. All data blocks are stored in various datanodes and there is a software called "block" software which runs on all datanodes which will keep checking if the data blocks are in good state or corrupted state.

All datanodes will send this block report to NameNode which knows which block of which file is corrupted. Lets say that a 3rd block of file abc.txt is corrupted , then the NameNode will send a request to one of the datanodes to replace the corrupted block or when a client sends a request to read the data of abc.txt, the no-corrupted block is used from the replicated copy.

Thanks