Q&A
Ask and answer questions to make information more available to wider audiences.
Carson Kydons
@carsonkydons 07, Jun 2023 12:00 AM
too many small files in a cluster
What would happen if I store too many small files in a cluster on HDFS?
answers 1
Answer 1
Jose Grimsbro
@josegrimsbro 12, Jun 2023 07:37 PM
Storing several small files on HDFS generates a lot of metadata files. To store these metadata in the RAM is a challenge as each file, block, or directory takes 150 bytes for metadata. Thus, the cumulative size of all the metadata will be too large.