refactor: refine writer interface to support directory hierarchy

[ZENOTME]

This PR is intended to resolve #891. On our original design, we allow the user to create the writer builder and combine them and build them finally. However, original design is not support to change the config of builder after we combine them. But in some case, we hope to recreate the writer for different partition using writer builder. E.g. in #819, we want to regenerate the table location for different partition. So I follow the design from iceberg-java https://github.com/apache/iceberg/blob/d96901b843395fe669f6bd4f618f8e5e46c0eed4/core/src/main/java/org/apache/iceberg/io/BaseTaskWriter.java#L334 :

1. separate the OutputFile from FileWriter.
2. create a new abstract SinglePartitionWriterBuilder.
3. introduce the OutputFileGenerator following https://github.com/apache/iceberg/blob/d96901b843395fe669f6bd4f618f8e5e46c0eed4/core/src/main/java/org/apache/iceberg/io/OutputFileFactory.java#L36

It's the responsibility of SinglePartitionWriterBuilder to create the OutputFile. And the build function of SinglePartitionWriterBuilder take a partition value which means that we can create different partition of writer from this Builder.

[hackintoshrao]

Thanks, @ZENOTME for the PR. I'm looking into it. Does the current approach handle multiple partition columns so that we can write into a multi-level directory structure? Or is it currently limited to just one partition column?

[ZENOTME]

Thanks, @ZENOTME for the PR. I'm looking into it. Does the current approach handle multiple partition columns so that we can write into a multi-level directory structure? Or is it currently limited to just one partition column?

It can handle multiple-level, e.g we have partition <col1=1,col2=2>, it will generate location table/data/col1=1/col=2. See:

iceberg-rust/crates/iceberg/src/writer/output_file_generator.rs

Line 144 in 094655a

fn partition_dir_path(partition: &Struct, partition_type: &StructType) -> Result<String> {

. And it supports to custom the LocationGenerator