Clean up metadata handling

[natemcintosh]

Goal

Have the various nested components pass metadata up out of them, and have orchestrate_pipeline() handle saving the metadata.

Context

Metadata is currently saved by write_outputs(), which means passing information down into it and model_logic() about blob storage, which they don't need to know about.

Requirements

Ideally, the metadata would be returned from execute_model_logic() as a value, and orchestrate_pipeline() would be the function that writes the values to disk. Starting at the most nested function first:

• write_model_outputs() does not write the metadata. Alter it to return the samples_path, summaries_path, and model_path it used when writing the samples, summaries, and model.
  • Alter the tests that check the return value and metadata saving.
• execute_model_logic() accepts the returned paths from write_model_outputs(), and adds them to the metadata list. It then returns the metadata list. Also add model completion status to the metadata?
  • Alter the tests that check the return value.
• orchestrate_pipeline() accepts the metadata list from execute_model_logic(). It adds the blob storage container to the metadata. It writes the metadata to file, using the dirname() of the model path.
  • Alter tests that check the output of orchestrate_pipeline(). Add tests for checking that metadata has been saved to disk.

Out of scope

• Anything that isn't about setting up better handling of the metadata.

Related

• Make metadata easier to use. #121