Add functionality to correctly load old metagraphs

[oadams]

There occasionally are some problems loading old stored models. I've figured out how to resolve these issues (working on branch SavedModel) but need to integrate this functionality into model loading. The user should not need to specify the number of layers or hidden units as this is all stored in the metagraph.

There are related design issues here too. For example, it looks like adding variables when creating an rnn_ctc model puts them in a global graph, undermining the namespacing attempt behind using a class in the first place. This isn't a huge deal, as the user is only really going to be using one model at a time, but it seems like a way bugs can creep in so that should be changed.

[shuttle1987]

I'm a little concerned about this:

This isn't a huge deal, as the user is only really going to be using one model at a time, but it seems like a way bugs can creep in so that should be changed.

In the context of the Web API as more than one model is likely to be available for transcribing at once.

[oadams]

In the context of defining new models, or decoding from existing models this could be an issue. The solution is probably to use Tensorflow variable namespacing/scoping.

(Note that In the context of decoding from saved models, this won't be an issue as in the SavedModel branch, the decode() function just reads from a stored metagraph:

persephone/persephone/model.py

Lines 48 to 81 in e616919

	def decode(model_path_prefix: Union[str, Path],
	input_paths: Sequence[Path],
	label_set: Set[str]) -> List[List[str]]:
	model_path_prefix = str(model_path_prefix)
	# TODO Confirm that that WAVs exist.
	# TODO Confirm that the feature files exist. Create them if they don't.
	# TODO Change the second argument to have some upper bound. If the caller
	# requests 1000 WAVs be transcribed, they shouldn't all go in one batch.
	fn_batches = utils.make_batches(input_paths, len(input_paths))
	# Load the model and perform decoding.
	metagraph = load_metagraph(model_path_prefix)
	with tf.Session() as sess:
	metagraph.restore(sess, model_path_prefix)
	for fn_batch in fn_batches:
	batch_x, batch_x_lens = utils.load_batch_x(fn_batch)
	# TODO These placeholder names should be a backup if names from a newer
	# naming scheme aren't present. Otherwise this won't generalize to
	# different architectures.
	feed_dict = {"Placeholder:0": batch_x,
	"Placeholder_1:0": batch_x_lens}
	dense_decoded = sess.run("SparseToDense:0", feed_dict=feed_dict)
	# Create a human-readable representation of the decoded.
	indices_to_labels = labels.make_indices_to_labels(label_set)
	human_readable = dense_to_human_readable(dense_decoded, indices_to_labels)
	return human_readable

)

[shuttle1987]

I'll have a look into using the Tensorflow variable scoping.

[shuttle1987]

Will #197 close this issue?

[oadams]

It will. But only for using saved models to decode; not so that training can be resumed (that can be addressed when we use keras down the line).

[shuttle1987]

What exactly is stopping training from being resumed?

[oadams]

Based on your other emails I assume you figured this out, but let me know if you want clarification.

[shuttle1987]

I think this issue is closed, please reopen if I closed in error

[oadams]

Happy to leave this closed, but a note to the reader: while we can load metagraphs for decoding, training can't be resumed.

[shuttle1987]

I think #180 covers that issue about resuming training.