Prepack weights when model is loaded

[robertknight]

Other ML runtimes such as ONNX Runtime and TensorFlow Lite prepack weights for use with the selected matrix multiplication kernel when a model is loaded. This reduces inference latency when a model is run multiple times in a session, at the cost of longer load time.

RTen implements weight prepacking to amortize packing overhead when MatMul or Conv operators are applied to a batch of inputs. However it doesn't prepack inputs when the model is loaded, so packing costs are incurred for each inference.

References:

• https://pytorch.org/docs/stable/mobile_optimizer.html
• https://blog.tensorflow.org/2022/06/memory-efficient-inference-with-xnnpack.html