[FEATURE REQUEST] Hot Reload for Ray Serve

[movchan74]

Feature Summary

Right now, if we make any changes to the code, the whole SDK needs to be restarted. This is not ideal because it takes a lot of time to restart the SDK especially if we have a lot of models and some of them take a long time to load (e.g. BLIP2). Ray isolates the models into deployments so it should be possible to reload only the deployment that was changed.

As a subtask of this issue, we also should be able to automatically reload the pipeline when it's changed.

Justification/Rationale

This will improve the developer experience by reducing the time it takes to test changes to the code.

Proposed Implementation (if any)

I've tried to do something like this already by watching the files for changes and reloading the deployment when a change is detected (basically killing it and Ray restarts it automatically). However, this doesn't work because the code is not reloaded, it still uses old code. I think this is because the code is loaded when the deployment is created and it's not possible to reload it just by restarting the deployment.

Another approach would be to serve build. I've tried to do this as well but it seems to have similar issue with reloading the code.