Simplify deploy / webapp-public interaction

[PeterDSteinberg]

We have had bugs related to the interaction of this repo and the private deploy repo in API functions, such as passing jsons between the front end Django app (now webapp-public) and the back end services (celery - celery_tasks.py, flask - flask_server.py). Goals of this issue are to plan:

• Move most of deploy assets to this repo at a top level subfolder called deploy: python files mostly to subpackage, bash scripts case by case review based on how they are used.
• Change 2 existing API styles (file based reforms and UI reforms) in webapp / deploy have one API, so that file reforms or UI reforms go to same end point. Try to de-deduplicate code.
• Improve backwards compatibility of models in one API refactor
• Make a mocked system test harness so that webapp-public can prepare arguments and jsons as though it is going to submit them to the flask / celery system of deploy but instead calls Python function

Make one PR soon that moves the deploy repo to webapp-public and possibly some minor other changes while trying not affect existing usage of webapp. Then do >=1 PRs to clean up and address the bullets above.

Thoughts? @MattHJensen @brittainhard

[MattHJensen]

+1

[martinholmer]

What is the status of issue #543? Haven't some of the pull request merges of the past few weeks made progress on items listed in the first comment in #543?

@brittainhard @PeterDSteinberg @jbcrail @MattHJensen

[PeterDSteinberg]

Hi @MattHJensen - Regarding the last 2 checkboxes above:

• Has backwards compatibility been okay over the last month from your perspective? Any models you have run in the last month or 6 weeks that are no longer displaying results correctly? Let's discuss backwards compatibility again when we plan the OG-USA work, such as the meeting on Friday of this week. Backwards compatibility is less challenged by changes to the input forms to models, but when we do major changes to the results pages, such as addition of new tables or figures or interactivity, we should make backwards compatibility a check list item for the PR(s).
• Bullet 4 Make a mocked system test harness so that webapp-public can prepare arguments and jsons as though it is going to submit them to the flask / celery system of deploy but instead calls Python function is not addressed yet. I'll make a separate issue for it - I think it will make the tests more exhaustive to have Django CI tests or local Django usage be able to dump a json to file or one's browser without calling out to flask.

[PeterDSteinberg]

#580 covers the final bullet point.