Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Integrate Python Language Server to Monaco Editor (Closes #1908, #571) #1910

Open
wants to merge 43 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 9 commits
Commits
Show all changes
43 commits
Select commit Hold shift + click to select a range
542f549
WIP #1908: Add working example with basic pyls
umesh-timalsina Sep 17, 2020
7aeebe0
Provide Dockerfile and build for a separate language servers service
umesh-timalsina Sep 21, 2020
da381fb
WIP- Update deployment of language servers and minor eslint fixes
umesh-timalsina Sep 23, 2020
2802a5d
Use microsoft's python language server in favor of pyls(palantir)
umesh-timalsina Sep 25, 2020
4503ae4
WIP- Fix deployment issue with language servers
umesh-timalsina Sep 25, 2020
29775a7
WIP- Use components.json for languageserver configuration
umesh-timalsina Sep 30, 2020
b777e43
Merge remote-tracking branch 'origin/master' into 1908-pyls-integration
umesh-timalsina Sep 30, 2020
08291c5
WIP- Fix languageServers.json file; jq deployment; uninstall jsonrpc-…
umesh-timalsina Oct 1, 2020
dd9fe28
Merge remote-tracking branch 'origin/master' into 1908-pyls-integration
umesh-timalsina Oct 1, 2020
37c1ddc
WIP- Fix typo in config.extensions, rename token to tokens in languag…
umesh-timalsina Oct 2, 2020
03b41f5
WIP- Fix jsonrpc-ws-proxy call in Dockerfile.langservers entrypoint
umesh-timalsina Oct 2, 2020
c0fc77f
WIP-Fix jsonrpc-ws-proxy in langserver-entrypoint.sh
umesh-timalsina Oct 2, 2020
376fbc6
WIP_ Remove python-language-server from requirements
umesh-timalsina Oct 2, 2020
dcd5ae5
WIP- Fix typo in DeepforgeLanguageClient
umesh-timalsina Oct 2, 2020
e26e7c3
WIP- Use python3.7 anaconda in docker image
umesh-timalsina Oct 7, 2020
f4facd8
WIP- Move languageServers.json to .deployment
umesh-timalsina Oct 7, 2020
5ca1db3
WIP- Use inmemory models instead of file models
umesh-timalsina Oct 7, 2020
9e2525a
Resolve merge conflicts and merge branch master
umesh-timalsina Oct 12, 2020
8bd0f3e
Merge remote-tracking branch 'origin/1908-pyls-integration' into 1908…
umesh-timalsina Oct 12, 2020
ddbf23a
Update README for deployment
umesh-timalsina Oct 12, 2020
b1df3f6
WIP- Use componentSettings for LanguageServers; Docker build for lang…
umesh-timalsina Oct 22, 2020
2bd224e
Merge remote-tracking branch 'origin/master' into 1908-pyls-integration
umesh-timalsina Oct 28, 2020
21ad734
WIP- Test docker build for language servers
umesh-timalsina Oct 28, 2020
c8227d1
WIP- Fix Dockerfile.langservers path
umesh-timalsina Oct 28, 2020
a339b9c
WIP- Add npm install step in github workflows
umesh-timalsina Oct 28, 2020
087323b
WIP- Don't update conda in Dockerfile
umesh-timalsina Oct 28, 2020
aa84bc2
WIP- Use latest anaconda docker image
umesh-timalsina Oct 28, 2020
a9e321a
Merge remote-tracking branch 'origin/master' into 1908-pyls-integration
umesh-timalsina Nov 11, 2020
10bec66
WIP- Use anaconda3:2019.10 as image base
umesh-timalsina Nov 11, 2020
89b2e68
Revert to pip installing packages
umesh-timalsina Nov 12, 2020
20bd21f
WIP- Fix Keras version in dockerfile
umesh-timalsina Nov 12, 2020
193698d
Merge remote-tracking branch 'origin/master' into 1908-pyls-integration
umesh-timalsina Dec 16, 2020
d7b9a0d
WIP- Minor fixes to the language server settings
umesh-timalsina Dec 16, 2020
aa60904
Merge remote-tracking branch 'origin/master' into 1908-pyls-integration
umesh-timalsina Jan 20, 2021
61bd70f
WIP- Deploy Branch for Testing
umesh-timalsina Jan 20, 2021
f75f652
WIP- Publish KitchenSink image for testing
umesh-timalsina Jan 20, 2021
c45823b
WIP- Publish kitchen sink image for testing
umesh-timalsina Jan 20, 2021
3d84cdb
WIP- Add Push Tags
umesh-timalsina Jan 20, 2021
d5bfa3f
WIP- Fix Tags for kitchensink image
umesh-timalsina Jan 20, 2021
ff110bc
WIP- Fix docker-compose for testing
umesh-timalsina Jan 20, 2021
36c6723
WIP- Checkout branch in deployment script
umesh-timalsina Jan 20, 2021
81657c3
WIP- Fix branch checkout in workflow
umesh-timalsina Jan 20, 2021
71979a3
WIP- Fix entrypoint and setup debugging.
umesh-timalsina Jan 21, 2021
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 6 additions & 2 deletions .deployment/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,13 @@ The script `deploy-deepforge` is used for standard deployment of deepforge using

Additionally, this contains a file with customizations to the standard docker-compose.yml file which allows us to modify the entrypoint and install a version of tensorflow [compatible with the CPU of the deployment machine](https://github.com/deepforge-dev/deepforge/issues/1561).

The deployment is updated by first creating the custom docker compose file using [yaml-merge](https://github.com/alexlafroscia/yaml-merge):
Moreover, we also a proxy server with that spins up different language servers that we provide for
intelligent syntax
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks like some odd line breaking...

highlighting in deepforge's browser. For more information checkout the language server's docker [file](../docker/Dockerfile.langservers)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"in deepforge's browser" -> "in DeepForge"


The deployment is updated by first creating the custom docker compose file using [yq](https://github.com/mikefarah/yq):
```
yaml-merge docker-compose.yml .deployment/docker-compose-overrides.yml > custom-docker-compose.yml
yq m -a docker/docker-compose.yml "$DEEPFORGE_DEPLOYMENT_DIR"/docker-compose-overrides.yml > custom-docker-compose.yml
```
Next, the generated file can be used with docker-compose:
```
Expand Down
13 changes: 11 additions & 2 deletions .deployment/deploy-deepforge
Original file line number Diff line number Diff line change
Expand Up @@ -13,8 +13,9 @@ fi;
< config/components.json jq '.Compute.backends=(.Compute.backends | map(select(. != "local")))' \
> config/components2.json
< config/components2.json jq '.Storage.backends=(.Storage.backends | map(select(. != "gme")))' \
> config/components.json
rm config/components2.json
> config/components3.json
jq '.LanguageServers = input' config/components3.json utils/languageServers.json > config/components.json
rm config/components3.json config/components2.json

# Merging the custom override yml file
yq m -a docker/docker-compose.yml "$DEEPFORGE_DEPLOYMENT_DIR"/docker-compose-overrides.yml > custom-docker-compose.yml
Expand All @@ -25,4 +26,12 @@ docker-compose --file custom-docker-compose.yml -p deepforge stop $SERVER_NAME
docker-compose --file custom-docker-compose.yml -p deepforge rm -f $SERVER_NAME
docker-compose --file custom-docker-compose.yml -p deepforge up -d $SERVER_NAME

if [[ $SERVER_NAME = "server_stable" ]]; then
docker-compose --file custom-docker-compose.yml -p deepforge pull language_servers
docker-compose --file custom-docker-compose.yml -p deepforge stop language_servers
docker-compose --file custom-docker-compose.yml -p deepforge rm -f language_servers
docker-compose --file custom-docker-compose.yml -p deepforge up -d language_servers
fi;


docker image prune -f
15 changes: 15 additions & 0 deletions .deployment/docker-compose-overrides.yml
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ services:
environment:
- "DEEPFORGE_HOST=https://dev.deepforge.org"
- "DEEPFORGE_INTERACTIVE_COMPUTE_HOST=https://dev-compute.deepforge.org"
- "DEEPFORGE_LANGUAGE_SERVER_HOST=https://langservers.deepforge.org"
volumes:
- "${DEEPFORGE_DEPLOYMENT_DIR}:/.deployment"
- "${DEEPFORGE_DEPLOYMENT_DIR}/../config/components.json:/deepforge/config/components.json"
Expand All @@ -16,6 +17,7 @@ services:
- "DEEPFORGE_INTERACTIVE_COMPUTE_HOST=https://compute.deepforge.org"
- "DEEPFORGE_PUBLIC_KEY=/token_keys/public_key"
- "DEEPFORGE_PRIVATE_KEY=/token_keys/private_key"
- "DEEPFORGE_LANGUAGE_SERVER_HOST=https://langservers.deepforge.org"
image: deepforge/kitchen-sink:stable
ports:
- "9000:8888"
Expand All @@ -26,3 +28,16 @@ services:
- "${DEEPFORGE_DEPLOYMENT_DIR}/../config/components.json:/deepforge/config/components.json"
depends_on:
- mongo

language_servers:
image: deepforge/language-servers:latest
ports:
- "5000:5000"
volumes:
- "${DEEPFORGE_DEPLOYMENT_DIR}/../language-servers.yml:/root/language-servers.yml"
- "${DEEPFORGE_DEPLOYMENT_DIR}:/.deployment"
- "/tmp/python-models:/tmp/python-models"
entrypoint: /.deployment/langserver-entrypoint.sh
depends_on:
- server
- server_stable
8 changes: 8 additions & 0 deletions .deployment/langserver-entrypoint.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
#!/bin/bash
# Remove pypi tensorflow in favor of conda installation
source activate base
pip uninstall tensorflow -y
conda install tensorflow==1.14 -y

node $(npm root -g)/jsonrpc-ws-proxy/dist/server.js --port $PORT --languageServers ~/language-servers.yml

13 changes: 13 additions & 0 deletions .github/workflows/docker-build.yml
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,19 @@ jobs:
dockerfile: docker/Dockerfile.kitchensink
buildargs: "TAG=${{ env.TAG }}"

- name: Publish to docker hub (language-servers)
uses: elgohr/Publish-Docker-Github-Action@master

with:
name: deepforge/language-servers
username: ${{ secrets.DOCKER_USERNAME }}
password: ${{ secrets.DOCKER_PASSWORD }}
# ToDo: When should this image be updated? On every push to master??
tags: latest
dockerfile: dockerfile/Dockerfile.langservers
if: github.event_name != 'release'


- name: Deploy (dev.deepforge.org)
uses: appleboy/ssh-action@master
with:
Expand Down
12 changes: 12 additions & 0 deletions config/config.extensions.js
Original file line number Diff line number Diff line change
@@ -1,6 +1,18 @@
const components = require('./components.json');

function getLanguageServersConfig () {
if(components.LanguageServers) {
return {
hostName: process.env.DEEPFORGE_LANGUAGE_SERVER_HOST || components.LanguageServers.hostName,
servers: components.LanguageServers.servers
};
}
}


module.exports = config => {
config.extensions = {};
config.extensions.InteractiveComputeHost = process.env.DEEPFORGE_INTERACTIVE_COMPUTE_HOST;
config.extensions.LanguageServers = getLanguageServersConfig();
return config;
};
33 changes: 33 additions & 0 deletions docker/Dockerfile.langservers
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
FROM continuumio/anaconda3

LABEL maintainer1.name="Brian Broll"\
maintainer1.email="[email protected]"

LABEL maintainer2.name="Umesh Timalsina"\
maintainer2.email="[email protected]"

EXPOSE 5000

ENV PORT 5000

RUN apt-get update && curl -sL https://deb.nodesource.com/setup_12.x | bash - &&\
apt-get install -y nodejs && apt-get install unzip

RUN npm install -g npm && npm -g config set user root && npm install -g jsonrpc-ws-proxy js-yaml

RUN conda update conda -yq && conda install python=3.7

COPY src/plugins/GenerateJob/templates/environment.worker.yml .

COPY utils/install-microsoft-python-langserver.sh .

COPY docker/create-servers-yml.js .

COPY utils/languageServers.json utils/

RUN ./install-microsoft-python-langserver.sh && npm link js-yaml &&\
node ./create-servers-yml.js && rm create-servers-yml.js

RUN conda env update -n base --file environment.worker.yml && rm environment.worker.yml && conda clean -afy

ENTRYPOINT node $(npm root -g)/jsonrpc-ws-proxy/dist/server.js --port $PORT --languageServers ~/languageServers.yml
22 changes: 22 additions & 0 deletions docker/create-servers-yml.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
const yaml = require('js-yaml');
const fs = require('fs');
const path = require('path');
const langServers = require(path.resolve(__dirname, '..', 'utils', 'languageServers'));

function dumpLangServerYaml(fileName) {
const serversYml = {
langservers: {}
};

Object.keys(langServers.servers).forEach(server => {
const command = langServers.servers[server].command;
serversYml.langservers[server] = Array.isArray(command) ? command : [command];
});

fs.writeFileSync(fileName, yaml.safeDump(serversYml));
}


if (require.main === module) {
dumpLangServerYaml('languageServers.yml');
}
1 change: 1 addition & 0 deletions environment.server.yml
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
name: deepforge
dependencies:
- python=3.7
- python-language-server
umesh-timalsina marked this conversation as resolved.
Show resolved Hide resolved
Loading