Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Integrate Python Language Server to Monaco Editor (Closes #1908, #571) #1910

Open
wants to merge 43 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
43 commits
Select commit Hold shift + click to select a range
542f549
WIP #1908: Add working example with basic pyls
umesh-timalsina Sep 17, 2020
7aeebe0
Provide Dockerfile and build for a separate language servers service
umesh-timalsina Sep 21, 2020
da381fb
WIP- Update deployment of language servers and minor eslint fixes
umesh-timalsina Sep 23, 2020
2802a5d
Use microsoft's python language server in favor of pyls(palantir)
umesh-timalsina Sep 25, 2020
4503ae4
WIP- Fix deployment issue with language servers
umesh-timalsina Sep 25, 2020
29775a7
WIP- Use components.json for languageserver configuration
umesh-timalsina Sep 30, 2020
b777e43
Merge remote-tracking branch 'origin/master' into 1908-pyls-integration
umesh-timalsina Sep 30, 2020
08291c5
WIP- Fix languageServers.json file; jq deployment; uninstall jsonrpc-…
umesh-timalsina Oct 1, 2020
dd9fe28
Merge remote-tracking branch 'origin/master' into 1908-pyls-integration
umesh-timalsina Oct 1, 2020
37c1ddc
WIP- Fix typo in config.extensions, rename token to tokens in languag…
umesh-timalsina Oct 2, 2020
03b41f5
WIP- Fix jsonrpc-ws-proxy call in Dockerfile.langservers entrypoint
umesh-timalsina Oct 2, 2020
c0fc77f
WIP-Fix jsonrpc-ws-proxy in langserver-entrypoint.sh
umesh-timalsina Oct 2, 2020
376fbc6
WIP_ Remove python-language-server from requirements
umesh-timalsina Oct 2, 2020
dcd5ae5
WIP- Fix typo in DeepforgeLanguageClient
umesh-timalsina Oct 2, 2020
e26e7c3
WIP- Use python3.7 anaconda in docker image
umesh-timalsina Oct 7, 2020
f4facd8
WIP- Move languageServers.json to .deployment
umesh-timalsina Oct 7, 2020
5ca1db3
WIP- Use inmemory models instead of file models
umesh-timalsina Oct 7, 2020
9e2525a
Resolve merge conflicts and merge branch master
umesh-timalsina Oct 12, 2020
8bd0f3e
Merge remote-tracking branch 'origin/1908-pyls-integration' into 1908…
umesh-timalsina Oct 12, 2020
ddbf23a
Update README for deployment
umesh-timalsina Oct 12, 2020
b1df3f6
WIP- Use componentSettings for LanguageServers; Docker build for lang…
umesh-timalsina Oct 22, 2020
2bd224e
Merge remote-tracking branch 'origin/master' into 1908-pyls-integration
umesh-timalsina Oct 28, 2020
21ad734
WIP- Test docker build for language servers
umesh-timalsina Oct 28, 2020
c8227d1
WIP- Fix Dockerfile.langservers path
umesh-timalsina Oct 28, 2020
a339b9c
WIP- Add npm install step in github workflows
umesh-timalsina Oct 28, 2020
087323b
WIP- Don't update conda in Dockerfile
umesh-timalsina Oct 28, 2020
aa84bc2
WIP- Use latest anaconda docker image
umesh-timalsina Oct 28, 2020
a9e321a
Merge remote-tracking branch 'origin/master' into 1908-pyls-integration
umesh-timalsina Nov 11, 2020
10bec66
WIP- Use anaconda3:2019.10 as image base
umesh-timalsina Nov 11, 2020
89b2e68
Revert to pip installing packages
umesh-timalsina Nov 12, 2020
20bd21f
WIP- Fix Keras version in dockerfile
umesh-timalsina Nov 12, 2020
193698d
Merge remote-tracking branch 'origin/master' into 1908-pyls-integration
umesh-timalsina Dec 16, 2020
d7b9a0d
WIP- Minor fixes to the language server settings
umesh-timalsina Dec 16, 2020
aa60904
Merge remote-tracking branch 'origin/master' into 1908-pyls-integration
umesh-timalsina Jan 20, 2021
61bd70f
WIP- Deploy Branch for Testing
umesh-timalsina Jan 20, 2021
f75f652
WIP- Publish KitchenSink image for testing
umesh-timalsina Jan 20, 2021
c45823b
WIP- Publish kitchen sink image for testing
umesh-timalsina Jan 20, 2021
3d84cdb
WIP- Add Push Tags
umesh-timalsina Jan 20, 2021
d5bfa3f
WIP- Fix Tags for kitchensink image
umesh-timalsina Jan 20, 2021
ff110bc
WIP- Fix docker-compose for testing
umesh-timalsina Jan 20, 2021
36c6723
WIP- Checkout branch in deployment script
umesh-timalsina Jan 20, 2021
81657c3
WIP- Fix branch checkout in workflow
umesh-timalsina Jan 20, 2021
71979a3
WIP- Fix entrypoint and setup debugging.
umesh-timalsina Jan 21, 2021
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 6 additions & 2 deletions .deployment/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,13 @@ The script `deploy-deepforge` is used for standard deployment of deepforge using

Additionally, this contains a file with customizations to the standard docker-compose.yml file which allows us to modify the entrypoint and install a version of tensorflow [compatible with the CPU of the deployment machine](https://github.com/deepforge-dev/deepforge/issues/1561).

The deployment is updated by first creating the custom docker compose file using [yaml-merge](https://github.com/alexlafroscia/yaml-merge):
Moreover, we also a proxy server with that spins up different language servers that we provide for
intelligent syntax
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks like some odd line breaking...

highlighting in deepforge's browser. For more information checkout the language server's docker [file](../docker/Dockerfile.langservers)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"in deepforge's browser" -> "in DeepForge"


The deployment is updated by first creating the custom docker compose file using [yq](https://github.com/mikefarah/yq):
```
yaml-merge docker-compose.yml .deployment/docker-compose-overrides.yml > custom-docker-compose.yml
yq m -a docker/docker-compose.yml "$DEEPFORGE_DEPLOYMENT_DIR"/docker-compose-overrides.yml > custom-docker-compose.yml
```
Next, the generated file can be used with docker-compose:
```
Expand Down
8 changes: 8 additions & 0 deletions .deployment/deploy-deepforge
Original file line number Diff line number Diff line change
Expand Up @@ -25,4 +25,12 @@ docker-compose --file custom-docker-compose.yml -p deepforge stop $SERVER_NAME
docker-compose --file custom-docker-compose.yml -p deepforge rm -f $SERVER_NAME
docker-compose --file custom-docker-compose.yml -p deepforge up -d $SERVER_NAME

if [[ $SERVER_NAME = "server_stable" ]]; then
docker-compose --file custom-docker-compose.yml -p deepforge pull language_servers
docker-compose --file custom-docker-compose.yml -p deepforge stop language_servers
docker-compose --file custom-docker-compose.yml -p deepforge rm -f language_servers
docker-compose --file custom-docker-compose.yml -p deepforge up -d language_servers
fi;


docker image prune -f
15 changes: 15 additions & 0 deletions .deployment/docker-compose-overrides.yml
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ services:
environment:
- "DEEPFORGE_HOST=https://dev.deepforge.org"
- "DEEPFORGE_INTERACTIVE_COMPUTE_HOST=https://dev-compute.deepforge.org"
- "DEEPFORGE_LANGUAGE_SERVER_HOST=https://langservers.deepforge.org"
volumes:
- "${DEEPFORGE_DEPLOYMENT_DIR}:/.deployment"
- "${DEEPFORGE_DEPLOYMENT_DIR}/../config/components.json:/deepforge/config/components.json"
Expand All @@ -16,6 +17,7 @@ services:
- "DEEPFORGE_INTERACTIVE_COMPUTE_HOST=https://compute.deepforge.org"
- "DEEPFORGE_PUBLIC_KEY=/token_keys/public_key"
- "DEEPFORGE_PRIVATE_KEY=/token_keys/private_key"
- "DEEPFORGE_LANGUAGE_SERVER_HOST=https://langservers.deepforge.org"
image: deepforge/kitchen-sink:stable
ports:
- "9000:8888"
Expand All @@ -26,3 +28,16 @@ services:
- "${DEEPFORGE_DEPLOYMENT_DIR}/../config/components.json:/deepforge/config/components.json"
depends_on:
- mongo

language_servers:
image: deepforge/language-servers:latest
ports:
- "5000:5000"
volumes:
- "${DEEPFORGE_DEPLOYMENT_DIR}/../language-servers.yml:/root/language-servers.yml"
- "${DEEPFORGE_DEPLOYMENT_DIR}:/.deployment"
- "/tmp/python-models:/tmp/python-models"
entrypoint: /.deployment/langserver-entrypoint.sh
depends_on:
- server
- server_stable
8 changes: 8 additions & 0 deletions .deployment/langserver-entrypoint.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
#!/bin/bash
# Remove pypi tensorflow in favor of conda installation
source activate base
pip uninstall tensorflow -y
conda install tensorflow==1.14 -y

node $(npm root -g)/jsonrpc-ws-proxy/dist/server.js --port $PORT --languageServers ~/language-servers.yml

13 changes: 13 additions & 0 deletions .github/workflows/docker-build.yml
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,19 @@ jobs:
dockerfile: docker/Dockerfile.kitchensink
buildargs: "TAG=${{ env.TAG }}"

- name: Publish to docker hub (language-servers)
uses: elgohr/Publish-Docker-Github-Action@master

with:
name: deepforge/language-servers
username: ${{ secrets.DOCKER_USERNAME }}
password: ${{ secrets.DOCKER_PASSWORD }}
# ToDo: When should this image be updated? On every push to master??
tags: latest
dockerfile: dockerfile/Dockerfile.langservers
if: github.event_name != 'release'


- name: Deploy (dev.deepforge.org)
uses: appleboy/ssh-action@master
with:
Expand Down
24 changes: 24 additions & 0 deletions config/config.extensions.js
Original file line number Diff line number Diff line change
@@ -1,6 +1,30 @@
const yaml = require('js-yaml');
const path = require('path');
const fs = require('fs');

const SERVERS_YML = path.join(__dirname, '..', 'language-servers.yml');

function getAvailableLanguageServers () {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am a little wary about this. It seems to be assuming that any servers defined in language-servers.yml are available but since that file is checked in to git, it seems likely that this will end up with invalid language servers.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep. can it be solved by moving language-servers.yml to the deployment directory and using the config in production mode only?

const parsed = yaml.safeLoad(fs.readFileSync(SERVERS_YML));
return parsed.langservers ? Object.keys(parsed.langservers): [];
}

function getWorkspaceURIs() {
const availableServers = getAvailableLanguageServers();
const workspaces = {};
availableServers.forEach(server => {
workspaces[server] = `file:///tmp/${server}-models/`;
});
return workspaces;
}

module.exports = config => {
config.extensions = {};
config.extensions.InteractiveComputeHost = process.env.DEEPFORGE_INTERACTIVE_COMPUTE_HOST;
config.extensions.languageServers = {
host: process.env.DEEPFORGE_LANGUAGE_SERVER_HOST,
servers: getAvailableLanguageServers(),
workspaceURIs: getWorkspaceURIs(),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After looking at how this is used, maybe this should be called rootURIs or rootWorkspaceURIs? When I first read it, I was concerned that all users may be sharing the same workspace (which I don't think will be an issue).

};
return config;
};
17 changes: 17 additions & 0 deletions docker/Dockerfile.langservers
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
FROM continuumio/anaconda3

EXPOSE 5000

ENV PORT 5000

RUN apt-get update && curl -sL https://deb.nodesource.com/setup_12.x | bash - && apt-get install -y nodejs

RUN npm install -g npm && npm install -g jsonrpc-ws-proxy

RUN conda update conda -yq && conda install python=3.7

COPY src/plugins/GenerateJob/templates/environment.worker.yml .

RUN conda env update -n base --file environment.worker.yml && rm environment.worker.yml && conda clean -afy

ENTRYPOINT node $(npm root -g)/jsonrpc-ws-proxy/dist/server.js --port $PORT --languageServers ~/language-servers.yml
1 change: 1 addition & 0 deletions environment.server.yml
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
name: deepforge
dependencies:
- python=3.7
- python-language-server
umesh-timalsina marked this conversation as resolved.
Show resolved Hide resolved
1 change: 1 addition & 0 deletions environment.worker.yml
Original file line number Diff line number Diff line change
Expand Up @@ -6,3 +6,4 @@ dependencies:
- matplotlib==3.2.2
- simplejson
- plotly
- python-language-server==0.25.0
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same comment as above.

5 changes: 5 additions & 0 deletions language-servers.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
langservers:
python:
- pyls
- --log-file
- pyls.log
52 changes: 48 additions & 4 deletions package-lock.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions package.json
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,7 @@
"express": "^4.14.0",
"graceful-fs": "^4.1.10",
"js-yaml": "^3.13.1",
"jsonrpc-ws-proxy": "0.0.5",
"lodash.difference": "^4.1.2",
"lodash.merge": "^4.6.2",
"lodash.template": "^4.4.0",
Expand Down
103 changes: 103 additions & 0 deletions src/visualizers/widgets/TextEditor/DeepForgeLanguageClient.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,103 @@
/* globals define */

define([
'./lib/vscode-ws-jsonrpc.min',
'./lib/monaco-languageclient.min',
'./lib/reconnecting-websocket.min',
], function (
vscodeWSJSONRpc,
LangaugeClient,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Langauge -> Language

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My bad on the typos. Sorry.

RS
) {
const {ReconnectingWebSocket} = RS,
{listen} = vscodeWSJSONRpc;
const {
MonacoLanguageClient,
CloseAction,
ErrorAction,
Services,
MonacoCommands,
MonacoWorkspace,
ConsoleWindow,
MonacoLanguages,
MonacoToProtocolConverter,
ProtocolToMonacoConverter,
createConnection
} = LangaugeClient;

class DeepForgeLanguageClient {
constructor(editor, serverURL, opts) {
const services = createServices(editor, opts);
Services.install(services);
this.serverURL = serverURL;
this.socket = createReconnectingWebSocket(this.serverURL, opts);
this._initializeClient(opts);
}

_initializeClient(opts) {
listen({
webSocket: this.socket,
onConnection: connection => {
const languageClient = createLanguageClient(connection, opts);
const disposable = languageClient.start();
connection.onClose(() => disposable.dispose());
}
});
}
}

const createLanguageClient = function (connection, opts) {
return new MonacoLanguageClient(
{
name: opts.name || 'DeepForge Language Client',
clientOptions: {
documentSelector: [opts.language || 'python'],
errorHandler: {
error: () => ErrorAction.Continue,
closed: () => CloseAction.DoNotRestart
}
},
connectionProvider: {
get(errorHandler, closeHandler, /*outputChannel*/) {
return Promise.resolve(
createConnection(connection, errorHandler, closeHandler)
);
}
}
}
);
};

const createReconnectingWebSocket = function (url, opts = {}) {
const socketOpts = {
maxReconnectionDelay: opts.socket.maxReconnectionDelay || 10000,
minReconnectionDelay: opts.socket.minReconnectionDelay || 1000,
reconnectionDelayGrowFactor: opts.socket.reconnectionDelayGrowFactor || 1.3,
connectionTimeout: 10000,
maxRetries: Infinity,
debug: opts.socket.debug || false
};

return new ReconnectingWebSocket(
url,
[],
socketOpts
);
};

const createServices = function (editor, opts) {
const m2p = new MonacoToProtocolConverter();
const p2m = new ProtocolToMonacoConverter();
let services = {
commands: new MonacoCommands(editor),
languages: new MonacoLanguages(p2m, m2p),
workspace: new MonacoWorkspace(p2m, m2p, opts.rootUri)
};
if (opts.debug) {
services.window = new ConsoleWindow();
}
return services;
};

return DeepForgeLanguageClient;
});
Loading