-
Notifications
You must be signed in to change notification settings - Fork 2
/
Copy pathindex.html
384 lines (316 loc) · 40.2 KB
/
index.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
<!DOCTYPE html><html lang="en"><head><meta charset="utf-8"><title>anode@microsoft</title><meta name="author" content="elad ben-israel"><link href="css/bootstrap.min.css" rel="stylesheet"><style type="text/css">body {
padding-top: 60px;
padding-bottom: 40px;
}
.sidebar-nav {
padding: 9px 0;
}</style><link href="css/index.css" rel="stylesheet"></head><body><div class="navbar navbar-fixed-top"><div class="navbar-inner"><div class="container"><a href="#" class="brand"><strong>anode@microsoft</strong> blog</a><ul class="nav"></ul></div></div></div><div class="container"><div class="row"><div class="span8"><br><br><div class="row"></div><a name="acli" class="hash"> </a><div id="entry-acli" class="entry"><h1>Anode Command Line Interface</h1><p><small>Posted on Monday, May 28, 2012 by <a href="https://github.com/amiturgman">Ami Turgman</a></small><br/></p><p>Hi!
</p>
<p>So, now that our <a href="http://anodejs.org">anode</a> farm is being used for prototyping as well as for production services, it's time to talk a bit of how we manage and track our applications.
For this, let me introduce you to <a href="https://github.com/amiturgman/ACLI">Anode Command Line Interface</a>.
</p>
<p>ACLI is a command line interface, developed mainly for the use in the <a href="http://anodejs.org">anode project</a>.
</p>
<p><strong>Why do you need another command line interface, there are already few out there...</strong><br>Well, that's true... but believe me, the things these eyes have seen...
</p>
<p><strong>No, really... why do you need another command line interface?</strong><br>From my experience they were'nt very easy to use. Some were too buggy, some couldn't get HTML objects as a command execution result, and others were too limited of how to configure the control look & feel, command line prompt text, and more... each of the existing CLIs had some of the required functionality, but did not support other features that we needed.<br>The most suitable library for our needs was the <a href="https://github.com/mozilla/gcli">GCLI</a> component, which was the main inspiration for implementing ACLI, mostly in the area of the command structure.
</p>
<p><strong>What are you using it for?</strong><br>We use it to
</p>
<ul>
<li>Manage the farm <ul>
<li>Getting information from all servers such as settings, processes and counters </li>
<li>Invoking different actions on all servers like forcing sync process</li>
<li>Getting application list</li>
</ul>
</li>
<li>Manage applications<ul>
<li>Viewing latest commits (integrated with github)</li>
<li>Getting information for an application, such as process info, ports, etc'</li>
<li>Restarting an application</li>
</ul>
</li>
<li>Each application that is hosted on the farm implements its own plugin that is integrated into the console, and allows the developers to manage it with its own specific set of commands.</li>
<li>View logs for our applications, filtered by verbosity level and other params</li>
<li>Invoking end-to-end tests and viewing results</li>
</ul>
<h2>Requirements</h2>
<p>In addition to the obvious features (commands history, clear/help commands), we also wanted the following:
</p>
<ul>
<li>Supports plugins- remote commands (remote service REST APIs) integrated into the console, using <a href="https://github.com/anodejs/node-docrouter">docRouter</a> metadata.<br> Supporting plugins with client side processing and styling. </li>
<li>Visualizing json data as an HTML table with auto collapsing deep elements.</li>
<li>Supporting broadcasting requests when working on a farm.</li>
<li>Managing environment variables and use them as part of the commands.</li>
<li>Keeping command line history, plugins and environment variables persistent.</li>
<li>Supporting working in parallel with few instances/tabs of the console.</li>
</ul>
<p><img src="https://github.com/amiturgman/ACLI/raw/master/cli_myboard_small.jpg" alt="Example for 'My Board' feature" title="ACLI with My Board">
</p>
<h2>So, why is this so exciting?</h2>
<ol>
<li><p>Since ACLI supports <strong>plugins</strong>, it's easy to use it as a <strong>management tool in any node.js application</strong>.<br>Let's assume you develop a website using nodeJS. You can create another page under <code>/management</code> which will host ACLI, and then on the server side, implement any REST API that will be integrated into the console as a command, such as getting logs, getting list of users, making operations on users, and everything you can think of.<br><em>Protecting this area by authentication/authorization mechanism will also be a good idea :)</em> </p>
</li>
<li><p>The powerful <strong>internal json-view</strong> control, which visualizes any json object provides a very easy-to-begin-with json-result visualizer.<br>You can start creating server side commands which are integrated into the console, without writing any client-side code. If you'd like more advanced/custom look for the results, you'll be able to add client side handler that generates any HTML/jQuery object in later stages. The server side can also return HTML instead of a json object of course. </p>
</li>
<li><p>Assuming that you are working on a <strong>farm</strong>, you will be able to create a command that <strong>collects data from all servers</strong>, displaying the progress of the process and then when all data is collected, displaying the results! This is a very powerful feature that allows you to create commands that collect the status from all servers, or invoking an action on all servers, such as resetting the application. </p>
</li>
<li><p><strong>Managing environment variables</strong> like any other native CLI will allow you to use them as part of any command </p>
<ol>
<li>Implicitly- for example as an environment variable default value for a parameter in a command, or </li>
<li>Explicitly- by using the <code>$</code> sign such as <code>log --top $myTop</code>.</li>
</ol>
</li>
<li><p>The console <strong>automatically keeps the state</strong> of the environment variables, the command line history and the installed plugins in the <strong>local storage</strong>.<br>Every time you open the console, it will be in the exact state that it was when you last closed it. You won't have to install the plugins again, or re-set environment values. In addition to that, the state is kept <strong>per each session/tab</strong> that we opened. This way we can create several <strong>work spaces</strong> in which each one of them has certain environment variables, certain installed plugins, and so on... and all that in the context of the application that we are managing.</p>
</li>
<li><p>The <strong>My Board</strong> feature which allows you to keep results <strong>always on screen</strong>. This is kind of a panel/container on the right side of the console, in which you can drag-n-drop any command execution result. In the example above, you can see that i'm keeping the environment variables panel (which is a json-view control by the way) on the <em>My Board</em> panel.
This way, I can always see the current environment variables setting state (the <code>set -o</code> command returns an online control which will be updated any time an environment variable is updated).
This panel can be toggled on/off at any time by pressing on its header. </p>
</li>
</ol>
<h2>Getting Started</h2>
<p>The following is an example of how to quickly start using the component.
</p>
<p>In addition to that, you can find basic and advanced <a href="https://github.com/amiturgman/ACLI/tree/master/samples">samples</a> which include a node.js application with a sample plugin on <a href="https://github.com/amiturgman/ACLI">github</a>.<br>The <a href="https://github.com/amiturgman/ACLI/blob/master/design.md">design</a> document includes all the details needed in order to smoothly start integrating plugins as commands into the console.
</p>
<p>HTML file:
</p>
<pre><code><body>
<div class="cli-output" id="cliOutput"></div>
<div class="cli-my-board" id="cliMyBoard">
<div class="cli-my-board-close"></div>
<div class="cli-my-board-title">My Board</div>
</div>
<div class="cli-input-container">
<span class="cli-promptText" id="cliPrompt">></span>
<input id="cliInput" class="cli-input" type="text">
</div>
</body></code></pre>
<p>client side js file:
</p>
<pre><code>var cli = $("#cliInput").cli(
{
resultsContainer: $("#cliOutput"),
promptControl: $("#cliPrompt"),
myBoard: $("#cliMyBoard"),
environment: {user: { type: 'string', value: '', description: 'The current user' }},
commands: [],
context: { some: 'object' },
welcomeMessage: "Welcome to anode console!<br/>. Type <b>help</b> to start exploring the commands currently supported!<br/>"
}
);</code></pre>
<p>server side plugin with a command that gets a template and a querystring parameters and returns a JSON object:
</p>
<pre><code>var express = require('express'),
app = express.createServer(),
docRouter = require('docrouter').DocRouter;
module.exports = app;
app.use(docRouter(express.router, '/api/someplugin', function(app) {
app.get('/json/:tparam', function(req, res) {
var tparam = req.params.tparam1;
var qparam = req.query['qparam'];
var o = {tparam: tparam, qparam: qparam};
res.writeHead(200, {'Content-Type': 'application/json' });
res.end(JSON.stringify(o));
},
{
id: 'sample_json',
name: 'json',
usage: 'json tparam qparam',
example: 'json tparam1 qparamValue',
doc: 'sample for a GET command getting a template param and a query param',
params: {
"tparam" : {
"short": "b",
"type": "string",
"doc": "template param",
"style": "template",
"required": "true"
},
"qparam" : {
"short": "q",
"type": "string",
"doc": "query string param",
"style": "query",
"required": "true"
}
}
}
);
}));</code></pre>
<p>You are more than welcome to use this plugin.<br>Your feedback is highly appreciated! feel free to test it, open issues on <a href="https://github.com/amiturgman/ACLI">github</a> or send questions and comments to <a href="mailto:[email protected]">Ami Turgman</a>.
</p>
<hr><div class="comment"><a href="#" data-disqus-identifier="/#acli" class="comment">Leave a comment</a><img src="data:image/gif;base64,R0lGODlhEAALAPQAAP///z2LqeLt8dvp7u7090GNqz2LqV+fuJ/F1IW2ycrf51aatHWswaXJ14i4ys3h6FmctUCMqniuw+vz9eHs8fb5+meku+Tu8vT4+cfd5bbT3tbm7PH2+AAAAAAAAAAAACH/C05FVFNDQVBFMi4wAwEAAAAh/hpDcmVhdGVkIHdpdGggYWpheGxvYWQuaW5mbwAh+QQJCwAAACwAAAAAEAALAAAFLSAgjmRpnqSgCuLKAq5AEIM4zDVw03ve27ifDgfkEYe04kDIDC5zrtYKRa2WQgAh+QQJCwAAACwAAAAAEAALAAAFJGBhGAVgnqhpHIeRvsDawqns0qeN5+y967tYLyicBYE7EYkYAgAh+QQJCwAAACwAAAAAEAALAAAFNiAgjothLOOIJAkiGgxjpGKiKMkbz7SN6zIawJcDwIK9W/HISxGBzdHTuBNOmcJVCyoUlk7CEAAh+QQJCwAAACwAAAAAEAALAAAFNSAgjqQIRRFUAo3jNGIkSdHqPI8Tz3V55zuaDacDyIQ+YrBH+hWPzJFzOQQaeavWi7oqnVIhACH5BAkLAAAALAAAAAAQAAsAAAUyICCOZGme1rJY5kRRk7hI0mJSVUXJtF3iOl7tltsBZsNfUegjAY3I5sgFY55KqdX1GgIAIfkECQsAAAAsAAAAABAACwAABTcgII5kaZ4kcV2EqLJipmnZhWGXaOOitm2aXQ4g7P2Ct2ER4AMul00kj5g0Al8tADY2y6C+4FIIACH5BAkLAAAALAAAAAAQAAsAAAUvICCOZGme5ERRk6iy7qpyHCVStA3gNa/7txxwlwv2isSacYUc+l4tADQGQ1mvpBAAIfkECQsAAAAsAAAAABAACwAABS8gII5kaZ7kRFGTqLLuqnIcJVK0DeA1r/u3HHCXC/aKxJpxhRz6Xi0ANAZDWa+kEAA7AAAAAAAAAAAA" class="loading"></div><div class="comments"><a href="#" class="nocomment">Hide comments</a><div class="disqus_thread"></div></div></div><hr><a name="securing-mongo-with-ssltunnel" class="hash"> </a><div id="entry-securing-mongo-with-ssltunnel" class="entry"><h1>Securing MongoDB traffic with ssltunnel on Windows</h1><p><small>Posted on Thursday, March 29, 2012 by <a href="https://github.com/dimastopel">Dima Stopel</a></small><br/></p><p>Hi guys,
</p>
<p>Today I'd like to discuss <a href="https://github.com/anodejs/node-ssltunnel">ssltunnel</a>. So, what is it? ssltunnel is a lightweight TCP over SSL / TLS tunnel running over node. If you need to add confidentiality (privacy), integrity, and authenticity to your TCP stream this is the tool for you. ssltunnel is available as node package via <a href="http://search.npmjs.org/#/ssltunnel">npm</a>. It is distributed under MIT license.
</p>
<h2>Intro</h2>
<p>In order to make the discussion about the deeper parts more concrete let's take a concrete example. Let's say that you use mongodb as your database and you need to connect to your CLI client (mongo.exe) running on you PC to your mongo server (mongod.exe) running on your remote VM. Now suppose that you want to assure that all the traffic is encrypted and that only you can connect to your mongo server. Here ssltunnel becomes handy.
</p>
<p>ssltunnel consists of two parts: <em>sslproxy</em> and <em>sslserver</em>. The sslproxy part is running on the client machine communicating with the real client, mongo.exe in our case, and sslserver. The sslserver part is running on the server machine and communicating with sslproxy and the back-end server, mongod.exe in our case. sslproxy authenticates sslserver via SSL server certificate. sslserver authenticates sslproxy via SSL client certificate. The traffic itself is encrypted using standard SSL / TLS protocol.
</p>
<h2>Tunneling mongo traffic through ssltunnel</h2>
<p>So, let's create this secure tunnel step by step. Let's suppose the following:
</p>
<ol>
<li>all parts are running on local machine (for the sake of simplicity)</li>
<li><code>mongod.exe</code> listening port is 50080</li>
<li><code>sslserver</code> listening port is 80443</li>
<li><code>sslclient</code> listening port is 50081</li>
</ol>
<h3>step 1: installation</h3>
<p>Please download the <a href="http://nodejs.org/#download">latest node</a>. Open <em>cmd</em> and install ssltunnel package via npm. I'll install it on c:\ (I run Windows).
</p>
<pre><code>anydir/> cd /d c:\
c:\> npm install ssltunnel</code></pre>
<p>You should now see <em>node_modules</em> directory created under <code>c:\</code>. Congratulations, you've successfully install
ssltunnel :)
</p>
<h3>step 2: running the mongo server</h3>
<p>If you don't have mongo please <a href="http://www.mongodb.org/downloads">download</a> the latest version now.
Extract it in the directory of your choice. Run <em>cmd</em> and navigate to this directory. Now you can run the server.
For the sake of simplicity I instruct it to put data in <em>data\db</em> folder.
</p>
<pre><code>d:\src\mongodb-win32-x86_64-2.0.2\bin>mongod --port 50080 --dbpath data\db</code></pre>
<p>You should see something like this:
</p>
<pre><code>Tue Mar 27 16:41:56 [initandlisten] MongoDB starting : pid=3232 port=50080 dbpath=data\db 64-bit host=Dimast-laptop
Tue Mar 27 16:41:56 [initandlisten] db version v2.0.2, pdfile version 4.5
Tue Mar 27 16:41:56 [initandlisten] git version: 514b122d308928517f5841888ceaa4246a7f18e3
Tue Mar 27 16:41:56 [initandlisten] build info: windows (6, 1, 7601, 2, 'Service Pack 1') BOOST_LIB_VERSION=1_42
Tue Mar 27 16:41:56 [initandlisten] options: { dbpath: "data\db", port: 50080 }
Tue Mar 27 16:41:56 [initandlisten] journal dir=data/db/journal
Tue Mar 27 16:41:56 [initandlisten] recover : no journal files present, no recovery needed
Tue Mar 27 16:41:56 [initandlisten] waiting for connections on port 50080
Tue Mar 27 16:41:56 [websvr] admin web console waiting for connections on port 51080</code></pre>
<h3>step 3: establishing the tunnel</h3>
<p>Let's navigate to the <em>bin</em> directory of ssltunnel:
</p>
<pre><code>c:\>cd c:\node_modules\ssltunnel\bin</code></pre>
<p>Now we will create sslserver. Note that you need server certificate with private key and public client certificate
in order to be able to verify the client. I have provided test certificates as part of the package.
<em>Please generate and use your own for production systems</em>.
See how to do it <a href="https://github.com/anodejs/node-ssltunnel">here</a>.
</p>
<p>So we instruct the sslserver (<em>-r server</em>) to listen on port <em>50443</em> and connect to back end server on
host <em>localhost</em> (the default, actually) and port <em>50080</em>. We also provide public and private server certificates
and public client certificate which are stored in decrypted pem files.
</p>
<pre><code>c:\node_modules\ssltunnel\bin>ssltunnel.cmd
-r server
--proxy_port 50443
--server_port 50080
--server-host localhost
--srv_pub_cert ..\testcerts\sc_public.pem
--srv_prv_cert ..\testcerts\sc_private.pem
--clt_pub_cert ..\testcerts\cc_public.pem</code></pre>
<p>This is the output you should get:
</p>
<pre><code>Running 'server' role. Listening on 50443, decrypting and forwarding to real server machine on localhost:50080
ssltunnel's server is listening on port: 50443</code></pre>
<p>Now let's start the client:
</p>
<p>Here we instruct the sslproxy (<em>-r client</em>) to listen on port <em>50081</em> and connect to sslserver on
host <em>localhost</em> (also the default) and port <em>50443</em>. We also provide public and private client certificates and
sslserver's public certificate.
</p>
<pre><code>c:\node_modules\ssltunnel\bin>ssltunnel.cmd
-r client
--proxy_port 50081
--server_port 50443
--server-host localhost
--srv_pub_cert ..\testcerts\sc_public.pem
--clt_pub_cert ..\testcerts\cc_public.pem
--clt_prv_cert ..\testcerts\cc_private.pem</code></pre>
<p>You should see something like this:
</p>
<pre><code>Running 'client' role. Listening on 50081, encrypting and forwarding to ssltunnel's server on localhost:50443
ssltunnel's client is listening on port: 50081</code></pre>
<p>Congrats! You have an established secure tunnel.
</p>
<h3>step 3: connecting though the tunnel</h3>
<p>Let's try to connect now. Fire up <em>cmd</em> and navigate to mongo's bing directory. Then run mongo.exe and instruct
it to connect to <em>localhost:50081</em>.
</p>
<pre><code>d:\src\mongodb-win32-x86_64-2.0.2\bin>mongo localhost:50081
MongoDB shell version: 2.0.2
connecting to: localhost:50081/test
> show dbs
local (empty)
test 0.078125GB
></code></pre>
<p>You have successfully connected to your mongo server through ssltunnel!
</p>
<h2>Epilogue</h2>
<p>Few additional words on this. In addition to the above ssltunnel enables setting TCP keep-alive between sslproxy and sslserver. This makes it possible to overcome problems with servers with low TCP timeouts. It also supports setting various logs verbosity.
</p>
<p>ssltunnel can also be used via node script. You just populate the <em>options</em> object with all the configuration details and run either <em>ssltunnel.createClient()</em> to create sslproxy or <em>ssltunnel.createServer()</em> to create sslserver. See <a href="https://github.com/anodejs/node-ssltunnel/blob/master/bin/run_ssltunnel.js">this</a> file for example (scroll to the bottom).
</p>
<p>If you use ssltunnel and missing a feature feel free to send a pull request or just ask me to do it. If you have any questions do not hesitate to contact me at [email protected]
</p>
<p>Cheers!<br><a href="https://github.com/dimastopel">Dima Stopel</a></p>
<hr><div class="comment"><a href="#" data-disqus-identifier="/#securing-mongo-with-ssltunnel" class="comment">Leave a comment</a><img src="data:image/gif;base64,R0lGODlhEAALAPQAAP///z2LqeLt8dvp7u7090GNqz2LqV+fuJ/F1IW2ycrf51aatHWswaXJ14i4ys3h6FmctUCMqniuw+vz9eHs8fb5+meku+Tu8vT4+cfd5bbT3tbm7PH2+AAAAAAAAAAAACH/C05FVFNDQVBFMi4wAwEAAAAh/hpDcmVhdGVkIHdpdGggYWpheGxvYWQuaW5mbwAh+QQJCwAAACwAAAAAEAALAAAFLSAgjmRpnqSgCuLKAq5AEIM4zDVw03ve27ifDgfkEYe04kDIDC5zrtYKRa2WQgAh+QQJCwAAACwAAAAAEAALAAAFJGBhGAVgnqhpHIeRvsDawqns0qeN5+y967tYLyicBYE7EYkYAgAh+QQJCwAAACwAAAAAEAALAAAFNiAgjothLOOIJAkiGgxjpGKiKMkbz7SN6zIawJcDwIK9W/HISxGBzdHTuBNOmcJVCyoUlk7CEAAh+QQJCwAAACwAAAAAEAALAAAFNSAgjqQIRRFUAo3jNGIkSdHqPI8Tz3V55zuaDacDyIQ+YrBH+hWPzJFzOQQaeavWi7oqnVIhACH5BAkLAAAALAAAAAAQAAsAAAUyICCOZGme1rJY5kRRk7hI0mJSVUXJtF3iOl7tltsBZsNfUegjAY3I5sgFY55KqdX1GgIAIfkECQsAAAAsAAAAABAACwAABTcgII5kaZ4kcV2EqLJipmnZhWGXaOOitm2aXQ4g7P2Ct2ER4AMul00kj5g0Al8tADY2y6C+4FIIACH5BAkLAAAALAAAAAAQAAsAAAUvICCOZGme5ERRk6iy7qpyHCVStA3gNa/7txxwlwv2isSacYUc+l4tADQGQ1mvpBAAIfkECQsAAAAsAAAAABAACwAABS8gII5kaZ7kRFGTqLLuqnIcJVK0DeA1r/u3HHCXC/aKxJpxhRz6Xi0ANAZDWa+kEAA7AAAAAAAAAAAA" class="loading"></div><div class="comments"><a href="#" class="nocomment">Hide comments</a><div class="disqus_thread"></div></div></div><hr><a name="launching-anodejsorg" class="hash"> </a><div id="entry-launching-anodejsorg" class="entry"><h1>We work at Microsoft and we use node.js</h1><p><small>Posted on Tuesday, March 20, 2012 by <a href="http://eladb.github.com">Elad Ben-Israel</a></small><br/></p><p>We recently spent some time with <a href="http://channel9.msdn.com/Blogs/Charles/anode-An-Experimental-nodejs-Platform-for-Windows-Azure">Charles Torre from Channel 9</a>, discussing node.js at Microsoft and the project we have been working on, <strong>anode</strong>.
</p>
<iframe style="height:288px;width:512px" src="http://channel9.msdn.com/Blogs/Charles/anode-An-Experimental-nodejs-Platform-for-Windows-Azure/player?w=512&h=288" frameBorder="0" scrolling="no"></iframe>
<p>We thought it would be a nice opportunity to launch our blog and share some of our experiences. Currently there are no plans to release anode as a service, but we are pleased to share the modules we have created as part of the project.
</p>
<h3>Some background</h3>
<p>Microsoft is probably the most diverse software company in the world. We build almost every type of software out there. It's amazing to witness how almost every software piece we use at the company is 100% home grown. I don't think there's any other company in the world like that: the operating systems we use on our desktops, laptops, servers and phones, the office suite, the IDE, compiler, source control, build system, issue tracking, project management, docs management, databases, our game room has Xbox and Kinect. Hell, even the phone system now uses Lync. Crazy. Inspiring. Addictive...
</p>
<p>With that in mind, when designing new systems, decisions are apperently simple: run on Windows, host on IIS, write in .NET, use WCF, source control in TFS, data on SQL and so forth. However, good engineers understand that it is important to choose the right tools for the job. When you only have one option for each part of your stack, you don't make choices and naturally you will end up with sub-optimal solutions.
</p>
<p>And there are some really good engineers at Microsoft!
</p>
<p>Luckily, one of those engineers led our team a while back. He understood that he needs to keep us on our toes and make sure we don't find ourselves in this nice and happy <a href="http://en.wikipedia.org/wiki/Not_invented_here">NIH syndrome</a> cosiness. He used to send out those emails encouraging us to play around and try new technologies and kept reminding us that we need to keep looking for the right tools, even if, god forbid, they were not created in Redmond.
</p>
<p>One of these emails was about <a href="http://nodejs.org">node.js</a>. That was 8 months ago, so the node.js community was already pretty crazy. There were about 5,000 modules at the npm repository back them (today there are over 8,000) and things have been moving fast. Two of us decided to spend a day and play around.
</p>
<h3>Not optimized for prototyping</h3>
<p>One of the pain points we had at the time was the turn-around time for publishing new code. We were doing a lot of experimentation and prototyping and the stack we were using (.NET/WCF/IIS/Azure; msbuild/mstest/TFS) practically meant a turn-around of about 2 hours:
</p>
<ol>
<li>Build and test locally using Azure dev fabric and mocks</li>
<li>Submit for the TFS build server to build and create a package</li>
<li>Upload package to azure</li>
<li>Deploy to staging</li>
<li>Verify nothing broke by running tests against staging</li>
<li>VIP-swap to production</li>
</ol>
<p>Another big pain was the fact that it took about one minute for logs to be transfered from our roles into the Azure Table, from which we needed to download them and only then figure out what went wrong.
</p>
<p>Now all this process was needed not nessesarily because we had millions of users who needed super high quality code (a lot of the stuff we did was experimental in nature). The main reason we needed all this was because of the 2h/1m turn-around. Since you couldn't really "develop on the cloud", you had to make sure things are going to work before you deployed, because once something didn't work (usually it was one of those "it all worked locally, damn it" bugs), 2 more hours went out the window...
</p>
<p>We kept trying to improve the process: reduce testing time, improve our simulators to make sure they behave like the cloud, build in parallel, aggregate changes into less deployments, use log viewers we found to monitor the system. But we were an order of magnitude away from just writing a few lines of code, see if they worked okay on the cloud and integrated well with everything else and repeatedly do that over and over. And that's how we wanted to work…
</p>
<h3>From 7,200 to 10 in one day</h3>
<p>Amazingly, after a day of work in a nice little coffee place in Tel-Aviv, borrowing ideas from <a href="http://smarxrole.codeplex.com">Smarx Role</a> and other PaaS providers, we managed to create an Azure role that "listened" on a blob account. When a blob container changed, it downloaded the code from that container, spawned <code>node index.js</code> (with an allocated <code>process.env.PORT</code>) and using <a href="https://github.com/nodejitsu/node-http-proxy">http-proxy</a>, routed incoming requests into these apps. When you went into the blob and updated one of the files, the role re-fetched the changes and respawned the app. We also grabbed <code>stdout/err</code> and pushed it almost immediately into an Azure Table. We wrote a little web app that tailed the table and showed recent logs in almost real-time.
</p>
<p>So turn-around dropped from 2 hours to 10 seconds.
</p>
<h3>Magic!</h3>
<p>Our team was pretty excited. We felt that there's a new tool in the toolbox that's worth trying out. Gradually, people started using node.js for their experiments and protoypes and hosted their apps on our nice little PaaS-like role. People were happy that they can actually write the code and run it on the cloud so quickly, and if something didn't look good, they just updated it and it's instantly published.
</p>
<p>Node.js and the ecosystem around it proved to be an incredibly friendly stack to learn and use. We found many useful node modules and a lot of high quality documentation and conversation shared openly by some awesome hackers.
</p>
<p>Today we have a team of about 30 people (located in Tel-Aviv, San Francisco and Seattle) that use node.js and host their apps on our little platform.
</p>
<p>Another coincidental development was that two other teams at Microsoft started looking at node.js seriously around the same time: <strong>(1)</strong> The folks at the developer division joined efforts with <a href="http://www.joyent.com">Joyent</a> in order to create a native Windows port for node.js (we initially used the cygwin port), so today we have node.js running and behaving beautifully on Windows; and <strong>(2)</strong> the Azure team started working on <a href="https://github.com/tjanczuk/iisnode">iisnode</a> and the <a href="http://www.windowsazure.com/en-us/develop/nodejs">Azure Node.js SDK</a>, which makes our lives so much easier running our node.js PaaS on Azure.
</p>
<p>Ever since, we added some nice improvements, but we try to keep things simple and tailored to our actual needs:
</p>
<ul>
<li>Code is automatically fetched from git and not from blob storage. Working in deltas makes so much sense in this context.</li>
<li>We deploy multiple git branches as means to isolate apps in development from production (but still want them all on the cloud).</li>
<li>We run tests against the deployed apps when we merge code to production.</li>
<li>We provide a MongoDB as a service for apps.</li>
<li>We use a fun web command line console that to interact with the system and apps.</li>
<li>We measure useful metrics for apps and provide standard request logging.</li>
</ul>
<p>Currently, there is no plan to make anode externally available as a service, but we do have a commitment to open-source as many components of the system as we can and share our experience.
</p>
<p><strong>We started this site as home to these components and we plan to provide some more context on what we do through the blog.</strong>
</p>
<p>Feel free to <a href="https://github.com/anodejs">contact us</a> if you have any questions or comments,<br><strong>The anode crew</strong></p>
<hr><div class="comment"><a href="#" data-disqus-identifier="/#launching-anodejsorg" class="comment">Leave a comment</a><img src="data:image/gif;base64,R0lGODlhEAALAPQAAP///z2LqeLt8dvp7u7090GNqz2LqV+fuJ/F1IW2ycrf51aatHWswaXJ14i4ys3h6FmctUCMqniuw+vz9eHs8fb5+meku+Tu8vT4+cfd5bbT3tbm7PH2+AAAAAAAAAAAACH/C05FVFNDQVBFMi4wAwEAAAAh/hpDcmVhdGVkIHdpdGggYWpheGxvYWQuaW5mbwAh+QQJCwAAACwAAAAAEAALAAAFLSAgjmRpnqSgCuLKAq5AEIM4zDVw03ve27ifDgfkEYe04kDIDC5zrtYKRa2WQgAh+QQJCwAAACwAAAAAEAALAAAFJGBhGAVgnqhpHIeRvsDawqns0qeN5+y967tYLyicBYE7EYkYAgAh+QQJCwAAACwAAAAAEAALAAAFNiAgjothLOOIJAkiGgxjpGKiKMkbz7SN6zIawJcDwIK9W/HISxGBzdHTuBNOmcJVCyoUlk7CEAAh+QQJCwAAACwAAAAAEAALAAAFNSAgjqQIRRFUAo3jNGIkSdHqPI8Tz3V55zuaDacDyIQ+YrBH+hWPzJFzOQQaeavWi7oqnVIhACH5BAkLAAAALAAAAAAQAAsAAAUyICCOZGme1rJY5kRRk7hI0mJSVUXJtF3iOl7tltsBZsNfUegjAY3I5sgFY55KqdX1GgIAIfkECQsAAAAsAAAAABAACwAABTcgII5kaZ4kcV2EqLJipmnZhWGXaOOitm2aXQ4g7P2Ct2ER4AMul00kj5g0Al8tADY2y6C+4FIIACH5BAkLAAAALAAAAAAQAAsAAAUvICCOZGme5ERRk6iy7qpyHCVStA3gNa/7txxwlwv2isSacYUc+l4tADQGQ1mvpBAAIfkECQsAAAAsAAAAABAACwAABS8gII5kaZ7kRFGTqLLuqnIcJVK0DeA1r/u3HHCXC/aKxJpxhRz6Xi0ANAZDWa+kEAA7AAAAAAAAAAAA" class="loading"></div><div class="comments"><a href="#" class="nocomment">Hide comments</a><div class="disqus_thread"></div></div></div><hr></div><div class="span4"><div class="sidebar-nav"><ul class="nav nav-list"><p>All our modules are available on <a href='https://github.com/anodejs'>GitHub</a> and published under the <a href='http://en.wikipedia.org/wiki/MIT_License'>MIT License</a>.</p><p>We welcome contributions and feedback!</p><div class="well"><li class="nav-header"><h3><a href="https://github.com/anodejs/node-rinuts">rinuts</a></h3></li><dl><dt></dt><p>An HTTP server which exposes unit tests via REST API.<br/>Rinuts is extensible via 'rinuts-drivers'. Currently, we maintain: <ul><li>A <a href='https://github.com/urigolani/rinuts-nodeunitDriver'>nodeunit</a> driver</li><li>An mstest (!!) driver for Windows</li></ul></p><dt></dt><p class="usage">Each app that runs on <i>anode</i> may choose to expose black box tests using <strong>rinuts</strong> (or rinuts-compatible HTTP API). This allows us to automatically run app tests against any version of the code when we integrate code into master or during development.</p><dt></dt></dl><a href="https://github.com/anodejs/node-rinuts" class="btn">Source »</a><span class="credits"><a href="https://github.com/urigolani">urig</a></span></div><div class="well"><li class="nav-header"><h3><a href="https://github.com/anodejs/node-docrouter">docrouter</a></h3></li><dl><dt></dt><p>A Connect/Express router wrapper which exposes a formatted description of the available services of a server. The documentation is available is multiple formats (Html, Wadl, Json).</p><dt></dt><p class="usage">We use <strong>docrouter</strong> to automatically create documentation and command line completion of our internal APIs and for APIs of apps created on top of <i>anode</i>.</p><dt></dt></dl><a href="https://github.com/anodejs/node-docrouter" class="btn">Source »</a><span class="credits"><a href="https://github.com/saary">saary</a>,<a href="https://github.com/gilad61">gilado</a>,<a href="https://github.com/amiturgman">amiturgman</a></span></div><div class="well"><li class="nav-header"><h3><a href="https://github.com/anodejs/node-gits">gits</a></h3></li><dl><dt></dt><p>A Windows-friendly <a href='http://git-scm.com/'>git</a> wrapper for node.js with some useful sugar.</p><dt></dt><p class="usage">Deployment on <i>anode</i> uses git and <a href='https://github.com'>github</a>. Whenever we push stuff upstream, a github post-commit trigger is sent to our system and broadcasted to all the servers. We use <strong>gits</strong> to mirror the local filesystem from the remote repository and deploy apps continuously onto the various clusters.</p><dt></dt></dl><a href="https://github.com/anodejs/node-gits" class="btn">Source »</a><span class="credits"><a href="https://github.com/gilad61">gilado</a>,<a href="https://github.com/yosefd">yosefd</a>,<a href="http://eladb.github.com">eladb</a></span></div><div class="well"><li class="nav-header"><h3><a href="https://github.com/anodejs/node-mongueue">mongueue</a></h3></li><dl><dt></dt><p>A MongoDB-based queue with some neat tricks. Uses <a href='http://www.mongodb.org/'>MongoDB</a>'s <code>findAndModify</code> to implement a scalable queue with support for timeouts, retries and cron-like tasks.</p><dt></dt><p class="usage">We use <strong>mongueue</strong> in <i>anode</i> to implement our cron/pushqueue facility. The pushqueue facility pushes HTTP request definitions into a queue and distributes them in parallel to apps.</p><dt></dt></dl><a href="https://github.com/anodejs/node-mongueue" class="btn">Source »</a><span class="credits"><a href="https://github.com/orkaplan">ork</a>,<a href="http://eladb.github.com">eladb</a></span></div><div class="well"><li class="nav-header"><h3><a href="https://github.com/anodejs/node-rebus">rebus</a></h3></li><dl><dt></dt><p>State-based intra-server pub/sub library. Listen to changes on <i>data</i> rather on <i>events</i>.</p><dt></dt><p class="usage">Our apps repository and system topology are exposed using <strong>rebus</strong> to any apps running on <i>anode</i>. Apps can monitor these structures for changes and act upon them.</p><dt></dt></dl><a href="https://github.com/anodejs/node-rebus" class="btn">Source »</a><span class="credits"><a href="https://github.com/yosefd">yosefd</a></span></div><div class="well"><li class="nav-header"><h3><a href="https://github.com/anodejs/node-spinner">spinner</a></h3></li><dl><dt></dt><p>Spin and maintain child processes with a few nice features:<ul><li>Wait for children to bind to a port</li><li>Shutdown after idle timeout</li><li>Automatically restart on file changes</li><li>Faulty state if terminates too fast</li><li>Pipe stdout/stderr</li></ul></p><dt></dt><p class="usage">This module is used by <a href='#farmjs'>farmjs</a> to spin up apps.</p><dt></dt></dl><a href="https://github.com/anodejs/node-spinner" class="btn">Source »</a><span class="credits"><a href="http://eladb.github.com">eladb</a>,<a href="http://mojodna.net">mojodna</a></span></div><div class="well"><li class="nav-header"><h3><a href="https://github.com/amiturgman/aCLI">aCLI</a></h3></li><dl><dt></dt><p>Lean and mean jQuery plugin that implements a text console with HTTP plugins, help and many more goodies</p><dt></dt><p class="usage">We use this control as the main dashboard of the anode system. Apps create plugins for management purposes and developers use the console to view logs and investigate app conditions.</p><dt></dt></dl><a href="https://github.com/amiturgman/aCLI" class="btn">Source »</a><span class="credits"><a href="https://github.com/amiturgman">amiturgman</a></span></div><div class="well"><li class="nav-header"><h3><a href="https://github.com/anodejs/node-ctxobj">ctxobj</a></h3></li><dl><dt></dt><p>Overloads javascript objects with <code>pushctx()</code> and <code>popctx()</code> functions. They can be used to create derived objects overloaded with a context stack. This module also exports a <code>console()</code> function that can be used to wrap any console-like object in context stack which will be printed as a prefix to any log line.</p><dt></dt><p class="usage">We use this library to create super readable and easy to read logs for our distributed system. Each log line contains a context stack that can be used to track down flows across processes and servers.</p><dt></dt></dl><a href="https://github.com/anodejs/node-ctxobj" class="btn">Source »</a><span class="credits"><a href="http://eladb.github.com">eladb</a></span></div><div class="well"><li class="nav-header"><h3><a href="https://github.com/anodejs/node-fsmjs">fsmjs</a></h3></li><dl><dt></dt><p>A general-purpose finite state machine module that can be used to implement complex control flows.</p><dt></dt><p class="usage"><strong>fsmjs</strong> is used by <a href='#spinner'>spinner</a> to maintain the lifetime of child processes.</p><dt></dt></dl><a href="https://github.com/anodejs/node-fsmjs" class="btn">Source »</a><span class="credits"><a href="http://eladb.github.com">eladb</a></span></div><div class="well"><li class="nav-header"><h3><a href="https://github.com/anodejs/node-ssltunnel">ssltunnel</a></h3></li><dl><dt></dt><p>A lightweight TCP over SSL / TLS tunnel running over node. If you need to add confidentiality (privacy), integrity, and authenticity to your TCP stream this is the tool for you.</p><dt></dt><p class="usage">We use <strong>ssltunnel</strong> to connect securely and transparently to our storage servers.</p><dt></dt><h4>References</h4><ul><li><a href="#securing-mongo-with-ssltunnel">Creating a secure MongoDB channel with ssltunnel</a></li></ul></dl><a href="https://github.com/anodejs/node-ssltunnel" class="btn">Source »</a><span class="credits"><a href="https://github.com/dimastopel">dimast</a>,<a href="https://github.com/yosefd">yosefd</a></span></div><div class="well"><li class="nav-header"><h3><a href="https://github.com/eladb/node-statestream">statestream</a></h3></li><dl><dt></dt><p>Quickly creates HTTP downstreams for state changes. Just plug in an in-memory object (or a function) and consumers will be updated on changes.</p><dt></dt><p class="usage">We use <strong></strong> to enable web-based real-time view of some of the system elements.</p><dt></dt></dl><a href="https://github.com/eladb/node-statestream" class="btn">Source »</a><span class="credits"><a href="http://eladb.github.com">eladb</a></span></div><div class="well"><li class="nav-header"><h3><a href="https://github.com/anodejs/node-ypatterns">ypatterns</a></h3></li><dl><dt></dt><p>Misc javascript programming patterns.</p><dt></dt><p class="usage"></p><dt></dt></dl><a href="https://github.com/anodejs/node-ypatterns" class="btn">Source »</a><span class="credits"><a href="https://github.com/yosefd">yosefd</a></span></div><div class="well"><li class="nav-header"><h3><a href="https://github.com/anodejs/node-sharp">node-sharp</a></h3></li><dl><dt></dt><p>A native node.js extension which allows loading .NET assemblies and use them within node.js.</p><dt></dt><p class="usage">We use <strong>node-sharp</strong> to access some .NET code we didn't want to rewrite and some Windows-specific facilities.</p><dt></dt></dl><a href="https://github.com/anodejs/node-sharp" class="btn">Source »</a><span class="credits"><a href="https://github.com/JettJones">jettj</a>,<a href="https://github.com/saary">saary</a></span></div></ul></div></div></div><footer>© Microsoft Corporation 2012
|<a href="rss.xml" class="feed">subscribe</a> |
built with<a href="http://twitter.github.com/bootstrap">bootstrap</a>, <a href="http://jade-lang.com">jade</a>,<a href="https://github.com/chjj/marked">marked</a></footer></div></body><script type="text/javascript" src="js/jquery-1.7.1.min.js"></script><script type="text/javascript" src="js/index.js"></script></html>