Ease the pain of update nodejs externs

[mikicho]

I open this issue to gather the efforts to ease the pain of updating the nodejs externs, this issue came up in few places across this repo:

1. comment
2. nodejs 6.x is here #72
3. adapt to node 4.0.0 #41

and more older issues.

I think we can achieve this by doing one of the following:

• Semi-automatic solution: Create a generator that will be able to generate dump externs.
• Manual - Create a tool that will hint which functions need to be updated to match the current externs to the new node version.

Dump generator

I see two options to implement this:

1. Use node JSON docs (it seems like to have all the needed information):

Docs for fs.read():

Docs for fs.readfilesync():

2. Use the typescript definitions file

Diff tool

I create a POC tool that do that: https://github.com/mikicho/NodeDocsCrawler

Third option:

@mundusnine #103 (comment)

Use the AST from the typescript compiler.

---

P.S: I'd do all by myself, but I'm afraid that I'm not understand in Haxe enough to know what better and can't see the whole picture so I opened it up to a public discussion, I can take the implementation on me if not include too many macros :)

[Gama11]

Regarding the TS definitions, I wonder how good the results would be when running those through https://github.com/Simn/ts2hx.

[gene-pavlovsky]

It looks like a really nice effort already! Is the tool able to detect which existing entries had been deprecated/undeprecated/removed?

[mikicho]

@gene-pavlovsky Thanks! I think that is able detect deprecated: https://github.com/mikicho/NodeDocsCrawler/blob/master/src/NodeDocsCrawler.hx#L29

I don't remember about removed...
And didn't see un-deprecated things.

[dionjwa]

I'm available for work and input, I write a lot of externs, and would really like to not spend so much time maintaining. It also makes selling haxe for js much easier.

[mundusnine]

Third option:

• Use the AST from the typescript compiler.

I am working on one right now with this solution. It just takes js files and outputs to haxe( I use intermediary xml files because I want the haxe generator to be reusable for the native extern generator that I am writing).

Using the typescript definitions(if its possible) in conjunction with the typescript compiler could be the best option for hxnodejs for now. Removes the burden of having to infer types and having to handle class creation through functions :

function Apple(type) 
{   var num = 1; // Is number but considered undefined by tsc
    this.type = type;
}

My work is only in its infancy so don't get too exited though 😛

[mikicho]

@Gama11
I tried to apply ts2hx on node definition files and I get the following error when tried to convert the index.d.ts file:

PS E:\projects\haxe\neko\ts2hx> neko .\run.n tsTestFile\index.d.ts
Called from ? line 1
Called from Main.hx line 69
Called from Main.hx line 114
Called from Main.hx line 102
Called from Main.hx line 93
Called from Main.hx line 138
Called from C:\path/haxelib/hxparse/4,0,0/src/hxparse/Utils.hx line 28
Uncaught exception - ts:2479: characters 11-12: Unexpected { def => TAssign, pos => ts:characters 110475-110476 }

But it succeed to convert the inspector.d.ts file and it work not bad, although not perfect:

1. It seems like it doesn't know to handle properly with partial-application declaration:

post(method: "Debugger.enable", callback?: (err: Error | null) => void): void;
post(method: "Debugger.disable", callback?: (err: Error | null) => void): void;

Convert to

@:overload(function(method:String, ?callback:haxe.extern.EitherType<Error, Null> -> Void):Void { })
@:overload(function(method:String, ?callback:haxe.extern.EitherType<Error, Null> -> Void):Void { })

Which is ok, but not optimal.

2. I'm not sure it produces Top Level API:

// Top Level API

    /**
     * Activate inspector on host and port. Equivalent to node --inspect=[[host:]port], but can be done programatically after node has started.
     * If wait is true, will block until a client has connected to the inspect port and flow control has been passed to the debugger client.
     * @param port Port to listen on for inspector connections. Optional, defaults to what was specified on the CLI.
     * @param host Host to listen on for inspector connections. Optional, defaults to what was specified on the CLI.
     * @param wait Block until a client has connected. Optional, defaults to false.
     */
    export function open(port?: number, host?: string, wait?: boolean): void;

    /**
     * Deactivate the inspector. Blocks until there are no active connections.
     */
    export function close(): void;

    /**
     * Return the URL of the active inspector, or undefined if there is none.
     */
    export function url(): string;

I can't find the externs for the above declarations...

3. It also inserts the Session class which is top-level class into HeapProfiler namespace.

and maybe there is more...
I created a little gist with the results:
https://gist.github.com/mikicho/ca2c6a8b3cf7425fe338f64283ad2658

[mikicho]

@mundusnine why use AST is better than the json docs?

[mundusnine]

@mikicho

Well the question is more, have the JSON docs generalized use in pretty much all javascript projects ? Is generating the JSON docs painless and non-time consuming?(tried looking into documentation generation and it seems to be only a node.js thing, I am probably wrong but I can't find the info).

On my end what I am trying to do is just have one dependency aka the typescript-compiler that takes in any javascript file and can then output the needed information. The advantage is by only having source code as a prerequisite to generating haxe externs. The AST output can also have the body of a function ergo making it possible to inline the javascript in the haxe externs(I know we can do this with cpp don't know for js)

[mikicho]

@mundusnine hm.. Nice, IMHO, here we are trying to ease the pain of nodes extern specifically, so if we already have a great official json documentation , I think we can happily use it, isn't?
p.s: maybe you can take advantage of the API Extractor

[mundusnine]

@mikicho I think that what I am working on and the work you are doing are just complementary solutions. Node is pretty big and as of now the work I am doing is more haxed towards generalizing the way of making externs for Haxe, of which, the js-to-hx parser is a subset of. So my aim isn't specifically for hxnodejs but eventually could be used by it. Your solution is probably best suited for the project for now.

The API extractor is interesting, but it's basically AST outputed to a json format. The implementation I am going for just gets the data from the AST and generates the intermediary output that the externgenerator.hx will use to generate haxe code.