Org-server API for formats such as norg, org-mode, markdown etc

[hhoeflin]

Hi,

I like organization and code frameworks like org-mode, RMarkdown etc and I think the neorg effort is great. However, I also really like markdown and the infrastructure that exists for it (like mkdocs-material), and so I was looking for the best of both worlds.

In a way, the problem faced by org-mode being closely tied to emacs (with now a neovim clone) is similar to the issue programing IDEs had a while back, where every editor needed to natively implement parsers for every language.

Underneath it all, I think neorg should not depend on any document format, also (also not norg) and instead support any format for which an appropriate org-server and the lua-part of neorg just implements the API calls and the neovim interface.

Is this something the team here thinks about/has on the roadmap?

A lot of this work could then be extended to markdown, asciidoctor etc and they could potentially all benefit from organizing and note-taking in text format with more editor support.

Thanks and looking forward to hear your thoughts.

[hhoeflin]

I think new tooling request like #851 also go in a similar direction.

[SevorisDoe]

Speaking as one of the proponents behind Norgberg - I would not exclude that Norgberg can be so generalized that it is usable for other document standards in tems of resolving links, tag-alikes, and other structured metadata, and I have project ideas that would leverage the graph structure explicitly. Those could be ported to other languages.

However, that all comes with the caveat of the language syntax. Consider neorg attached modifiers - they can be linked onto basically any other Neorg object and attach arbitrary metadata, which is then readily parsed from the Treesitter tree and used by the macro system or other plugins. These ideas do not necessarily have ready equivalent mappings in other languages. So now we have a bunch of problems getting concepts introduced into other languages and need to define custom syntax extensions or store metadata in different ways. That will impact how to generalize tools like GTD and, so some degree, Zettelkasten, plus any tools beyond. And it may make the entire thing more fragile than need be.

I think with purposefully declaring an interest in making the (core) programmatic tools for manipulating and working with neorg files standalone native modules you could compile for any plattform and use in any other app or software, a lot has already been done for breaking the file format and core utilities away from any single app.

[hhoeflin]

Yes it would probably not be easy or needs a certain modularity in terms of what functionality is required for which set of functions. How to do it in markdown - not sure. An option may be to put information inside HTML comment tags, for certain metadata) just as initial idea.

For someone like me, I probably only want a small fraction of the functionality of a tool like neorg. Initially nothing more than review lists of open tasks, maybe by due date. For something like that, switching formats away from what I know is not convenient.

[SevorisDoe]

I can only speak for what I‘m willing to work on, but:
Since Norgberg is modular at basis, and consumes data coming in from a variety of parsing modules, with abstractions in the API anyway to permit plug-and-play utility for anyone else - and I‘m already eyeing capturing basically anything interesting you can capture from the norg files at basis so people can just use the DB cache instead of digging around in vault-wide TS trees on their own - there‘s room there to generalize. Depending on how other people plan to consume TS trees into their modules, maybe similar adapters to other languages can be written for other modules. (That depends on those contributors and maintainers)

That said, I also wouldn‘t blame anybody for not wanting to work around with Markdown. (Markdown parsing isn‘t much fun)

[hhoeflin]

Thanks for taking the time to comment on this so extensively. When I have some time I will probably try and dig deeper a bit at some point.

And yes - parsing markdown is certainly no fun. But one idea of separate servers would be that they can be written in any language, and parsers that turn markdown into ASTs exist in several of them (certainly one advantage of MD, despite its drawbacks, is that support for it is excellent). So, one could already start at a higher level than the raw MD file. i

[SevorisDoe]

Supplying a description of communalities and conflict points can certainly help a lot with navigating the uncertainties and guide design decisions for anyone else creating code!

You may also want to keep an eye on our efforts adjacent to Norgberg for defining the modular interop (probably using gRPC for rust-to-rust communications) since that is another region where abstraction design decisions will happen, which may influence this idea space.

[hhoeflin]

I looked around a bit on how such an approach could be done in Markdown, and there are 2 things I have found.

1. Attribute lists extension. https://python-markdown.github.io/extensions/attr_list/. These allow for the definition of custom attributes in html tags. Under HTML5, any attribute with 'data-' is valid and intended for javascript processing, so could be attached at will. This works for headings, paragraphs etc. This information is then contained in the exported html, but would not have a result on rendering unless specific javascript modules are provided that use this information.
2. Providing a custom extension. One example of a sophisticated one that could be adapted is e.g. https://facelessuser.github.io/pymdown-extensions/extensions/critic/. It uses tags of the form '{++ ... ++}' to proivde extended behaviour. Something like '{*** }' or variants thereof to provide context specific information or reserve later processing, markup on export etc.

Combining the two approaches could potentially provide simple but rich markup opportunities for tasks, dates etc.

I don't know enough about the internals of org-mode or neorg to know if any of this is close to sufficient, but just noting this here in case this discussion comes up again.

P.S. Important: None of these requires writing parsers from scratch, they can all be done using plugins and by modifying existing other plugins.
P.P.S. They are available in python, so could readily hook into a rich ecosystem for publishing such as mkdocs-material or executable-books.