-
Notifications
You must be signed in to change notification settings - Fork 0
bullet log
Notes / Tasks (.) / Events (o) / Thoughts (-)
Collections:
It’s nice that in Clojure you can say (-> foo :a :b :c) takes advantage of -> adding parens if you don’t provide any cq would currently need (| foo (get :a) (get :b) (get :c)) So, should keywords eval to themselves (as do numbers, strings, etc.) or look themselves up in a map? We could change the default to lookup and require quote to eval to self: (| foo :a :b :c) and (| {‘:a 10, ‘:b 20} (get ‘:a)) bleh That’s ugly – try to make in context-sensitive
jq complains. Our new nav-based approach does a pretty useless degenerative result: (cq/eval 42 (cq/with-refer-all [cq] (modify 43 .))) ;; => (42) If we want to complain, we could check in:
- modify: as soon as we invoke the nav-mf, assert all items are navs
- or have the Object INavigation complain, but have nav-get (and all other nav ctors) check at construct time that their parent implements INavigation and wrap it in a terminal nav if not.
- In both cases, how to detect if an object is an nav or not when all objects have a default impl.
- Maybe use chart (empty vector is not a nav), or some new protocol method like `is-root?`?
Maybe don’t complain (Clojure complains on empty vectors for update, and we hate it), and for jq-compatiliby wrap modify in jq-modify that checks the naviness (still need to choose how)
Describing this to kids via Minecraft analogy Tension between parts and wholes oo says make an object (parts are hidden, “enapsulated”) functional programmers say it’s all data (you can see the data parts) jq/cq give us a different way to describe partial-wholes: compare to lazy seq pipelines and transducers?
only way to collect a stream is with array syntax, thus also you can’t create arrays with combinatorics
there is a symmetry between span and collect that you can’t see because jq’s collect [] is not multi-arity
“jq is powerful – what if we had that power in Clojure?”
a[0][1][2] a[0][1][2] = 42
(get-in a [0 1 2]) (assoc-in a [0 1 2] 42)
o Discuss talk . remove synonyms so that | & and collect are the only three names o There is a relationship between | and & in how they cause things to be called multiple times: (& a b c) (| a (& 1 2) c)
(f (g x))
;; These are the same for auto-lifted g/f: (| x (g .) (f .)) (f (& (g (& x))))
;; But not for stream-aware fns like collect: (| (& 1 2 3) (collect .)) ;=> [1] [2] [3] (collect (& 1 2 3)) ;=> [1 2 3] (collect ^:$ (& 1 2 3)) ;=> [1] [2] [3]
(^:not-stream-aware collect (& 1 2 3)) This may be why they both “feel” right as punctuation.
(& (a .) (b .)) (?mc (& a b) .)
(comment (| x (g .) (f (collect .))) (mapcat #(f (collect %)) (mapcat g x)))
What if the “stream-aware” were per-parameter metadata? But that’s just `collect` on that parameter (or nav-aware collect), but beware navs embedded in deep data. Then pipe would be doing the ->-like rewriting, but nothing else special – you could use mapcat to get the same beahvior yourself. o Potential axis for grouping functions: stream-aware / not nav-aware / not macro / fn paired operations inverse: collect/each update: modify/assign navigate: pick/slice unpaired: path consumes dot (modify, assign) / not rebinds dot (modify, |) / not
o Initial talk outline o discuss let (stream-aware local variables??) #_#_ ;; This is not right: let* (let [[bindings & body] args] `(let* [~@(apply concat (for [[sym valform] (partition 2 bindings)] [sym (expand-form valform)]))] ~@(map expand-form body))) Dynamic metadata option. Change leaf case of go macro to check metadata on list to see if it’s supposed to be a stream or not. `(let [x# ~form] (if (-> x# meta :stream?) x# (with-meta (list x#) {:stream? true})) o Discussed functions inside of our magical cq land. One open question is: how should we support conveninent cq style functions.
. write a nav-pick that works on vectors, lists, lazy seqs, maps, sets, etc. . change each to use nav-pick
- implement collect in monadland, banish cq/first DONE
- change expand-form to understand ^:cq/nav DONE
- make (go (mapcat #(+ % 10) (collect (& 1 2 3)))) work DONE
- If we are going to make regular clojure functions like first, take, etc interoperate using (collect), maybe collect could get a symbol like (|) (pipe) and (&) (span)? For example: ‘$’.
. make explict wordy versions of (|) and (&). Useful for teaching?
- Hypothesis: & concat is a relic of jq, and our primitive should do combinations producing a stream. Example: (& (& 1 2) (& 3 4)) => (1 2 3 4) or .... nothing. I got nothing. This a dumb hypothisis.
- Hypothesis: ^:stream-aware (etc.) metadata only makes sense on functions, not macros. Macros must be either agnostic or aware of all the things and handle them properly.
o Attempted to make macro | just rebind ., with no mapcat Found that this then requires many other things to do it themselves. Default lifted fns are fine, as they use combinations on args, possibly including . but it doesn’t really matter. However, & needs to mapcat (see t3). and.. others? I think? I forget. What was the problem?
- Ah, but & doesn’t just need to mapcat, it also needs to be a macro so it can “see” the implicit . arg. Surely better if things that can be functions are, and there are fewer macros.
- Ok, we’re not sure this is quite right. But we believe that switching | back to non-mapcat will only effect a few list-aware functions (&, modify, … ?) So we’re postponing this decision for now – maybe later we can add another metadata flag to tell go* to pass in . as a first arg so that & can be a mapcatting fn and not a macro.
o With auto-lift, why do we want to collide/overwrite Clojure’s get? To avoid explaing why all other Clojure word are autolifted by get is replaced, we can:
- provide nav/nth and nav/get with Clojure-like semantics for each
- provide fancy short new nav fn that works across lists and maps; new semantics, new short name, “feels” like jq but doesn’t have to follow slavishly as we can provide jq-get to do that.
o Why a deep code-walking macro?
- Autolift all Clojure fns
- Our own functions can more clearly say what they know about (nav vs list) and let the macro fix anything they don’t understand (piecemeal auto-lifting)
- Such a macro has an opportunity to do some post-processing of the form tree to optimize away redudant code
- Output of a macro may be easier to understand than a tree of monadic fns (without extra emit machinery)
o Disussed optimizer. Agreed that we’ll start with emiting real Clojure forms, and attempt to optimize that as a late step. But if we find we want to add extra info to the emitted data, we should switch to an AST of our own, and emit to forms later.
o Is “invoke” more like “read” than “eval”? but read is char-stream->structured data, so not quite. Maybe more like “macroexpand” or “expand”, in that it is structure->structure? o Options for modify:
- take 3 args, whoo! unusable power! Keep things separate for now – easy to provide ease later But this is really ugly to use, and may provide opporutiny for weird errors
- take 2 args, use implicit threaded arg as input to digger Why not? Because maybe there’s benefit to having no exceptions to the simple rule of . is only used when you say it.
- make modify a macro that does crazy things (basically 2 in macroish)
- take 2 args, use the root of the first arg nav as the seed of the reduce
Can’t seem to exract root using modify, unless we provide a special value
for acc, so maybe just provide a new method on the nav protocol instead
- a problem with this is t37: (modify (select . (> . 0)) #(* % 2)) That is, when select wants to remove something, it returns an empty stream, leaving modify nothing to find the root of. I suppose we could have navs play in list-monads; this would allow a each-lens and an empty-nav (the latter would have an origin, but navigate to an empty stream)
- take 2 args, use the root of the first nav (like 4) unless there are no navs (the navs didn’t explode and require reduce, but instead collapsed to nothing) … now we have no nav to get an origin of so we peek at . after all. :-(
We don’t have the most principled argument against 1, but it sure looked bad in our tests. So we choose 2/3, basically the same jq. o Discussed how modify could add another link in the nav chain, until `navigate` is called o jq is about modify. modify is about reducing across a stream of navs, apply an update function to the navigate of each and lens-putting the result into the implicit arg. If you take list-monad behavior out of modify, you end up with modify* which takes 3 args, one of them a scarily-arbitrary base, and does just one update (no reduce) o If your only have | (no ->) you would be tempted to use it for everything
o Discussed collect-into vs collect. Latter would return a sequential (lazy sez? vector?) (cq/do (into {} (collect (& [10 20] [30 40])))) o Discussed implicit arg again, found only: get, slice, all. They all return navs! We could stack using Clojure’s ->, to keep our purity but have the stacking feeling of jq navs Wrote t63 o jq-compat + jq-compiler top level emits clojure code that passes all jq tests sister layer cq convenience layer: and lens library transliteratable between jq and cq o lens library (protocol) o clojure-compatible lenses (vectors, maps, sets) o jq-compatible lenses o mapcat augment library: deep-walking macro, auto-lift, combinations, collect o “If jq were in Clojure, what would it look like?” Why not use jq syntax, like regex? Running thread of things where syntax pushed one solution (or confusion) where we can consider alternatives arrays had to flatten (because comma was stream and also array constructor) infix modify hides the fact that the left-argument must fully describe the navigation What’s so great about mapcat? power of map, filter, concat o Tension between explicit navigate (wordy, no magic) and auto-navigate (applied where? magic complected into list-monad macro)
o Discussed the flattening behavior of sequences in [vector] constructors o Do we want to lift vector to get combinatorics and multiple returns? And leave [] to act like jq (ie. [& …]) jq carefully documents the “collecting” behavior of arrays, but doesn’t call out that you have lost the ability to return multiple vectors. The language needs both exploding (jq’s `combinations`) and collecting (jq’s []) How do we want to provide these? We could expose combinations and keep our [] like jq We could expose list and/or vector for exploding (aligns well with only-explicit .) and for collecting… …something like into? `collect-into` …but no way to collect into a lazy-seq . Make [] act like lifted vector (abandon jq) . Write collect-into (ignore lazy-seqs for now); make combinations from target collection? o Discovered jq’s `combinations` o Discussed, again, the possiblity of having none of our cq functions directly use the threaded value. Still pass it through to arg mfs. Are there any examples where it’s really important to follow jq here? We haven’t thought of any. Got one! You don’t have to define behavior of a stream arg because threaded value can’t be a stream . update all fns and the jq-compiler to abandon all implicit threaded args . fix t61 o Docs should be clear about which args are used for navs vs not
o Discussed names of invoke (), eval, navigate o Reviewed assign o Discussed Callable keywords o Discussed value of pairing vs code reviews o Discussed bullet journaling method o How shall we communicate about this project? o Discussed modify on non-paths o Discovered jq is inconsistent in cartesian-product order: slice vs + vs {} ??? so, we need a cartesian-product-> and cartesian-product-<-, if we want to follow jq