Binancial secs: futes for the peepz! #520

@guilledk

First draft originally by @guilledk but update by myself 2 years later xD. Will crash at runtime but at least has the machinery to setup signed requests for auth-ed endpoints B) Also adds a generic `NoSignature` error for when credentials are not present in `brokers.toml` but user is trying to access auth-ed eps with the client.

Instead of having a buncha logic branches for 'get', 'post', etc. just pass the `method: str` and do a attr lookup on the `asks` sesh. Also, adjust the `trades_dialogue()` ep to switch to paper mode when no client API key is detected/loaded.

@guilledk

Patch again originally from @guilledk and adds a sesh for futures testnet as well as a order canceller method B)

@guilledk

From @guilledk, - Drop Decimal quantize for now - Minor tweaks to trades_dialogue proto

…ce/quantity

Add the usd-futes "Pair" type and thus ability to load all exchange (info for) contracts settled in USDT. Luckily we don't seem to have to modify anything in the `Client` interface (yet) other then a new `.mkt_mode: str` which determines which endpoint set to make requests. Obviously data received from endpoints will likely need diff handling as per below. Deats: - add a bunch more API and WSS top level domains to `.api` with comments - start a `.binance.schemas` module to house the structs for loading different `Pair` subtypes depending on target market: `SpotPair`, `FutesPair`, .. etc. and implement required `MktPair` fields on the new futes type for compatibility with the clearing layer. - add `Client.mkt_mode: str` and a method lookup for endpoint parent paths depending on market via `.mkt_req: dict` Also related to live feeds, - drop `Struct` typecasting instead opting for specific fields both for speed and simplicity atm. - breakout `subscribe()` into module level acm from being embedded closure. - for now swap over the ws feed to be strictly the futes ep (while testing) and set the `.mkt_mode = 'usd_futes'`. - hack in `Client._pairs` to only load `FutesPair`s until we figure out whether we want separate `Client` instances per market or not..

Not sure why it seemed like futures pairs didn't have this field but add it to the parent `Pair` type as well as drop the overridden `.price/size_tick` fields instead doing the same as in spot as well. Also moves the `MarketType: Literal` (for the `Client.mkt_mode: str`) and adds a pair type lookup table for exchange info loading.

The beginning of supporting multi-markets through a common API client. Change to futes market mode in the client if `.perp.` is matched in the fqme. Currently the exchange info and live feed ws impl will swap out for their usd-margin futures market equivalent (endpoints).

Since there are indeed multiple futures (perp swaps) contracts including a set with expiry, we need a way to distinguish through search and `FutesPair` lookup which contract we're requesting. To solve this extend the `FutesPair` and `SpotPair` to include a `.bs_fqme` field similar to `MktPair` and key the `Client._pairs: ChainMap`'s backing tables with these expanded fqmes. For example the perp swap now expands to `btcusdt.usdtm.perp` which fills in the venue as `'usdtm'` (the usd-margined fututes market) and the expiry as `'perp'` (as before). This allows distinguishing explicitly from, for ex., coin-margined contracts which could instead (since we haven't added the support yet) fqmes of the sort `btcusdt.<coin>m.perp.binance` thus making it explicit and obvious which contract is which B) Further we interpolate the venue token to `spot` for spot markets going forward, which again makes cex spot markets explicit in symbology; we'll need to add this as well to other cex backends ;) Other misc detalles: - change USD-M futes `MarketType` key to `'usdtm_futes'`. - add `Pair.bs_fqme: str` for all pair subtypes with particular special contract handling for futes including quarterlies, perps and the weird "DEFI" ones.. - drop `OHLC.bar_wap` since it's no longer in the default time-series schema and we weren't filling it in here anyway.. - `Client._pairs: ChainMap` is now a read-only fqme-re-keyed view into the underlying pairs tables (which themselves are ideally keyed identically cross-venue) which we populate inside `Client.exch_info()` which itself now does concurrent pairs info fetching via a new `._cache_pairs()` using a `trio` task per API-venue. - support klines history query across all venues using same `Client.mkt_mode_req[Client.mkt_mode]` style as we're doing for `.exch_info()` B) - use the venue specific klines history query limits where documented. - handle new FQME venue / expiry fields inside `get_mkt_info()` ep such that again the correct `Client.mkt_mode` is selected based on parsing the desired spot vs. derivative contract. - do venue-specific-WSS-addr lookup based on output from `get_mkt_info()`; use usdtm venue WSS addr if a `FutesPair` is loaded. - set `topic: str` to the `.bs_fqme` value in live feed quotes! - use `Pair.bs_fqme: str` values for fuzzy-search input set.

As just added for binance move to using an explicit `.<venue>.kraken` style for spot markets which makes the current spot symbology expand to `<PAIR>.SPOT` from the new `Pair.bs_fqme: str`. Reasons for why are laid out in the equivalent patch for binance. Obviously this also primes for supporting kraken's futures venue APIs as well 🏄 https://docs.futures.kraken.com/#introduction Detalles: - add `.spot.kraken` parsing to `get_mkt_info()` so that if the venue token is not passed by caller we implicitly expand it in. - change `normalize()` to only return the `quote: dict` not the topic key. - rewrite live feed msg loop to use `match:` syntax B)

Also add a lookup helper for getting addrs by venue: `get_api_eps()` which returns the rest and wss values.

Since we need them for accounting and since we can get them directly from the usdtm futes `exchangeInfo` ep, just preload all asset info that we can during initial `Pair` caching. Cache the asset infos inside a new per venue `Client._venues2assets: dict[str, dict[str, Asset | None]]` and mostly be pedantic with the spot asset list for now since futes seems much smaller and doesn't include transaction precision info. Further: - load a testnet http session if `binance.use_testnet.futes = true`. - add testnet support for all non-data endpoints. - hardcode user stream methods to work for usdtm futes for the moment. - add logging around order request calls.

Untested fully but has ostensibly working position and balance loading (by delegating entirely to binance's internals for that) and an MVP ems order request handler; still need to fill out the order status update task implementation.. Notes: - uses user data stream for all per account balance and position tracking. - no support yet for `piker.accounting` position tracking. - no support yet for full order / position real-time update via user stream.

Since the request handler task will work concurrently across venues (spot, futes, margin) we need to be sure that we look up the correct venue to update the order dialog and this is naturally determined by the FQME-style symbol in the `BrokerdOrder` msg; the best way to map that symbol-key to the correct venue/`Pair` is by using said `._pairs: ChainMap`. Further, handle limit order errors by catching and relaying back an error response to the EMS. Fix the "account name" to be `binance.usdtm` so that we can eventually and explicitly support all venues by name.

For crypto derivatives (at least futes), yes they are margined, but generally not around a single unit of vlm (like equities or commodities futes) so don't pre-set the order mode allocator to use a #unit limit, $limit is fine.

… `int` one..

…ream msgs

…requests

Was using the wrong key before from our old code (not sure how that slipped back in.. prolly doing too many git stashes XD), so fix that to properly match against order update events with 'ORDER_TRADE_UPDATE'. Also, don't match on the types we want to *cast to*, that's not how match syntax works (facepalm), so we have to typecast prior to EMS msg creation / downstream logic. Further, - try not bothering with binance's own internal `'orderId'` field tracking since they seem to support just using your own user version for all ctl endpoints? (thus we only need to track the EMS `.oid`s B) - log all event update msgs for now. - pop order dialogs on 'closed' statuses. - wrap cancel requests in an error handler block since it seems the EMS is double sending requests from the client?

There was one trick which was that it seems that binance will often send the account/position update event over the user stream *before* the actual clearing (aka FILLED) order update event, so make sure we put an entry in the `dialogs: OrderDialogs` as soon as an order request comes in such that even if the account update arrives first the `BrokerdPosition` msg can be relayed without delay / order event order considerations.

Only a couple tweaks to make this work according to the docs: https://binance-docs.github.io/apidocs/futures/en/#modify-order-trade - use a PUT request. - provide the original user id in a `'origClientOrderId'` msg field. - don't expect the same oid in the PUT response. Other broker-mode related details: - don't call `OrderDialogs.add_msg()` until after the existing check since we want to check against the *last* msgs contents not the new request. - ensure we pass the `modify=True` flag in the edit case.

As you'd expect query and sync the EMS with existing live orders reported by the market venue by packing them in `Status` msgs and sending over the order dialog stream before starting the handler tasks. XXX CAVEAT: - there appears to be no way (at least on the usdtm market/venue) to distinguish between different contracts such as perps vs. the quarterlies? - for now we just assume that the perp is being used since there's no indicator otherwise in the 'symbol' field? - we should maybe open an issue with the futures-connector project to see how they'd recommend solving this discrepancy?

It's finally a decent little design / interface and definitely can be used in other backends like `kraken` which rolled something lower level but more or less the same without a wrapper class.

Do parsing of the `'symbol'` and check for an `_<expiry>` suffix, in which case we re-format in capitalized FQME style, do the `Client._pairs[str, Pair]` lookup and then send the `Pair.bs_fqme` in the `Order.fqme: str` field.

Use dynamic lookups instead by mapping to the correct http session and endpoints path using the venue routing/mode key. This let's us simplify from 3 methods down to a single `Client._api()` which either can be passed the `venue: str` explicitly by the caller (as is needed in the `._cache_pairs()` case) or falls back to the client's current `.mkt_mode: str` setting B) Deatz: - add couple more tables to suffice all authed-endpoint use cases: - `.venue2configkey: dict[str, str]` which maps the venue key to the `brokers.toml` subsection which should be used for auth creds and testnet config. - `.confkey2venuekeys: dict[str, list[str]]` which maps each config subsection key to the list of venue name keys for doing config to venues lookup. - always build out testnet sessions for spot and futes venues (though if not set the sessions obviously won't ever be used). - add and use new `config.ConfigurationError` custom exceptions when api creds are missing. - rename `action: str` to `method: str` in `._api()` since it's the proper ReST term and switch what was "method" to be `endpoint: str`. - mask out `.get_positions()` since we can get that from a user stream wss request (and are doing that). - (in theory) import and use spot testnet url as necessary.

…it flag set

Drop the older `dict[str, ChainMap]` prototype we had since the new `OrderDialogs` built-out while adding `binance` order support is more refined and general. Also, handle new and now expect `.spot` venue token in FQMEs since kraken too has futes markets that we'll likely want to support eventually.

Since crypto backends now also may expand an FQME like `xbteur.kraken` -> `xbteur.spot.kraken` (by filling in the venue token), we need to use this identifier when looking up per-market order dialogs or submitting new requests. The simple fix is to simply look up that expanded from from the `Feed.flumes` table which is always keyed by the `MktPair.fqme: str` - the expanded form.

After #520 we've moved to better supporting explicit venues for cex backends which is important where a provider offers both spot and derivatives markets (kraken, binance, kucoin) and we need to distinguish which is being traded given a common asset pair (eg. BTC/USDT). So, make this work for `kraken`'s brokerd such that requests and pre-existing live order are (un)packed to/from EMS messaging form.

We might as well start standardizing on `brokerd` init such that it can be used more generally in client code (such as the `.accounting.cli` stuff). Deats of `broker_init()` impl: - loads appropriate py pkg module, - reads any declared `__enable_modules__: listr[str]` which will be passed to `tractor.ActorNursery.start_actor(enabled_modules=<this>)` - loads the `.brokers._daemon._setup_persistent_brokerd As expected the `accounting.cli` tools can now import directly from this new location and use the common daemon fixture definition.

Should be the final production backend to switch this over B) Also tidy up the `update_and_audit_msgs()` validator to log vs. raise when `validate: bool` is set; turn it off by default to avoid raises until we figure out wtf is up with ib ledger processing or wtv..

Like you'd think: - `load_ledger()` -> ._ledger - `load_accounrt()` -> ._pos Also fixup the old `load_pps_from_ledger()` and expose it from a new `.accounting.cli.disect` cli cmd for trying to figure out why pp calcs are totally mucked on stupid ib..

Allows for tracking paper engine orders despite the ems not necessarily being opened by the current order mode instance (UI) in "paper" execution mode; useful for tracking bots/strats running against the same EMS daemon.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Binancial secs: futes for the peepz! #520

Binancial secs: futes for the peepz! #520

Commits on Jun 27, 2023

Binancial secs: futes for the peepz! #520

Are you sure you want to change the base?

Binancial secs: futes for the peepz! #520

Commits on Jun 27, 2023