binance live (futures) trading support

[goodboy]

Somewhat monumental, this is our first lurker sponsored feature
request and task set :party:

Feat summary

As part of utilizing piker in a real-time event driven crypto
trading strategy (using news.treeofalpha.com) we have a request to
add live trading support in for the binance broker backend, first
for their futures market and later as well for spot markets.

There actually was work done almost 2 years ago by @guilledk in
#182 which was never brought to
full completion and testing, but a lot of the initial API
integration work was prototyped to a MVP.

Obviously a lot has changed in many subsystems since then so
there will be some adjustments needed after rebase to latest main
branch.

Required base functionality:

• live data feeds for l1 and possibly l2 books for futures:

  • https://dev.binance.vision/t/websocket-api-for-futures/14930

  • https://github.com/binance/binance-futures-connector-python

  • APIs:

    • futes ws: https://binance-docs.github.io/apidocs/futures/en/#websocket-market-streams
    • futes ex. client: https://github.com/binance/binance-futures-connector-python#websocket

  • capacity and concurrency desirables:

    • Stream aggTrade for all symbols with 50mil+ market cap.

    • If it's viable, stream all symbols.

    • If not, start streaming on news event

  • connectivity implementation requirements:

    • All interactions with the HTTP API should be done using sessions.

      • luckily we already do this implicitly using asks

    • All calls should have retry in case something fails

      • in other words retry logic around all requests such that we
        don't error on (presumably) ephemeral network outages?

    • The connection should be always open and the ping endpoint polled every
      1-2 minutes to ensure that the connection is open.

      • ping/pong support: https://github.com/binance/binance-futures-connector-python#heartbeat
      • see the

  • Function for getting +/-1.5% of liquidity from orderbook P3

    • Call this function every .1s and display it in a custom GUI

    • IOW, something which offers a predicate for how quickly
      a certain sized order will clear without causing a liquidity
      gap B)
      • use techtonicdb?
        • L2 subsystem(s) #173
        • TectonicDB support #174
      • just run a local book mirroring actor?

  • Function for getting all the data for the symbols (exchangeInfo)

  • This function should be called once every 2-3hrs to update the
    symbols in case there was a new symbol added to the exchange

  • we already have this in our current backend data layer
    but we need to do this for the futes set right?

---

• live order mgmt (futes)

  • https://www.binance.com/en/support/faq/how-to-access-mock-trading-in-binance-futures-b3706b248f2b4b1caabb4bf253bf067f

  • https://www.binance.com/en/support/faq/how-to-test-my-functions-on-binance-testnet-ab78f9a1b8824cf0a106b4229c76496d

  • Stream all user actions regarding orders and positions

    • https://binance-docs.github.io/apidocs/futures/en/#user-data-streams
      • so we can rx all trade dialog updates much like we do over
        ws in kraken backend B)
    • depending on market set (for eg. futes vs. spot) we must
      use a http API in anything that is not spot for
      requests:
      • https://binance-docs.github.io/apidocs/futures/en/#contract-info-stream

  • Write wrappers for opening a limit order, closing opened position(s), increasing/reducing position, etc.

    • we already have this via our piker.OrderClient API though
      we need to add a close_all() helper which brings the dst
      asset holding / balance to net-zero.

  • Opening the positions should be in USDT

    • Increasing and decreasing the position should be in both
      USDT and contract

    • IOW, dolla_size based slot allocation as is already supported in
      our order mode pane but exposed through an API B)

  • Order is considered as open if it's received from the user stream P2

    • Order status is updated from the user stream P2

    • There can be a situation where SL is hit during the opening of a limit order

    • In this case, the remaining order should be canceled first
      and then the position closed

      • IOW, no stale position should be left when a stop loss is
        triggered.
      • If we're going to implement stops using the existing
        dark orders from our EMS we're going to need to add
        an algorithm for the stop's limit-price and/or offer
        simply stop-market submissions.

---

• live position tracking

  • Function for getting the current balance

  • It can be either data from P2 or run in a separate thread and fetched
    from the exchange after every open/close position event.

  • IOW, balance tracking which, again, we're more or less
    already doing in piker.accounting and friends 🏄

  • Function to get price percentage change from P1

    • IOW, realtime pnl tracking and display, again we have in
      piker.accounting and friends 🏄

---

This requirements section is the future nice to have / end goal
requirements and will likely require more coordination for testing
and design with the lurker sponsor 😉

• real-time ticker/symbol/market name parsing and dynamic search,
  load, and cache of real-time data feeds from first section above.

  • For each symbol received from news.treeofalpha.com, immediately send
    the symbol so the code can prepare (i.e., get the price before the
    news from P1 to calculate price percentage change).

---

binance resources:

• dev forum: https://dev.binance.vision/

• high level FAQ:

  • "https://www.binance.com/en/support/faq/api-frequently-asked-questions-360004492232"

• futures:

  • APIs:

    • "https://binance-docs.github.io/apidocs/futures/en/#symbol-price-ticker"
    • "https://dev.binance.vision/t/websocket-api-for-futures/14930/2"
    • common questions: https://dev.binance.vision/t/api-frequently-asked-questions/37

  • testnet:

    • https://www.binance.com/en/support/faq/how-to-access-mock-trading-in-binance-futures-b3706b248f2b4b1caabb4bf253bf067f
    • all forum search:
      https://dev.binance.vision/search?q=testnet
    • https://dev.binance.vision/t/testnet-pairs-do-not-support-quoteorderqty/9677
    • https://dev.binance.vision/t/testnet-trading-volume/16140

  • orders:

    • https://www.binance.com/en/support/faq/crypto-derivatives?c=4&navId=4#18-62

  • understanding different future type classes: COIN-margined vs. USD-margined:

    • intro: https://www.binance.com/en/support/faq/crypto-derivatives?c=4&navId=4#18-36
    • detailed breakdown: https://www.binance.com/en/support/faq/what-are-usd%E2%93%A2-margined-futures-and-coin-margined-futures-85eac2bba0b342819122dc9bd4745e9b

• spot api (which has ws ctl 💥):

  • testnet help: https://testnet.binance.vision/
  • "https://binance-docs.github.io/apidocs/websocket_api/en/#place-new-order-trade"
  • "https://t.me/binance_api_english"
  • "https://github.com/binance/binance-spot-api-docs/blob/master/user-data-stream.md#general-wss-information"

• history endpoints:

  • "https://www.binance.com/en/landing/data"
  • https://github.com/binance/binance-public-data/blob/master/python/download-trade.py
  • "https://github.com/binance/binance-public-data/"
  • "https://github.com/binance/binance-public-data/tree/master/python"
  • 1w OHLC B0
    https://binance-docs.github.io/apidocs/websocket_api/en/#klines

• recent drama (for fun):

  • "https://concoda.substack.com/p/is-binance-the-next-domino-to-fall?publication_id=3892&post_id=93751490&isFreemail=true",

---

Detailed impl TODO:

• bring Binance updates #182 up to date

  • rebase to master
  • Binancial secs: futes for the peepz! #520

• reorg .brokers.binance into subpkg like other backends

  • 7231731, 4392de6, f7686bb

• add futures ws live feeds

  • exchange info lookup / search
    • should only require additional FutesPair type for handling
      mkt info without having to change Client interface?
      • see 3b66efc,
    • probably we need a special error for when there exists no
      futures contract for a pair?

• finish full emsd based order control support for futures
  and spot:

  • testnet setup: https://www.binance.com/en/support/faq/api-frequently-asked-questions-360004492232

  • we can use a reqid to tie oid from EMS directly to binance
    dialogs B)
    https://github.com/binance/binance-futures-connector-python#request-id

  • implement rate limiting logic to avoid bans client side:

    • https://binance-docs.github.io/apidocs/futures/en/#limits
    • https://www.binance.com/en/support/faq/api-frequently-asked-questions-360004492232

much more to come.. !

---

testing TODO:

• tests for round trip latency of live limit submissions
• stop loss via dark orders
• stop loss via binance API submission
• connectivity resilience under network outage(s)
  • particularly dumping a pp if/when network returns

(maybe) much more to come here!