vbt accuracy versus event driven libs like Zipline?

[jb33k]

I'm currently evaluating different backtesting libraries and vectorbt looks incredibly interesting. However, whenever you research backtesting tools, you hear warnings against vectorized backtesting tools due to them not simulating realistic trade execution, so I was wondering how does vectorbt handle the trade execution, liquidity, slippage, etc.? I checked the docs but didn't find anything about this.

[polakowo]

@jb33k although vectorbt has 'vector' in its name, it doesn't actually do any vectorized backtesting - it follows a sequential approach just like most other libraries. But in contrast to them, vectorbt can accept inputs as arrays, broadcast them in a smart way, and do the backtesting by going row by row, column by column. You can still simulate in an event-driven mode using Portfolio.from_order_func, for example, which gives you the ability to define an arbitrary logic for each time step and have the view of the past data only.

The main limitations are the following:

• no proper order management - once you issue an order command, it gets executed/rejected immediately (just like a market order, but you can easily implement your own logic for limit and stop orders). This means order execution is state-less - it simply receives a command with inputs and gives you the result of the execution including the new cash balance and other metrics. There is no order queue or anything like that, otherwise, it would reduce performance drastically.
• only one order command within a bar - although this can be easily expanded
• shorting is implemented but (advanced) margin and futures trading are not
• you need good knowledge of NumPy otherwise you will get burned hard by misplacing a single element anywhere in your code + Numba at its vanilla settings give you neither warnings nor errors if you access the wrong element of an array (you must set environment variables to enable this)

vectorbt has some room to grow to compete with other backtesters in terms of functionality, but it clearly outperforms them in terms of speed, extensibility, and the number of analysis tools. Plus its backtesting engine is becoming more and more validated against other frameworks such as backtrader (as demonstrated here), it just requires a slightly different way of thinking.

[shiosai]

@polakowo

no proper order management - once you issue an order command, it gets executed/rejected immediately (just like a market order, but you can easily implement your own logic for limit and stop orders). This means order execution is state-less - it simply receives a command with inputs and gives you the result of the execution including the new cash balance and other metrics.

I'm no expert at all but e.g. coming from backtrader is not one typical (perhaps just example) use case to do some kind end-of-day trading? E.g. making a buy decision for the next day based on all information available after "close"? Backtrader would not buy the next day in case of a price jump (for open?) but vectorbt would already have "bought" the same day "after close"?

I have to admit, a lot of points are not clear at all for me at the moment but the project looks really awesome!

[polakowo]

@shiosai there is no built-in mechanism that would prevent you from cheating such as executing on/after close or whatever. You're responsible for checking and supplying the correct trade information at each tick. For example, to avoid generating the signal and executing it at the same tick (which is unrealistic but most examples do it anyway for simplicity) you can simply shift the order array one tick forward, such that each order is based on the previous close, leading to the same result as in backtrader.

Regarding the after-the-close checks, you can develop ones yourself - there are callbacks that can be called between ticks to do exactly this. There is potentially a lot of stuff that can be automated, but you always need to take into account the tradeoff between flexibility, user-friendliness, and performance.

No worries, the learning curve is sometimes steep, but it makes a lot of fun once you know what you're doing.