Evaluation: Performance Test

Performance is the key in many numerical application, so I did one initial evaluation of Owl today. Frankly, I have been very busy in building up the whole system without spending too much time in optimising the performance, hence I was not sure how well Owl can perform. However, the initial results seem very promising. This encourages me to keep developing Owl and further overall optimisation in the future.

In the evaluation, I focus on the performance of operations on n-dimensional arrays. I used this version of Owl, and I compare to Numpy (version 1.8.0rcl) and Julia (version 0.5.0). The evaluation is done on my MacBook Air (1.6GHz Intel Core i5, 8GB memory). Last, note that the evaluation is performed on 2016.12.13.

Evaluated Operations

I evaluate eight operations with the detailed information listed as below. Each operation is performed 10 times and the average time is reported. All the ndarrays used in the evluation are of the shape 10 x 1000 x 10000, therefore 100 million elements in each.

• empty : create an empty ndarray of the shape 10 x 1000 x 10000 without initialising the elements.
• create : create an empty ndarray then initialise all the elements to 3.5
• x + y : add two ndarrays element-wise (both have the same shape mentioned before).
• x * y : multiply two ndarrays element-wise.
• x + 2 : add a constant 2 to all the elements in a ndarray.
• abs x : calculate the absolute value of each element in a ndarray.
• map x : apply a user-defined f function to each element and save the result in a new ndarray. In our case, f(x) = sin(x) + 1.
• iter x : iterate each element in a ndarray and some operation. Herein, we only check if the element is positive or negative.

Note that most operations will generate a new ndarray for saving the results except iter x.

Evaluation Results

	Owl (OCaml)	Numpy (Python)	Julia (Julia)
empty	0.0000	0.0000	0.0001
create	0.4051	0.4155	0.4874
x + y	0.5402	0.5698	0.7514
x * y	0.5330	0.5963	0.8649
x + 2	0.4791	0.5246	0.6299
abs x	0.4956	0.5186	0.5932
map x	2.2181	51.4562	2.2582
iter x	0.4429	37.6902	6.4385

Caveats & Further Investigation