LUA memory optimization tips & tricks

[JimB40]

Hello guys,

Last month I've spent to research LUA scripts memory optimisation.
For simple print("Hello world") it doesn't matter but once you start to write something complex like my ToolBox or @yaapu Telemetry script you will soon hit memory cap which is roughly around 60kB on BW LCD radios.

There are always trade-offs but if memory usage is priority you have to consider them
So here are few tips and findings:

1. Amount of code matters if you are going to publish non-compiled version (*.lua).

Obvious but still worth mentioning

local  a = 2
local  b = 'testVar'

for k,v in pairs(T.G) do
  b = M.S[v]
end

is more mem-efficient than

local  thisIsVariableA = 2
local  thisIsVariableB = 'testVar'

for key,value in pairs(TABLE.GLOBAL_VARIABLES) do
  thisIsVariableB = MODEL_VARIABLES.SENSORS[v]
end

TRADE-OFFs: readability
LUA code maintenance is harder

2. Keep associative arrays' keys short

local sensors = { s1 = 12, s2 = 3.8, s3 = 99}

is more mem-efficient than

local sensors = { RSSI = 12, RxBt = 3.8, RQly = 99}

TRADE-OFFs: readability

3. Functions are mem-expensive

local display = function (x,y,name,value)
  lcd.drawText(x,y,'name,SMLSIZE)
  lcd.drawText(x+40,y,'value,SMLSIZE)
end

display(10,20,'Test',5)

is more mem-efficient than

local displayName = function (x,y,name)
  lcd.drawText(x,y,'name,SMLSIZE)
end

local displayValue = function (x,y,value)
  lcd.drawText(x,y,'vailue,SMLSIZE)
end

displayName(10,20,'Test')
displayValue(50,20,5)

Whenever possible gather code in bigger one function instead of scattering it among several small ones

4. Clean unused allocated memory with collectgarbage()

Lua uses automatic memory management (garbage collector)
If you declare variable via local statement it will be auto-purged once exiting scope of this variable

But what if you hold variables,functions or other tables in tables?

local INPUTS = {}
INPUTS.SWITCHES { 'sa', 'sb', 'sc'}
INPUTS.SLIDERS = {'ls', 'rs'}

It's good practice to force garbage collector

INPUTS.SLIDERS = nil
collectgarbage()

5. Garbage collection right way

Well it is well know LUA garbage collection is not top-class :) So

To be sure everything is purged use explicit clear function (recursive) before collectgarbage()

local tableClear = function(t)
  if type(t)=="table" then
    for k,v in pairs(t) do
      if type(v) == "table" then
        tableClear(v)
      end
      t[k] = nil
    end
  end
  t = nil
end

local INPUTS = {}
INPUTS.SWITCHES { 'sa', 'sb', 'sc'}
INPUTS.SLIDERS = {'ls', 'rs'}

tableClear(INPUTS)
collectgarbage()

6. Keep code in several files and load it when needed

local FUNCTIONS = {}
local x,y = 20,40

FUNCTIONS.displayHud = loadScript('hud.lua')
FUNCTIONS.displayHud(x,y) 
tableClear(FUNCTIONS.displayHud)
collectgarbage()

7. Compile to LUA byte-code *.luac

As compiler is now standard EdgeTX feature you may consider compiling *.lua to *.luac
LUA byte-code is

• smaller (20-50% memory usage reduction depending on coding style)
• faster

If you have LUA script in one file it will be auto-compiled once it is run
In case of multiply files compilation can be forced during runtime using loadScript mode argument

loadScript('myScript') -- this will auto-compile
loadScript('myScript', 'tc') -- this will force compilation before load
loadScript('myScript', 'tx') -- this will block compilation before load
loadScript('myScript', 'b') -- this will to load myScript.luac instead of myScript.lua

Compiler consumes some amount of memory so in case of big LUA app with several files it's good practice to write pre-compilation code that is run before first LUA app usage. (BF LUA script uses this technique)

To manipulate easily loading and compiling process use load-proxy function

local loadMode = 'bx'

local myLoad = (file,...)  -- global load function
  local rv = loadScript(file,loadMode)  (...)
  return rv
end

local FUNCTIONS = {}
local x,y = 20,40

FUNCTIONS.displayHud = myLoad('hud.lua', arg1, arg2)
FUNCTIONS.displayHud(x,y) 

So now you can modify globally what happens during load.

• globally change loading *.luac instead of *.lua
• globally change files locations path
• pass variable to every loaded file
• check if loaded lua returns properly value
• garbage collect after file loading
  etc

main.lua (main script located wherever)

local PATH = '/SCRIPTS/TOOLS/MYSCRIPT/' -- files location 
local loadMode = 'b' -- load *.luac only

local CONFIG = {) -- config variables table
CONFIG.hud_x = 20
CONFIG.hud_y = 10
CONFIG.hud_w = 100
CONFIG.hud_h = 60

local myLoad = (file,...)  -- global load function
 -- 
  local rv = loadScript(PATH..file,loadMode)  (CONFIG,...) -- add PATH and pass additional CONFIG variable to every loaded file
  if rv == 'nil' then -- check if object (table) is returned
    error('Script '..file...' returned nil')
   end
   collectgatbage() -- force grabage collector to purge anything left during LUA loading 
  return rv
end

return run = function()
  local HUD = myLoad('hud')
  HUD.display(20,30)
  return 0
end 

hud.lua (loaded script located in /SCRIPTS/TOOLS/MYSCRIPT/)

local CONFIG = ...

HUD = {}
HUD.display = function(pitch,roll)
 lcd.drawHudRectangle(pitch, roll, CONFIG.hud_x, CONFIG.hud_x+CONFIG.hud_witdh,CONFIG.hud_y, CONFIG.hud_y+CONFIG.hud_height)
end

return HUD

TRADE-OFFs

1. *.luac compiled on radio (STM32 MCU) is not compatible with *.luac compiled on Companion Simulator

• if you distribute *.lua files then pre-compilation code mentioned before will be solution
• if are not going to distribute *.lua files. All *.lua files should be compiled to *.luac on any radio then included as released files.

Hope that helps.

PS.
Thanks @yaapu for 5 and @lshems for 6.
If you have any other LUA memory optimisation tips & tricks share it here

[lshems]

Some remarks: variable name lengths are relevant in LUA as everything is stored as key-value pair, and a variable name is a key.
For 'library like' or 'constants' variables that you use often but are not neccesary for readability, it is strongly recommended to keep them short!

Item 2 is for me counter intuitive, as a 'sequence' in LUA is just a key-value pair with integers as keys. So creating two arrays with identical integers as references to keys and values, instead of one key-value array, would take more memory according to my way of viewing LUA.
I might be wrong, but make dure your statement is correct, as it is a real code burden to do it the way you suggest.

[pfeerick]

Since you've commented, I'll just take this opportunity to toot your horn so you don't have to be worried about doing it yourself 🤭 ... as you have a nice set of articles on Lua tips and tricks, which while they are OpenTX focused, are quite relevant for EdgeTX... I have it bookmarked for that day I start trying to do some real Lua scripting (not sure if/when that will actually happen) ... so also have to bookmark this one now as well :)

https://www.rcgroups.com/forums/showthread.php?2979802-Tips-and-Tricks-for-LUA-on-OpenTX

[JimB40]

Well those findings are based on hard numbers from many tests with large data sets.
LUA implementation in OpenTX/EdgeTX is .. as it is :)

AD 1.
You are right but as I wrote this matters when *.lua code is interpreted. You need memory space for this long names.
When intretpreted it is is indeed stored as key which is probably just a number.
If compiled *.luac is executed it doesn't matter as this process was already done during compilation.
That's why LUA byte-code is significantly smaller.

[lshems]

Compilation doesn't shorten variable names! They are containing information because of the key value pair concept of LUA.

So compiled or not compiled, long variable names eat memory...

Or did I understand you wrongly?

[JimB40]

Well Guido as I said my approach to testing was that I didn't look at documentation but at test results.

Here is test script for tables.
In both cases (numeric and string keys) I've used same number (4) of 'characters' to check how table's key is stored

local toolName = "TNS|Table Test|TNE"

local tmp = {}

print('899 numeric table keys')
for i = 1100,1999 do
 tmp[i] = 'xyz'
end

-- print('899 string table keys')
-- for i = 100,999 do
--  tmp['a'..i] = 'xyz'
-- end

print('after table items created',collectgarbage('count'))

return { run = function(evt) return 1 end }

and here are results

899 numeric table keys
after table items created	33.5625	576

899 string table keys
after table items created	83.41796875	428

But indeed it's about storing key. So if create additional table with keys as second table's values, memory usage will be the same. Thx for pointing out.

[lshems]

About compilation: for some strange reason compilation on opentx is compatible with radio compilation, and will also simply work. At least in my case with several hundreds of files to be compiled....

[JimB40]

Nope. Talking about same compilation settingd ie 'tc' without debbugin info.

[lshems]

Lol. In Sicily now, on foot of Etna. No radio at hand...

[lshems]

Forgot because having a good time...

[gagarinlg]

You should always carry a radio with you, wherever you are

[lshems]

Nop. Not a good advice. Spent two hours waiting at airport police in Morocco because of a radio in my bag pack. Eventually they decided I was allowed, but the plane was gone. Spent two days extra on my own cost over there...