Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

dead loop in err_unwind() due to lj_mem_newgco() memory allocation failure #98

Open
uttinie opened this issue Jun 19, 2020 · 0 comments

Comments

@uttinie
Copy link

uttinie commented Jun 19, 2020

openresty version : openresty-1.13.6.1
perf:Stuck in err_unwind and nginx process CPU 100% higher
error.log:nginx: lua atpanic: Lua VM crashed, reason: not enough memory
STACK1:
Traceback (most recent call last):

File "luajit21.py", line 588, in invoke

bt = lj_debug_dumpstack(L, 0, 30, base, full)

File "luajit21.py", line 423, in lj_debug_dumpstack

frame, size = lj_debug_frame(L, base, level, bot)

File "luajit21.py", line 218, in lj_debug_frame

if frame_gc(frame) == obj2gco(L):

File "luajit21.py", line 177, in frame_gc

return gcref(frame['fr']['func'])

File "luajit21.py", line 170, in gcref

return r['gcptr32'].cast(typ("uintptr_t")).cast(typ("GCobj*"))

g
gdb.MemoryError: Cannot access memory at address 0xfffffffffffffff8
E
Error occurred in Python command: Cannot access memory at address 0xfffffffffffffff8

STACK2:
#0 err_unwind (L=L@entry=0x41e9f378, stopcf=0x7ffcddad1520, errcode=errcode@entry=0) at lj_err.c:109

#1 0x00007f988b3504a5 in lj_err_unwind_dwarf (version=, actions=1, uexclass=5500374307216568836, uex=0x7f988c51e760, ctx=0x7ffcddad1170) at lj_err.c:240

#2 0x00007f988a347c33 in _Unwind_RaiseException () from /lib64/libgcc_s.so.1

#3 0x00007f988b350329 in err_raise_ext (errcode=4) at lj_err.c:301

#4 lj_err_throw (L=L@entry=0x41e9f378, errcode=errcode@entry=4) at lj_err.c:515

#5 0x00007f988b350563 in lj_err_mem (L=L@entry=0x41e9f378) at lj_err.c:551

#6 0x00007f988b350000 in lj_mem_newgco (L=L@entry=0x41e9f378, size=20) at lj_gc.c:833

#7 0x00007f988b3530ec in func_newL (L=L@entry=0x41e9f378, pt=pt@entry=0x41ee8010, env=0x41ea09d8) at lj_func.c:122

#8 0x00007f988b353310 in lj_func_newL_gc (L=0x41e9f378, pt=, parent=0x415cde28) at lj_func.c:160

#9 0x00007f988b34bbe9 in lj_BC_FNEW () from /usr/local/NSP/luajit/lib/libluajit-5.1.so.2

#10 0x00007f988b35d0bd in lua_pcall (L=L@entry=0x41e9f378, nargs=nargs@entry=0, nresults=nresults@entry=1, errfunc=errfunc@entry=0) at lj_api.c:1129

#11 0x00000000004f9c75 in ngx_http_lua_cache_load_code (log=log@entry=0x2062db70, L=L@entry=0x41e9f378, key=key@entry=0x21327480 "nhlf_cdbb7e422b70672672ea5d04e77ae42c")

at ../ngx_lua-0.10.11/src/ngx_http_lua_cache.c:56

#12 0x00000000004f9f1c in ngx_http_lua_cache_loadfile (log=0x2062db70, L=L@entry=0x41e9f378, script=0x31727c38 "/usr/local/NSP/etc/router/router_ext.luac",

cache_key=0x21327480 "nhlf_cdbb7e422b70672672ea5d04e77ae42c") at ../ngx_lua-0.10.11/src/ngx_http_lua_cache.c:232

#13 0x00000000004fb9c7 in ngx_http_lua_access_handler_file (r=0x31726cf0) at ../ngx_lua-0.10.11/src/ngx_http_lua_accessby.c:219

#14 0x00000000004fb5d0 in ngx_http_lua_access_handler (r=0x31726cf0) at ../ngx_lua-0.10.11/src/ngx_http_lua_accessby.c:163

#15 0x0000000000459dbb in ngx_http_core_access_phase (r=, ph=0x17ab770) at src/http/ngx_http_core_module.c:1087

#16 0x000000000045562d in ngx_http_core_run_phases (r=r@entry=0x31726cf0) at src/http/ngx_http_core_module.c:862

#17 0x0000000000455722 in ngx_http_handler (r=r@entry=0x31726cf0) at src/http/ngx_http_core_module.c:845

#18 0x000000000046024e in ngx_http_process_request (r=0x31726cf0) at src/http/ngx_http_request.c:1986

#19 0x0000000000460ac7 in ngx_http_process_request_line (rev=0x7f981e7851c8) at src/http/ngx_http_request.c:1076

#20 0x000000000044a09c in ngx_epoll_process_events (cycle=, timer=, flags=) at src/event/modules/ngx_epoll_module.c:968

#21 0x00000000004413df in ngx_process_events_and_timers (cycle=cycle@entry=0x15ac050) at src/event/ngx_event.c:266

#22 0x0000000000447f21 in ngx_worker_process_cycle (cycle=cycle@entry=0x15ac050, data=data@entry=0x17) at src/os/unix/ngx_process_cycle.c:821

#23 0x00000000004468c6 in ngx_spawn_process (cycle=cycle@entry=0x15ac050, proc=proc@entry=0x447ea0 <ngx_worker_process_cycle>, data=data@entry=0x17,

name=name@entry=0x5a58e5 "worker process", respawn=respawn@entry=-4) at src/os/unix/ngx_process.c:198

#24 0x0000000000448240 in ngx_start_worker_processes (cycle=cycle@entry=0x15ac050, n=32, type=type@entry=-4) at src/os/unix/ngx_process_cycle.c:396

#25 0x0000000000448ef5 in ngx_master_process_cycle (cycle=0x15ac050, cycle@entry=0x15a6230) at src/os/unix/ngx_process_cycle.c:250

#26 0x0000000000421583 in main (argc=, argv=) at src/core/nginx.c:387

@uttinie uttinie changed the title nginx process CPU 100% higher dead loop in err_unwind() due to lj_mem_newgco() memory allocation failure Jun 19, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant