Context
Use of the addr2line tool to convert stacktrace addresses into code lines in order to troubleshoot a server crash. If running on a different node than the one that crashed, ensure that the OS and the Aerospike Server Version are the same as the crashed node.
Method
Create an Aerolab cluster matching OS and Aerospike Versions
Save stacktrace to a file
grep 'stacktrace' aerospike.log > MY_STACKTRACE_FILE.txt
Example Stack trace
May 13 2020 16:09:00 GMT: WARNING (as): (signal.c:187) SIGSEGV received, aborting Aerospike Enterprise Edition build 4.8.0.4 os el7
May 13 2020 16:09:00 GMT: WARNING (as): (fault.c:1178) stacktrace: registers: rax 0000000007121f60 rbx 00007fdb3d59be78 rcx 00007f96d5400000 rdx 0000000000001000 rsi 00007f96d5400000 rdi 00007f96d5400000 rbp 00007f96d5401f78 rsp 00007f966e9fa1a0 r8 00007f96d543f000 r9 00007f96d5401f60 r10 000000000000000d r11 0000000000001fe0 r12 00007f97675e7180 r13 00007f966e9fb100 r14 00000019deae8ba0 r15 ffffffffffffffff rip 00000000007961be
May 13 2020 16:09:00 GMT: WARNING (as): (fault.c:1192) stacktrace: found 14 frames: 0x6f664b 0x4f289f 0x7f976b1617e0 0x7961be 0x7b7faf 0x78c4e5 0x5bd245 0x5be9ab 0x5d6765 0x5f1d65 0x5f55a5 0x5d4e6b 0x7f976b15740b 0x7f9769f97e7f offset 0x0
.
.
.
Move MY_STACKTRACE_FILE.txt to Aerolab Cluster.
Attempt to run addr2line in one line
addr2line -i -C -f -e /usr/bin/asd $(grep "stacktrace: found" MY_STACKTRACE_FILE.txt | grep -Eo "0x[0-9A-Za-z]{5,}" | sed '$d' | tr '\n' ' ')
Example Output
??
??:0
reraise_signal
/work/source/as/src/base/signal.c:94 (discriminator 3)
as_sig_handle_usr1
/work/source/as/src/base/signal.c:221 (discriminator 3)
??
??:0
??
??:0
cf_log_write_no_return
/work/source/cf/src/log.c:520 (discriminator 1)
as_sindex_gc_record
/work/source/as/src/sindex/gc.c:112 (discriminator 1)
as_record_done
/work/source/as/src/base/record.c:126
drop_local
/work/source/modules/ee/as/src/transaction/delete_ee.c:125
expire_reduce_cb
/work/source/as/src/base/nsup.c:473
as_index_sprig_reduce_no_rc
/work/source/modules/ee/as/src/base/index_ee.c:249 (discriminator 1)
as_index_reduce_from
/work/source/as/src/base/index.c:269
as_index_reduce_live
/work/source/modules/ee/as/src/base/index_ee.c:129
run_expire
/work/source/as/src/base/nsup.c:453
joinable_shim_fn
/work/source/cf/src/cf_thread.c:405
??
??:0
??
??:0
Alternative method for running addr2line on the list of frames printed directly if first method fails.
addr2line -e /usr/bin/asd -i -f -C 0x6f664b 0x4f289f 0x7f976b1617e0 0x7961be 0x7b7faf 0x78c4e5 0x5bd245 0x5be9ab 0x5d6765 0x5f1d65 0x5f55a5 0x5d4e6b 0x7f976b15740b 0x7f9769f97e7f > MY_BACKTRACE_CASEXXXX.txt
Replace hexcodes shown in bold with the hexcodes shown in your MY_STACKTRACE_FILE.txt
The output of this code should be included in Jira for engineering investigation
Applies To Earliest Version
Pre 4.9
Applies To Latest Version
Current Version
