Workaround for Linux Perf Probes issue for Oracle Tracing

As explained in my previous post there is some issues with uprobes and the recent kernel/oracle version.Based on the workaround that i described i will show in  this short blog post how we can put a probe point on oracle function using Linux Perf. Sadly i haven’t figured out a way to do that using systemtap (Special thanks to Frank Ch. Eigler for his help)

Continue reading

Tracing Memory access of an oracle process : Intel PinTools

This blog post is motivated by a conversation with Frits Hoogland on his great blog post The curious case of the missing semctl call about how he managed to find  a useful memory address (suspecting a fixed SGA variable) used by a process in his investigation.So here i will show how we can easily generates a trace of all/range of memory addresses referenced by a program with an acceptable overhead.

Continue reading

Troubleshooting Latch Contention using sytemtap

The purpose of this blog post is to show how we can troubleshoot contention on  a specific latch using a systemtap script. This post is highly inspired by the “latchprof” script developed by Tanel Poder and his systematic approach for latch contention troubleshooting (For more info please check latch-contention-troubleshooting .)

This is what we are going to achieve :

Tested in : oracle 11.2.0.4/OEL6/UEK4

stap -v monitor_latch.stp  “latch_address” “latch#” “refresh_time”

capture-01

This script show a breakdown of latch holder by pid/session id/sql_hash for “cache buffers chains” latch with address “0x000000009F69FF60”

Continue reading

Dynamic tracing tools : Easier access to session/process address [ksupga_]

When troubleshooting a performance problem or investigating oracle internal using dynamic tracing tools like systemtap,it’s often useful to have the session address at hand. In fact, having the session address we can access many useful information such as : wait_event,p1 and p2  value,sql_id,and many other fields as stored in X$KSUSE (underlying table to V$SESSION). Luca Canali have already done a great work ,he identified that when the function “kskthewt” is called at the end of a wait event the register R13  (tested with Oracle 11.2.0.4 on RHEL6.5 and with Oracle 12.1.0.2 on OEL7 respectively) is pointing to the session addr with some offset and he manged also to determine the offset of the different column of X$KSUSE using X$KQFCO and X$KQFTA as in here.

The question is : Can we determine the session address without probing any function call ?

One way to answer this question is to determine how the value stored in the register R13 was set in the function “kskthewt”. Time to disassemble !

NOTE : This post contain no disassembly code of the oracle executable just the finding !

For basic info on reverse engineering please take look at my previous post.

Continue reading