uprobes issue with oracle 12c

If you are using dynamic tracing tools like systemtap/perf for user space probing (based on uprobes/uretprobes) with recent oracle/kernel version you may have hit this issue.As stated by Luca Canali Ref

“Issues with uprobes and Oracle versions: uprobes works OK for tracing Oracle 11.2. However, for Oracle 12.1 I find that uprobes works OK on  RHEL7.0 (kernel 3.10.0-123) and UEK (kernel 3.8.x), but does not work for kernels that ship with RHEL 7.1,7,2 and anything higher (including UEK4). When testing the easiest is to use Oracle 11.2 or if you want to test Oracle 12.1 use UEK3 or RHEL 7.0 kernel. More investigations are needed on this topic.”

Let’s check :

Using systemtap 3.0 /oracle 12.2/OEL6/UEK4 :

registration error (rc 0) !

Using perf  /oracle 12.2/OEL6/UEK4 :

Yep , another issue !

Time to take a closer look at uprobes !

Handling Uprobes directly ( via Linux ftrace )  with oracle 11.2.0.4 :

For more info check Documentation/trace/uprobetracer.txt, Brendan Gregg : linux-ftrace-uprobe.html

Let’s put a probe on function “kskthewt”(called at the end of an Oracle wait event) for the “pmon” process.First we have to find the function address :

The function address is “0x93f74f4”. Now we have to to subtract the base mapping address.

So we gonna place our probe point at address “0x8FF74F4”

We have placed a probe point which is not yet enabled !

Let’s take a look at the function prologue before enabling it : (for more info check Reverse engineering : What we need to know as a DBA ? )

Let’s now enable our probe point :

All is working as expected, the tracing is activated !

Let’s recheck our function prologue :

The original opcode was replaced with int3 (software interrupt).

Disable tracing :

Handling Uprobes directly  with oracle 12.2.0.1 :

Follow the same steps as described in the previous section to calculate the address for the probe point.(kskthewt)

Let’s take a look at the function prologue before enabling it :

A new instruction is there ! “xchg %ax,%ax” NOP operation

Let’s now enable our probe point :

For some reason we are not able to do that !  Unknown error 524 !

Are we done ? Let’s try to put a probe point on the next instruction (+2)

It work ! Let’s recheck our function prologue :

The original opcode was replaced with int3 (software interrupt).

So as a workaround we can use this trick to trace user space function with uprobes !

Brendan Gregg have written a wrapper for uprobes that automate must of this steps : here is an example :

Tracing oracle wait events of a specific process :

eventsname.sql

That’s it 😀