uprobes issue with oracle 12c

If you are using dynamic tracing tools like systemtap/perf for user space probing (based on uprobes/uretprobes) with recent oracle/kernel version you may have hit this issue.As stated by Luca Canali Ref

“Issues with uprobes and Oracle versions: uprobes works OK for tracing Oracle 11.2. However, for Oracle 12.1 I find that uprobes works OK on  RHEL7.0 (kernel 3.10.0-123) and UEK (kernel 3.8.x), but does not work for kernels that ship with RHEL 7.1,7,2 and anything higher (including UEK4). When testing the easiest is to use Oracle 11.2 or if you want to test Oracle 12.1 use UEK3 or RHEL 7.0 kernel. More investigations are needed on this topic.”

Let’s check :

Using systemtap 3.0 /oracle 12.2/OEL6/UEK4 :

Capture 01

registration error (rc 0) !

Using perf  /oracle 12.2/OEL6/UEK4 :

Capture 02

Yep , another issue !

Time to take a closer look at uprobes !

Handling Uprobes directly  with oracle 11.2.0.4 :

For more info check Documentation/trace/uprobetracer.txt, Brendan Gregg : linux-ftrace-uprobe.html

Let’s put a probe on function “kskthewt”(called at the end of an Oracle wait event) for the “pmon” process.First we have to find the function address :

Capture 03

The function address is “0x93f74f4”. Now we have to to subtract the base mapping address.

Capture 04

So we gonna place our probe point at address “0x8FF74F4”

Capture 05

We have placed a probe point which is not yet enabled !

Let’s take a look at the function prologue before enabling it : (for more info check Reverse engineering : What we need to know as a DBA ? )

Capture 06

Let’s now enable our probe point :

Capture 07

All is working as expected, the tracing is activated !

Let’s recheck our function prologue :

Capture 08

The original opcode was replaced with int3 (software interrupt).

Disable tracing :

Capture 09

Handling Uprobes directly  with oracle 12.2.0.1 :

Follow the same steps as described in the previous section to calculate the address for the probe point.(kskthewt)

Capture 10

Let’s take a look at the function prologue before enabling it :

Capture 11

A new instruction is there ! “xchg %ax,%ax” NOP operation

Let’s now enable our probe point :

Capture 12

For some reason we are not able to do that !  Unknown error 524 !

Are we done ? Let’s try to put a probe point on the next instruction (+2)

Capture 13

It work ! Let’s recheck our function prologue :

Capture 14

The original opcode was replaced with int3 (software interrupt).

So as a workaround we can use this trick to trace user space function with uprobes !

Brendan Gregg have written a wrapper for uprobes that automate must of this steps : here is an example :

Tracing oracle wait events of a specific process :

Capture 15

That’s it 😀

 

One thought on “uprobes issue with oracle 12c

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s