As explained in my previous post there is some issues with uprobes and the recent kernel/oracle version.Based on the workaround that i described i will show in this short blog post how we can put a probe point on oracle function using Linux Perf. Sadly i haven’t figured out a way to do that using systemtap (Special thanks to Frank Ch. Eigler for his help)
SYSTEMTAP
uprobes issue with oracle 12c
If you are using dynamic tracing tools like systemtap/perf for user space probing (based on uprobes/uretprobes) with recent oracle/kernel version you may have hit this issue.As stated by Luca Canali Ref
“Issues with uprobes and Oracle versions: uprobes works OK for tracing Oracle 11.2. However, for Oracle 12.1 I find that uprobes works OK on RHEL7.0 (kernel 3.10.0-123) and UEK (kernel 3.8.x), but does not work for kernels that ship with RHEL 7.1,7,2 and anything higher (including UEK4). When testing the easiest is to use Oracle 11.2 or if you want to test Oracle 12.1 use UEK3 or RHEL 7.0 kernel. More investigations are needed on this topic.”
Let’s check :
“CREATE ANY DIRECTORY”=DBA=SYSDBA ! Ouch !
As Kamil Stawiarski explained in some great articles :
“A lot companies consolidates databases into one appliance – like for example Oracle Exadata. So you can have a lot of different databases in one physical cluster. And what if I tell you that you can execute any OS command as an oracle user, having just access to a database user with appropriate privileges? What if I tell you that DBA=SYSDBA? And not just SYSDBA for one database but for every database in a cluster?” Ref1
This is possible using only three elements thanks to the PREPROCESSOR feature introduced in oracle 11G Ref2 :
Extending systemtap scripts with oracle session info
There are many useful systemtap scripts out there that can be used for troubleshooting performance problem or any other tasks. In some case it could be handy to extend them with session information to have a better picture of what’s going on.Based on my previous work on Dynamic tracing tools : Easier access to session/process address [ksupga_] here is some examples :
Latch acquisition/release call-graph including UltraFast Latch
In one of my previous post i showed that it was not possible to draw a complete latch call graph for a specific process based only on dedicated function call such as ksl_get_shared_latch,kslgetl and kslfre. The reason for this is that UltraFast latch can be acquired without the need for a dedicated function call (ex: inside kcbgtcr function).
So how to track them ?
Troubleshooting Latch Contention using sytemtap
The purpose of this blog post is to show how we can troubleshoot contention on a specific latch using a systemtap script. This post is highly inspired by the “latchprof” script developed by Tanel Poder and his systematic approach for latch contention troubleshooting (For more info please check latch-contention-troubleshooting .)
This is what we are going to achieve :
Tested in : oracle 11.2.0.4/OEL6/UEK4
stap -v monitor_latch.stp “latch_address” “latch#” “refresh_time”
This script show a breakdown of latch holder by pid/session id/sql_hash for “cache buffers chains” latch with address “0x000000009F69FF60”
Investigating Oracle locks using systemtap
A few weeks ago i was reading a great article written by Franck Pachot about investigating lock issue using event 10704 , so come to me the idea to use systemtap for this investigation as it give us more flexibility (For example : monitoring TM lock only on a specific table).
Simple ORACLE Data masking using systemtap
In my previous article i showed how we can write a mini database firewall using systemtap.In the second post of this series i will extend the previous script by adding a simple data masking functionality .This is what we are going to achieve :
systemtap : A mini oracle db firewall
As the blog post name suggest this article is about writing a “mini” program for displaying and filtering statement execution issued from a specific IP address (parameter 1) on a specific database (parameter 2). This is heavily based on the great work done by Luca Canali here (Must read !). Continue reading
Improved start-up times for ORACLE database in UEK4 (Concurrent Huge Page faults)
As you may know Unbreakable Enterprise Kernel Release 4 is finally here with many enhancement.So If you plan to upgrade your kernel to UEK4 and you are using Huge page for your database (And you probably should) you “may” see a great improvements in start-up times. So what have changed ?
Hatem Mahmoud's blog 