Simulating V$KERNEL_IO_OUTLIER on LINUX using Systemtap

With the 12C version oracle come with many amazing feature, one of them is the integration of ORACLE database with the awesome dynamic tracing framework  DTRACE on SOLARIS system.In fact we can see this in action using  V$KERNEL_IO_OUTLIER view.Like explained by Andrey Nikolaev in this view is based on a DTrace script running in the background.He also managed to rewrite the script so it can be used also on pre-12c Oracle databases.(That’s cool !!)

But what About Linux system ?

Please take look at Brendan Gregg presentation on Linux performance tools :

I have chosen Systemtap to simulate V$KERNEL_IO_OUTLIER on LINUX as it’s one of the more mature tool.So i am going here to write a monitoring  script  based on the script developed by Andrey Nikolaev :

TEST system :

  • Systemtap 2.7
  • OEL release 6.6
  • Kernel : 3.8.13-55.1.5.el6uek.x86_64
  • Oracle database using ASM
  • disk_asynch_io=FALSE

Note : As Mentioned by Andrey Nikolaev :

  • V$KERNEL_IO_OUTLIER was designed to work for ‘raw devices’ or ASM disks only.
  • V$KERNEL_IO_OUTLIER was designed to work with disk_asynch_io=false only.

First we have to choose the correct probe point.I have based my script on existing tapset so it can be portable between different kernel version.

Here is the probe points :

  • syscall.pwrite
  • ioblock.request : Fires whenever making a generic block I/O request.  
  • ioscheduler_trace.elv_issue_request : Fires when a request is  issued
  • ioblock.end : Fires whenever a block I/O transfer is complete.  
  • syscall.pwrite.return

To check the position of this probe point on the i/o block path to the device and back i used this additional probes as a check :

probe kernel.function(“*@block/*”).call {
if (pid() == target()) {
printf (“%s -> %s \n”, thread_indent(1), ppfunc())
probe kernel.function(“*@block/*”).return {
if (pid() == target()) {
printf (“%s <- %s \n”, thread_indent(-1), ppfunc())

Here is an extract :

   1 oracle_3364_tes(3364): <- blk_start_plug
     0 oracle_3364_tes(3364): -> submit_bio
PROBE:    ->ioblock.request
     7 oracle_3364_tes(3364):  -> generic_make_request
     9 oracle_3364_tes(3364):   -> generic_make_request_checks

  0 oracle_3364_tes(3364): -> blk_flush_plug_list
     2 oracle_3364_tes(3364):  -> __elv_add_request
PROBE: ->elv_add_request
     5 oracle_3364_tes(3364):   -> elv_rqhash_find
     7 oracle_3364_tes(3364):   <- elv_rqhash_find

And then checked the code source of some of listed functions to see what it does.

So now that we have our probe points we can write the scripts : kernel_io_outlier_simul

Here is an extract :


That’s it  😀

More on systemtap coming soon.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s