Probevue - a new AIX6 programers tool

AIX has included its Trace feature since 1990

Pre-built in hooks (code) in the AIX kernel and these mostly capture the stat and parameters of kernel functions
Designed for production use but be careful it collects so much information it can slow the performance

AIX 6 ProbeVue - in the AIX 6.1 release.

Dynamically added when needed for user code too
Zero code modifications
Once enabled they trigger actions in C "like" language = Vue
Designed for production use but very little effect on performance
Don't dump every thing
Use script to identify what you really need or find error conditions

Note: I strongly recommended that you read the Extended User Guide (you can download this from http://www.ibm.com/developerworks/aix/library/au-probevue) rather than the regular online documentation, which I found confusing and out of date (I am told this will be fixed at the next AIX documentation release).

ProbeVue - Five ways of starting

You can interactively run probevue (seems like hard work to me as you may have to type and retype the details!!) or run probevue via a script. There are at least five ways which can be initially confusing so let us explain the ways and you can decide which one way you want to try:

1) and 2) Start probevue with a script and either name the script as the input parameter or redirect the script into the command as follows (this is two ways):
```
probevue myscript.e
probevue <myscript.e
```

3) Auto-start scripts with the special first line clue to the shell to start probevue to handle the file content

Script (in this example called myscript) includes 1st line: #!/usr/bin/probevue
Make the file executable: chmod +x myscript
Then just run the script:  ./myscript.e

4) Some scripts need parameters (like shell scripts $1, $2 etc.). In particular, many need a prosess ID (PID)
```
Find the porcess ID: ps -ef | grep myprog
Run with the PID argument: probevue   myscript   43561
```

5) To use probevue directly with a user program

probevue   -X progname   -A prog-arguments   myscript

Note: the developers recommend that Probevue scripts end with a ".e" extension but its only a recommendation.

Vue Script - the Basics

The probevue script has the following structure (the <-- are comments i.e. NOT part of the script)

<- optional declaring function prot-type so probevue understands your function arguments
@@BEGIN
{
...                 <- optional start up code, good for initialisation of variables
}

@@<probe-spec>        <- Probe point specification Tuple (see below)
when <predicate>        <- Optional predicate
{
    statements here;	<- Probe actions in C like code
...
}

@@END
{
...                              <- Optional ending code.  Can print out final results. Gets runs if you hit Control+C
}

Probe point specification "Tuples"

First note that Tuple is a fancy name for an ordered list - once I looked this up it helped!

There are current three types of Tuple:

1) User Function Tracing probes (or uft) for entry of exit of user written code
- Syntax : @@uft:PID:*:FunctionName:[entry/exit]
- Example: Probe at entry into function foo() (main executable or libraries),PID = 34568
  - @@uft:34568:*:foo:entry
  - Note the "*" is a place holder for the next version

2) System Call Entry/Exit probes (or syscall)
- Syntax: @@syscall:PID:SystemCallName:[entry/exit]
- THE PID id eiter a process id or the * character meaning all processes
- Example: Probe at exit of read system call all processes
  - @@syscall:*:read:exit

3) Interval probes (fire at specific time intervals)
- Syntax: @@interval:*:clock:milliseconds
- The * character is mandatory in this release and only clock is supported in this release.
- Example: Probe to fire every 500 milliseconds (wall clock time).
  - @@interval:*:clock:500

More tuples are expected in later releases.
Only variable declarations are allowed outside of a tuple clause but all of them must appear before the first tuple clause where they are used.
Variables may be initialized anywhere, but the @@BEGIN probe would be a good choice generally.

First example probevue script

Here is a very simple script that actually works and does something useful.

#!/usr/bin/probevue                <-- autorun
@@BEGIN
{
int count, total;                       <-- declare the variables (later probevue versions will not need this)
}

@@syscall:*:read:entry            <-- System call: any process ID, read() function, at start of the function
{
   count++;                             <-- increment the counters
   total++;
}

@@interval:*:clock:1000          <-- once per second (1000 ticks per second)
{
   printf("Number of reads = %d\n", count);
   count=0;
}

@@END                                   <-- After you type Control-C
{
    printf("\nTotal reads = %d\n",total);
}

Here is some example output:

# ./count.e
Number of reads = 0
Number of reads = 1
Number of reads = 0
Number of reads = 0
Number of reads = 2
Number of reads = 0
Number of reads = 5
Number of reads = 269
Number of reads = 0
Number of reads = 0
^C
Total reads = 277

Second Example with return codes from a function

#!/usr/bin/probevue
@@BEGIN
{
int read(int fd, void *buf, int n);       <-- Declare function so ProbeVue understands it
int bad;
int good;
}
@@syscall:*:read:exit
when(__rv == -1)                         <-- __rv means the return value
{
   bad++;
}
@@syscall:*:read:exit
when(__rv != -1)
{
   good++;
}
@@interval:*:clock:1000                  <-- Once a second output counter
{
   printf("Reads good=%d bad=%d\n",good,bad);
   good=0;
}

Here is some sample output from the above script, I had to run a program to generate a little disk I/O. Note: The bad I/O is generated when a program tries to read from a file descriptor that is not valid (i.e. the open() function was not used to initialise the file descriptor):

# # ./goodandbad.e
Reads good=0 bad=0
Reads good=55 bad=0
Reads good=0 bad=0
Reads good=0 bad=0
Reads good=57 bad=1
Reads good=0 bad=1
Reads good=0 bad=1
Reads good=55 bad=1
Reads good=0 bad=1
Reads good=1 bad=1
Reads good=55 bad=1
Reads good=40 bad=1
Reads good=1 bad=2
Reads good=55 bad=3
Reads good=1 bad=3
Reads good=0 bad=3
Reads good=56 bad=4

Third example of monitoring a user program

I have a small program that generates lots of CPU work. The program called ncpu has a function "engine()" which uses up CPU cycles on the CPU. Below is the script to investigate the use of this function. Note: the program is already compiled and running and it includes the symbol table which probevue uses to find function entry and exit points (i.e. not stripped). The probevue script includes probevue functions to output the details it can find of the program and its environment:

#!/usr/bin/probevue

double engine(int p1, int p2);

@@uft:$1:*:engine:entry
{
     printf("PID=%d TID=%d PPID=%d PGID=%d UID=%d GID=%d InKernel=%d\n", __pid, __tid, __ppid, __pgid, __uid, __euid, __kernelmode);
      printf("ProgName=%s errno=%d\n", __pname, __errno);
       printf("---\n");
      stktrace(GET_USER_TRACE,20);
      printf("+++\n");
      stktrace(PRINT_SYMBOLS|GET_USER_TRACE,20);      exit(0);
}

Here are the commands to run this script and its output:

PID=2523154 TID=716845 PPID=315540 PGID=2523154 UID=0 GID=0 InKernel=0
ProgName=ncpu errno=0
---
0x100001dc
0x10000964
0x10001104
+++
.__start+0x8c
.main+0x60c
.work+0x1d0

Fourth Example check the parameters to a user program function

Let us now investigate this function called engine(). It takes two parameters and the first is a randomly generated number between 1 and 1 million - but are they? We will put the random number if to one of four "buckets". The idea is that if random the buckets should be roughly the same size - never exactly but at least similar in size:

#!/usr/bin/probevue

double engine(int p1, int p2);
int b1, b2, b3, b4;

@@uft:$1:*:engine:entry
{
        tmp = __arg1;
        if( tmp < 250000 ) b1++;
        if( 250000 <= tmp && tmp < 500000 ) b2++;
        if( 500000 <= tmp && tmp < 750000 ) b3++;
        if( tmp  >750000 ) b4++;
}

@@END
{
printf("\nFour bucket results are: %d %d %d %d\n",b1,b2,b3,b4);
}

Here is the commands and output of the above script:

ps -ef | grep ncpu
    root 299106 307250  27 14:02:00  pts/0 15:16 ./ncpu -p1 -z 90
    root 315596 307250   0 16:48:47  pts/0  0:00 grep ncpu
#  # ./random 299106
[[[Wait 30 seconds here]]]
^C
Four bucket results are: 101524 101262 101454 120657

Oh dear - there is more in more last bucket than randomisation would cause but that is not probevue's problem!! It is a genuine BUG in my code. Found out with a further more detailed probevue script that some values greater than 1 million!! Probevue added real value here for this "production" workload without interruption the application and no recode/recompiling/restart of the program. I can't think of any other way this could have been done.

Final thoughts on this first version of probevue

I did not cover more advanced features like:

Tentative capture - buffering results and later commit/discard the numbers once you know you have hit the point you are investigating,
Reading data from the user program,
Thread handling,
The probevctrl command which is used to alter default behavior

For AIX 6.1 this is initial release, I am told there may be an extended user guide - that will be an improvement on the current manual pages.

This first AIX6 version has things you should be aware of:

Kernel variables limited to those pinned in memory
User C code function entry requires symbol table
No C pre-processor support
WPAR limits - while using probevue your WPAR is not mobility (can't be relocated). Once you stop probevue the WPAR is mobile again. Also the syscall tuple must have a PID which is viewable from the WPAR (the "*" is not allowed).
Thread local strings are not supported

Many more features are planned for 2008 release.

More Information

By far the best source of information on Probevue is the Extended User Guide that you can download this from the Internet at http://www.ibm.com/developerworks/aix/library/au-probevue
Probevue is found on AIX6 i.e. POWER4, 5 or 6 machines and also the JS21 & JS22 IBM Blades.
For more information start at google.com, search for "probevue" take the AIX command link. Then also take the "Dynamic Tracing" at the end for Vue Language.
Manuals are not enough to get you started - expect a extended user guide. For AIX6 users you can find the Extended User guide in HMTL form on your system at /usr/lpp/bos.sysmgt/probevue.eug.html This runs to 60+ pages and includes 21 worked examples plus the definitions of all probevue functions and the supported systems calls, and kernel variables that can be used. However the version on the Internet (see above) will be the latest version.

If you have a good probevue script working - why not shared it here.