I added one new probe to it, that solves a simple but particularly vexing
issue: tracing and aggregating on the regular expression used to match a token
instead of the actual token or the returned token-id. This new probe is called
match_regex.

The returned token-id (value returned by yylex()) would only be adequate in
identifying the most commonly matched regular expressions, if yylex()
necessarily returned on every single match. It doesn’t.

The two closest ways to achieve what the new probe achieves, is by (1)
recording the final state that occurred before a match, and then digging
through the generated lex.yy.c file to see what regular expression that state
correlates to, and (2) using the action probe to see where in the switch
statement we are jumping to. (2) No longer works, as the action probe has
been removed (it’s redundant with match_regex, and less helpful).

The first method is convoluted and inelegant. The second method isn’t so bad,
and is actually remarkably similar to match_regex. With action, one can go
to the specific case within the switch-statement that gets exectuded. and find
that lex has helpfully placed a # line macro, with a line number that
matches the line of the regular expression in the .lex source file. Using
this information, one could find which regular expressions are being matched,
without depending on the return value of yylex().

The actual match_regex probe, allows one the luxury of not having to juggle
two files (the .lex source file and the generated lex.yy.c file), and to
instead focus on just one (the .lex source file) by giving the line number of
the defined regular expression as its first argument. And, should one have
access to the generated lex.yy.c file, but not the .lex file, one can use
the second argument to get the line number of the probe itself (it’s located
within the switch statement).

Now a quick demo.

Here is a simple lexer (.lex) for a calculator:

#include <stdio.h>
#include <stdlib.h>

%%

[0-9]+          { return (0); }
"-"[0-9]+       { return (1); }
"+"             { return (2); }
"-"             { return (3); }
"*"             { return (4); }
"/"             { return (5); }

%%

int
main(int ac, char **av)
{

    ++av; --ac;
    if (ac > 0) {
        yyin = fopen(av[0], "r");
    } else {
        yyin = stdin;
    }

    while (!feof(yyin)) {
        yylex();
    }

}

Here is an input file for the lexer to process:

2 + 2
-2 + 2
3 * 90
60 / 5
4321 - 1234
+ 2 2 3 8 9
- 9 8 1

A simple dtrace invocation that runs the lexer, and indicates which regular
expressions were matched the most:

pfexec dtrace -c './calc_lex calc_in' -n 'lex$target:::match_regex {@[arg0] = count();}'

 7                1
10                1
11                1
 9                2
 8                3
 6               17

A slightly different dtrace invocation that is the same as the previous one,
but indicates the part of the switch statement we jump to when we match a
regex:

pfexec dtrace -c './calc_lex calc_in' -n 'lex$target:::match_regex {@[arg1] = count();}'

10                1
13                1
14                1
12                2
11                3
99               17

Admittedly, this is less useful than the previous invocation, but it’s better
than nothing, and I’ve seen repositories where people mindlessly include just
the generated file, but leave the .lex file out.

Not being able to quickly find the most commonly matched regexes was a major
deficiency of the initial iteration of this provider. Only two lines of code
were added (since the last iteration) to the lex source code (which is now on
github) to make this happen. (See the diffs for sub1.c, for details).

I’ve only used lex a dozen times or so, thus far, but the new provider has
dramatically sped up the compile-test-debug cycles, by allowing one to ask very
specific questions and use DTrace constructs like aggregations. Ultimately the
amount of times one has to recompile is reduced, and the volume of debug-output
one has to burrow through is microscopic compared to the dump of debug-printf’s
that lex uses when #ifdef LEXDEBUG is true. Also, one can use the lex
provider in conjunction with other providers offered by Illumos and
DTrace-aware processes.

The modified code is located in my own branch of illumos-joyent, on
Github.

Happy lexing.

For instance,

dtrace -n 'syscall::brk:entry {@[arg0] = count();}'

will trace all the brk system calls made, and count the number of times that
an argument was passed to it.

However,

dtrace -n 'syscall::sbrk:entry {@[arg0] = count();}'

will not work.

This is because, while brk() and sbrk() are valid Unix interfaces, used to
modify the size of the calling process’s data segment, they aren’t system
calls. They are functions in the standard C library that comes with Illumos (or
some other DTrace-enhanced system), that wrap around a system call.

In our case, Illumos only supports a system call that is identified as brk.

] dtrace -l -n 'syscall:::entry' | grep brk
72078   syscall                     brk entry

As it turns out, both the sbrk() function and brk() function are
implemented in terms of the brk system call.

] dtrace -n 'sycall::brk:entry {@[ustack()] = count();}'
...SNIP...
libc.so.1`_brk_unlocked+0xa
libc.so.1`sbrk+0x2e
libc.so.1`_morecore+0x119
libc.so.1`_malloc_unlocked+0x189
libc.so.1`malloc+0x2e
libc.so.1`_findbuf+0x84
libc.so.1`_ndoprnt+0x91
libc.so.1`vfprintf+0x9f
dtrace`oprintf+0xa2
dtrace`main+0x1607
dtrace`0x4034bc
  1
...SNIP...

If you want to get the argument that’s passed to sbrk() you’ll have to use
the pid provider, like so:

] dtrace -p $PID -n 'pid$target::sbrk:entry {@[arg0] = count();}'

I have always found that planning is useless, but plans are indispensable.
-Dwight D. Eisenhower

Using the idea of resource allocation, I’ve created plan, a nifty command
line application that automatically schedules activities based on how long they
last, and when they should start. A truly automated digital replacement for the
much-hated daily planners. Other digital planners just digitize the tedium of
laying out your day. Replace your daily planners! Unix style.

Recently I came to the morbid realization that I had to manage my time, if I
wanted to get anything (besides programming) done. Upon realizing this most
unfortunate truth, I cried a little, found some graph paper, and shaded out the
grid, so that my weekly schedule would be apparent upon glimpsing at it. Sadly,
there was significant resistance from my stubborn circadian rhythm, which
wouldn’t allow me to wake up quite as early as I would have liked.

Instead of re-shading another piece of graph paper, I gave in and tried using
Google Calendar. It’s a very cute web application. Where “cute” is a euphemism
for “my grandma might like it”. And yet, despite its simplistic
point-click-drag paradigm, it was far too manual for my taste. But it got the
job the done, and I was actually managing my time. I was a living oxymoron.

Unfortunately, I have problems. More specifically, I have problems with the
implication of having a web-based time management system. The implication is
that time management is useful if you have an internet connection, and
unnecessary if you don’t. No connection, no Google Calendar, no time
management. And my connection isn’t exactly persistent because I also have
problems with my university’s IT department.

I could have found a local TMS. But that presents another problem, endemic
to all of today’s TMSs. Namely repetition. Modern TMSs present the user with a
calendar that spans infinitely (practically speaking) into the future, and
allows the user to mark it up with a point-and-click interface. The first
problem is the fact that you have to use the mouse. Compared to typing on the
keyboard, the mouse is dog slow, and its use should be reserved for very
specific activities like photo editing. The second problem is that these TMSs
don’t differentiate between activities that have a starting time and a
duration, and todos that merely have a due time. The last problem is that
these TMSs fail to take into account is that most of our activities follow a
repetative weekly routine, but diverge on certain dates. For example, in
college, my class schedule is weekly, but my Calculus class is cancled on
11/09/19 (I wish).

Also, my problem with graphical applications (and thus web applications), is
that they are not integrated with Unix. Graphical applications can’t be piped
into grep or awk. Sure they can be scripted (look at Firefox), but these
scripts likely aren’t reuseable with other applications (or even with other
scripts in the same application). Scriptable graphical applications create
walled gardens, hinder code reuse, and generally waste my time by forcing me
to be redundant and manual. And besides, I don’t know of any local graphical
TMS that is scriptable.

The Unix command line is perfection. And I won’t settle for anything less.

In the name of truly efficient time management, I’ve created a command line
application, named plan. First, I’ll demonstrate what it’s capable of, then
I’ll go into its implementation.

Introduction and Demonstration

It’s a single command that does the following:
* allocates activities on a per-day and per-date basis
* establishes todo’s on a per-day and per-date basis

When an activity is allocated on, say, Sunday, it is understood that it repeats
every Sunday. However when that activity is allocated on (11-09-18), it is
understood that it only ever occurs on (11-09-18). The same applies for todos.

Creating and Destroying Activities and Todos

We can create activities and todos either on a given weekday, or on a
particular date.

Here we create an activity on Monday, called class_calc.

#plan create Mon/class_calc

As you can see, day names can be abbreviated. ‘Monday’, ‘Mon’, ‘monday’, and
‘mon’, all refer to the same week day.

We’ll also create a todo called hw_calc, which also occurs on every Monday.

#plan create Mon/@hw_calc

Todos are always prefixed with an ampersand (@).

Also, we can create a general todo, which is a todo that has no specific day,
date, or time.

#plan create @go_skydiving

The general gist of general todos is “I have to do this anytime between now,
and the day I die”.

For the sake of populating more than just one weekday, we’ll also create an
activity on Tuesday called class_db (my databases class):

#plan create Tues/class_db

Creating a random activity on a specific date:

#plan create 11-10-09/random_activity

We can also specify the previous command as:

#plan create 11-Oct-09/random_activity

or:

#plan create 2011-Oct-09/random_activity

and so on.

To destroy an activity or todo, we replace create with destroy in the above
commands.

Listing Activities and Todos

First, let’s list all of the activities that occur on Monday.

#plan list -a Monday
(0/1440)
NAME                    DYN    TIME     DUR
class_calc             true     N/A  00h00m

The -a specifies that we want to list activities.

So, we have one activity, class_calc.

The first line of output indicates the ratio of allocated minutes to available
minutes on this particular day. The second column, DYN may be enigmatic at
first. It’s a boolean that indicated whether or not an activity is dynamic. A
dynamic activity has a duration, but no explicit starting time. Dynamic
activities have their starting time assigned by plan.

The third column, indicates that the starting time is not yet assigned. The
fourth one indicates that the duration is zero. When an activity has no
duration, it is effectively unallocated.

Now, let’s list all of the activities that occur over a generic week.

#plan list -a week
Monday
(0/1440)
NAME                    DYN    TIME     DUR
class_calc             true     N/A  00h00m

Tuesday
(0/1440)
NAME                    DYN    TIME     DUR
class_db               true     N/A  00h00m

So that’s pretty neat, we can see our weekly schedule.

But it gets better. We can also display the weekly schedule for the current
week which will, replace week days that diverge from the weekly routine, as a
result of creating an activity on a date that this week overlaps with. As you
can see this week diverges from our generic weekly routine (namely, Sunday has
a random activity).

#plan list -a this_week
Sunday (2011-10-09)
(0/1440)
NAME                    DYN    TIME     DUR
random_activity        true     N/A  00h00m

Monday (2011-10-10)
(0/1440)
NAME                    DYN    TIME     DUR
class_calc             true     N/A  00h00m

Tuesday (2011-10-11)
(0/1440)
NAME                    DYN    TIME     DUR
class_db               true     N/A  00h00m

Sometimes, we may want to display the schedule (or todos) for next week, in
which case may do the following. The next week is the same as any generic week.

#plan list -a next_week
Monday (2011-10-17)
(0/1440)
NAME                    DYN    TIME     DUR
class_calc             true     N/A  00h00m

Tuesday (2011-10-18)
(0/1440)
NAME                    DYN    TIME     DUR
class_db               true     N/A  00h00m

We can view todos in much the same fashion.

#plan list -t week
NAME                   TIME
hw_calc               00:00
#plan list -t this_week
NAME                   TIME
hw_calc               00:00
#plan list -t next_week
NAME                   TIME
hw_calc               00:00

Since we have not created any todos that on a particular date, instead of a
week day, the three above commands share the same output.

Durations, Starting Times, and Awake Times

Let’s attempt to put a starting time on class_calc.

#plan set time=08:35 mon/class_calc
Monday: Can't set time on activity class_calc. Activity doesn't have duration.

What you’re seeing now is an error. plan won’t set a starting time on an
activity if it has a duration of 0 minutes. plan essentially allocates time
to activities and resolves conflicts between activities. We just asked it to
allocate nothing, which is bogus (think umem_alloc(0, flag), which always
returns NULL).

We’ll have to set the duration first.

#plan set duration=01h15m mon/class_calc
#plan list -a mon
(95/1440)
NAME                    DYN    TIME     DUR
class_calc             true   00:00  01h35m

So class_calc now has a duration and starts at 0:00 (12:00 am), because that’s
the earliest time at which it can possibly start.

However, we can change that, by setting the awake property for Monday.

#plan set awake=07:00,16h00m mon
#plan list -a mon
(95/960)
NAME                    DYN    TIME     DUR
class_calc             true   07:00  01h35m

Setting a wake-up time changes when the day starts. If the day doesn’t start
before 07:00, it makes no sense to have the class_calc activity start at 00:00 (as
we’re asleep). So plan automatically adjusted it to start at the earliest
possible time.

Now let’s try to set the starting time again.

#plan set time=08:35 mon/class_calc
#plan list -a mon
(95/960)
NAME                    DYN    TIME     DUR
class_calc            false   08:35  01h35m

Yay, it worked. And now, the class_calc activity is no longer dynamic. Which
means that it can’t be automatically moved to a different block of time if
there is a conflict.

Let’s test this. What happens if we try to set the awake property to one hour
after my Calculus 2 class starts?

#plan set awake=09:35,16h00m mon
Monday: Activity class_calc can't fit in the alotted time
#plan list -a mon
(95/960)
NAME                    DYN    TIME     DUR
class_calc            false   08:35  01h35m

As you can see, we can’t set the awake time on Mondays to 9:35, because class_calc
starts an hour before that! And because class_calc is not dynamic, we can’t
move it. As the list command indicates, we didn’t change anything. The
16h00m indicates that this day lasts for 16 hours. There are limitations:
waking up at 13:00, would prevent you from having a day that lasts for more
than 11 hours. The reason this limitation exists, is that even though I can
implement a day that starts at 13:00 and lasts 16 hours, I wouldn’t be able to
wrap an activity, that lasts say 3 hours, and starts at 23:00, around the end
of the day, due to an implementation detail. And besides, that would encourage
the wrong kind of behaviour!

Sometimes allocating time to an activity will fail because there isn’t any free
block of time that is big enough for the activity. If this activity is, say
01h30m (and it doesn’t need to be contiguous), we can always split it into 2 or
three chunks like so:

#plan set duration=00h45m*2 [[activity-name]]

or

#plan set duration=00h30m*3 [[activity-name]]

Note that a chunked activity can’t have an explicit starting time.

Renaming Activities and Todos

So, class_calc represents my Calculus 2 class. But I think class_calc2
would be a more apt name for it. I can rename it using the rename subcommand.

#plan rename mon/class_calc mon/class_calc2
#plan list -a mon
(95/960)
NAME                    DYN    TIME     DUR
class_calc2           false   08:35  01h35m

My Weekly Schedule

Here’s the weekly schedule that I follow. It was entirely generated by plan.
I must say, that I’m far more productive now than I’ve ever been.

Sunday
(990/1080)
NAME                    DYN    TIME     DUR
shower                false   06:00  00h30m
study_calc             true   06:30  02h00m
study_algs             true   08:30  03h00m
brunch                false   11:45  00h30m
study_db               true   12:15  03h00m
study_calc             true   15:15  01h00m
dinner                false   16:30  00h30m
prj_zaps               true   17:00  02h00m
study_csoc             true   19:00  01h00m
study_phys             true   20:00  03h00m

Monday
(890/1080)
NAME                    DYN    TIME     DUR
shower                false   06:00  00h30m
prj_zaps               true   06:30  01h00m
breakfast             false   07:30  00h30m
class_calc2           false   08:35  01h35m
prj_zaps               true   10:10  01h00m
class_algs            false   11:25  01h15m
lunch                 false   12:40  00h30m
study_phys             true   13:10  01h30m
study_db               true   14:40  01h30m
dinner                false   16:30  00h30m
study_algs             true   17:00  01h30m
study_calc             true   18:30  01h30m
prj_zaps               true   20:00  02h00m

Tuesday
(850/1080)
NAME                    DYN    TIME     DUR
shower                false   06:00  00h30m
prj_zaps               true   06:30  01h00m
breakfast             false   07:30  00h30m
study_phys             true   08:00  01h30m
class_db              false   10:00  01h15m
lunch                 false   11:20  00h30m
study_algs             true   11:50  01h30m
class_phys            false   13:50  01h15m
prj_zaps               true   15:05  01h00m
dinner                false   16:30  00h30m
prj_zaps               true   17:00  00h30m
study_calc             true   17:30  00h45m
lab_phys              false   18:25  02h40m
study_calc             true   21:05  00h45m

Wednesday
(865/1080)
NAME                    DYN    TIME     DUR
shower                false   06:00  00h30m
study_phys             true   06:30  00h45m
breakfast             false   07:30  00h30m
study_db               true   08:00  00h30m
class_calc            false   08:35  01h15m
study_algs             true   09:50  01h30m
class_algs            false   11:25  01h15m
lunch                 false   12:40  00h30m
study_db               true   13:10  00h30m
lab_calc              false   13:50  01h15m
study_phys             true   15:05  00h45m
study_db               true   15:50  00h30m
dinner                false   16:30  00h30m
study_calc             true   17:00  00h45m
class_csoc            false   18:25  02h40m
study_calc             true   21:05  00h45m

Thursday
(930/1080)
NAME                    DYN    TIME     DUR
shower                false   06:00  00h30m
prj_yacc               true   06:30  00h30m
breakfast             false   07:00  00h30m
prj_zaps               true   07:30  02h00m
prj_yacc               true   09:30  00h30m
class_db              false   10:00  01h15m
lunch                 false   11:20  00h30m
study_db               true   11:50  01h30m
prj_yacc               true   13:20  00h30m
class_phys            false   13:50  01h15m
prj_zaps               true   15:05  01h00m
dinner                false   16:30  00h30m
study_phys             true   17:00  01h30m
study_algs             true   18:30  01h30m
study_calc             true   20:00  01h30m
prj_yacc               true   21:30  00h30m

Friday
(1020/1080)
NAME                    DYN    TIME     DUR
shower                false   06:00  00h30m
prj_yacc               true   06:30  01h00m
breakfast             false   07:30  00h30m
prj_yacc               true   08:00  00h30m
class_calc            false   08:35  01h15m
study_calc             true   09:50  01h30m
class_algs            false   11:25  01h15m
lunch                 false   12:40  00h30m
study_algs             true   13:10  01h30m
prj_yacc               true   14:40  01h00m
prj_zaps               true   15:40  00h30m
dinner                false   16:30  00h30m
study_calc2            true   17:00  01h30m
study_phys             true   18:30  01h30m
study_db               true   20:00  01h30m
prj_zaps               true   21:30  02h00m

Saturday
(930/1080)
NAME                    DYN    TIME     DUR
shower                false   06:00  00h30m
study_calc             true   06:30  02h00m
study_algs             true   08:30  03h00m
brunch                false   11:45  00h30m
study_phys             true   12:15  03h00m
study_calc             true   15:15  01h00m
dinner                false   16:30  00h30m
study_csoc             true   17:00  01h30m
study_db               true   18:30  03h00m
prj_zaps               true   21:30  00h30m

Eating Children

plan is still very much being field-tested by me. I’m making the code
available, but chances are, it will eat your children. I’d appreciate it if
bugs were reported to the github issue tracker.

Implementation Overview

And now onto the technical part: the implementation. You may have taken notice
that I’ve referred to plan as a time allocator. This is because time is a
resource that is allocated in a way that is very similar to the way we allocate
memory (a la interfaces such as malloc). We allocate a contiguous block of
time just as we would allocate a contiguous block of memory (we use a best-fit
algorithm to find a block of time that is free, and allocate it). Similarly,
just as the memory allocation process is susceptible to fragmentation, so is
the time allocation process.

Most operating systems textbooks demonstrate the principals of memory
allocation, but they do not indicate that memory allocation is a subset of
resource allocation. If I showed you how a resource allocator worked, you’d say
that it looks very much like a simplistic memory allocator. You’d be right. Any
resource that can be allocated as a block of integers (like process id’s,
device id’s, virtual addresses) could be allocated with a resource allocator.
Today you could implement a malloc interface using a couple dozen invocations
of a resource allocator’s functions and some system calls.

This only goes to show that modern memory allocators are far, far more complex
than the toy allocators found in systems programming texts. Modern allocators
use very advanced concepts like slabs, magazines, and cache-coloring to
maximize performance, and minimize fragmentation. In other words memory
allocation is a special case of resource allocation.

Instead writing a new resource allocator , I’ve decided to make use of the
existing resource allocator that is used as the back-end of libumem called
vmem. I could’ve used any resource allocator (like the ubiquitous rmalloc),
but seeing as how I use libumem for memory allocation anyway, vmem seemed
like the apparent choice.

So as you can probably tell, plan won’t be able to run on any systems that
don’t support, libumem. Fortunately, someone went through the trouble of
porting libumem over to a bunch of platforms.

The other important aspect I have to mention is data-storage. Apparently we
have to save the schedules we make. Most applications would use an on-disk
format, such as a plain-text file, or an embedded database, like SQLite.

Since I use ZFS on all of my machines, a database is automatically a
non-starter, because as they are modified, the data get fragmented. This is not
ZFS’s fault. The databases were designed with 1980′s era filesystems (which
were not COW or resilient) and data (which was rather small, relative to today)
in mind.

Also, databases love to checksum stuff, which is unnecessary as ZFS already
checksums all of the data. There is also a bunch of other overhead and
redundancy associated with databases, which makes them the wrong choice for
this particular problem.

The other option (plain text file) has to be parsed (using lex and yacc if
the format is complex enough).

I love parsing, but I love efficiency even more. Efficiency in both the
clock-cycle sense, and the programmer-productivity sense. In this case, I
decided to settle on a middle ground between text-file and database. In case I
didn’t make it obvious enough, I’m a die-hard unix developer. Unix has this
great thing called a hierarchical file system. It’s great for representing any
kind of hierarchical data. Unix also snagged the nifty concept of extended file
attributes from BeOS. Manipulating directories and extended attributes is very
easy, using system calls. By storing certain properties such as time and
duration in distinct extended attributes, I have a random-access interface to
these values (which are stored in their original binary representation). I
won’t make any claims to the superior time-efficiency of this solution compared
to the other two, but it sure is more convenient, as I didn’t have to roll a
custom parser (and serializer) or write single line of SQL code. Also, since
the data is structured at the filesystem level I didn’t have to use any tree
structures to hold the data efficiently in memory. The most significant
structures I use are an array, strings, an enum, and structures that hold
information about activities and todos.

So, to use plan you need at least libumem and Illumos extended attributes
(which are accessed via my new favourite system call: openat()). And it’s
preferable to have ZFS.

If you think plan would be useful on platforms that don’t meet the above
prerequisites, grab the code and port it.

And that’s the implementation overview. If you want more concrete details have
a look at the code.

The purpose of life is to do whatever turns you on. -me

Lex and Yacc are two tools that have aided Unix developers in writing
compilers for decades. Yet, these tools are lacking in basic debugging
facilities. This is my attempt to add some DTrace magic to Lex and Yacc.

Two unix tools that have always fascinated me were lex and yacc. Admittedly, it doesn’t take much to fascinate me; in fact, I often
get distracted by shiny stuff.

I’ve always wanted to understand how lex and yacc work, and I’ve wanted to
be able to observe what they’re doing in real time. Not for any particular
reason, aside from the fact that observability turns me on (I’m like that).

Come to think of it, lex and yacc, despite being staples of parsing on unix
systems, don’t have any kind of significant instrumentation at all. For
decades, these tools have been nothing more but generators of magical black
boxes. These black boxes are used all around unix — even DTrace uses lex+yacc
powered parsing for its D language — but no one can actually see what the
generated code is doing (short of tracing the program instruction for
instruction, which is, well, complex).

Apparently they work, and the users and creators of these tools did just fine
without these newfangled kernel-based instrumentation framerworks going around.
In fact, it’s questionable whether or not instrumentation would even be
helpful. Let’s face it: these tools were designed by people smarter than blokes
like us. So maybe this is a matter that should be left alone…

But, there’s still that… itch. That notion that keeps nagging at you while
you’re awake. Sometimes even when you’re asleep (ever dream about DTracing some
cool app; I do all the time). “My. I wonder if baking in some DTrace probes in
$APP_FOO would make it better… nah.” It’s like a mosquito buzzing about
your ear in the middle of the night, waking you up periodically. Maybe you
turn, you try to ignore it, but at some point your baggy, blood shot eyes shoot
open, and BAM! One blow, swift and impulsive. The nuissence is gone, the itch
has been scratched, now you can sleep well at night. Temporarily at least, you
will dream like normal people do. And then it starts all over again…

At any rate, I spent a few sleepless summer nights, fueled by a chaotic
enthusiasm, tracing and instrumenting lex (which, by the way, matches a bunch
of input tokens, using regular expressions, and either returns token values or
does some other specified action). Do a

man lex

for the full details.

Don’t get me wrong. I’d love to hold your hand and walk you through this but
who has the time? Not me, certainly. Callous, perhaps, but there you go.

I used the first example in the man page as a test bed:

%{
/* need this for the call to atof() below */
#include <math.h>
/* need this for printf(), fopen() and stdin below */
#include <stdio.h>
%}

DIGIT    [0-9]
ID       [a-z][a-z0-9]*
%%

{DIGIT}+                          {
    printf("An integer: %s (%d)\n", yytext,
    atoi(yytext));
    }

{DIGIT}+"."{DIGIT}*    {
    printf("A float: %s (%g)\n", yytext,
    atof(yytext));
    }

if|then|begin|end|procedure|function        {
    printf("A keyword: %s\n", yytext);
    }

{ID}                   printf("An identifier: %s\n", yytext);

"+"|"-"|"*"|"/"        printf("An operator: %s\n", yytext);

"{"[^}\n]*"}"         /* eat up one-line comments */

[ \t\n]+               /* eat up white space */

.                      printf("Unrecognized character: %s\n", yytext);

%%

int main(int argc, char *argv[])
{
    ++argv, --argc;  /* skip over program name */
    if (argc > 0)
        yyin = fopen(argv[0], "r");
    else
        yyin = stdin;

    yylex();
}

I began by tracing just the function calls made in the binary itself and
avoided any other module (like libc).

pfexec dtrace -F -c './lexy' -n
    'pid$target:lexy::entry, pid$target:lexy::return {}'

As can be seen below, everything is wrapped in a big function called yylex,
where “everything” refers to the large number of yylook functions.
Unfortunately, yylook takes no arguments and returns no values (I looked at
the source code). We’ll have to drill a little deeper to see what functions are
getting called in yylook. Either way simply using the pid provider is
insufficient to see the inner workings of lex.

A float: 9.9 (9.9)
An integer: 10 (10)
An integer: 5 (5)
A float: 3.333 (3.333)
CPU FUNCTION
  0  -> _start
  0    -> __fsr
  0    <- __fsr
  0    -> main
  0      -> yylex
  0        -> yylook
  0        <- yylook
  0        -> yylook
  0        <- yylook
  0        -> yylook
  0        <- yylook
  0        -> yylook
  0        <- yylook
  0        -> yylook
  0        <- yylook
  0        -> yylook
  0        <- yylook
  0        -> yylook
  0        <- yylook
  0        -> yylook
  0        <- yylook
  0        -> yylook
  0        <- yylook
  0      <- yylex
  0    <- main

In order to get meaningful data about the lexing process, I would have to embed
semantically meaningful probes into the generated lex program. The most
straight-forward way of doing this is to add the probes into the actions
specified for each match. (UPDATE: See this post, this
post, and this post about usdt probes.)

That would be kind of cool, but kind of trivial at the same time. The next
logical step would be to actually look at yylook, and probe the snot out of
it. This would be much cooler considering that lex is really just a state
machine, and I would able to see the states lex goes through as it lexes the
input.

Of course, manually modifying the lex-generated program is, well, manual. So I
did what any lazy, self-respecting computer scientist would do: I modified the
lex program to generate a DTrace source file and an instrumented lexer. All
that needs to be done is the linking at build time. I added a ‘-D’ option to
lex which toggles the generation of a probed lexer plus the D source file.
It’s in my git repo available here. Keep in mind, that lex
is part of Illumos, and you’d have to do a bit of dance involving some scripts
in order get the new and improved lex bits. See the install notes at the at of
this post. Yeah, it’s a pain. But it’s well worth it.

Unless.

Unless, of course, you weren’t one of us. Unless, you were but a mere muggle
watching, awe-struck and slack-jawed as a wizard puts on a show, demonstrating
the wickedness of his magic; making his caprice take its course.

To the wizardly, the inconvenience is worth it. Because magic is an end in and
of itself. Muggles, however, wish to leverage magic to their own low-brow,
pointy-haired purposes.

Enough about muggles. We’re wizards. And all wizards have a craving for one
thing: data.

Data, data, data. Embedded DTrace probes now allowed for far more data to be
collected during the lexing process. I’ll quickly demonstrate what one can do
with the new and improved lex, using a quick-and-dirty lexer for the C
programming language.

Here’s the code for the C lexer:

/* C lexer */
%a 1000
%p 2000
%{
#include "y.tab.h"
/*
 * Used for debugging purposes...
 */
int
indicator()
{
    return (0);
}

/*
 * We define a set of labled states for use in the C lexer, and then
 * some regex's for neatness.
 *
 * The states are:
 *
 * S0 - C code that is not a comment, a string, or a character.
 * S1 - C comment blocks
 * S2 - C line comment
 */
%}

%s S0 S1 S2

RGX_WS          [\f\r\t\v ]*
RGX_WS_DT       [\f\n\r\t\v ]
RGX_NL          \n+
/* C Pre Proc */
RGX_INCL        "#"{RGX_WS}"include"
RGX_INCLLIB     "<"{RGX_PATH}">"
RGX_INCLUSR     \"{RGX_PATH}\"
RGX_DEF         "#"{RGX_WS}"define"
RGX_UNDEF       "#"{RGX_WS}"undef"
RGX_MACIF       "#"{RGX_WS}"if"
RGX_MACELSE     "#"{RGX_WS}"else"
RGX_MACELIF     "#"{RGX_WS}"elif"
RGX_MACENDIF    "#"{RGX_WS}"endif"
RGX_IFDEF       "#"{RGX_WS}"ifdef"
RGX_IFNDEF      "#"{RGX_WS}"ifndef"
RGX_PRAGMA      "#"{RGX_WS}"pragma"
RGX_WARNING     "#"{RGX_WS}"warning"
RGX_ERROR       "#"{RGX_WS}"error"
RGX_LINE        "#"{RGX_WS}"line"
RGX_LINCONT     \\
/* C Num Constants */
RGX_DIGIT       [0-9]
RGX_INT         {RGX_DIGIT}+
RGX_FLOAT       {RGX_DIGIT}+"."{RGX_DIGIT}+
RGX_HEX         0x[0-9a-fA-F]+
/* C Types */
RGX_TCHAR       char
RGX_TINT        int
RGX_TFLOAT      float
RGX_TDBL        double
RGX_TLONG       long
RGX_TSHORT      short
RGX_SIGNED      signed
RGX_UNSIGNED    unsigned
RGX_EXT         extern
RGX_STAT        static
RGX_TYPEDEF     typedef
RGX_STRUCT      struct
RGX_ENUM        enum
RGX_UNI         union
RGX_SO          sizeof
/* C Control Flow*/
RGX_IF          "if"
RGX_THEN        "then"
RGX_ELSE        "else"
RGX_GOTO        "goto"
RGX_RET         "return"
RGX_SWITCH      "switch"
RGX_CASE        "case"
RGX_DEFAULT     "default"
RGX_BREAK       "break"
RGX_CONTIN      "continue"
RGX_LABLE       [a-zA-Z0-9]+:
RGX_DELIM       ";"
RGX_BEGIN       "{"
RGX_END         "}"
RGX_PARENO      "("
RGX_PARENC      ")"
RGX_COMMA       ","
/* C Operators */
RGX_MBR         "."
RGX_PMBR        "->"
RGX_INC         "++"
RGX_DEC         "--"
RGX_MULEQ       "*="
RGX_MULDEREF    "*"
RGX_DIVEQ       "/="
RGX_DIV         "/"
RGX_ADDEQ       "+="
RGX_ADD         "+"
RGX_MINEQ       "-="
RGX_MIN         "-"
RGX_MODEQ       "%="
RGX_MOD         "%"
RGX_TRTHEN      "?"
RGX_TRELSE      ":"
RGX_LAND        "&&"
RGX_LOR         "||"
RGX_LEQ         "=="
RGX_NEQ         "!="
RGX_GTE         ">="
RGX_GT          ">"
RGX_LTE         "<="
RGX_LT          "<"
RGX_BAND        "&"
RGX_BOR         "|"
RGX_XOR         "^"
RGX_NOT         "!"
RGX_RSH         ">>"
RGX_LSH         "<<"
RGX_EQ          "="
RGX_ONECMP      "~"
/* C String Constants */
RGX_WORD        [a-zA-Z0-9]+
RGX_NAME        [a-zA-Z0-9\_]+
RGX_PATH        [a-zA-Z0-9/\-\_\.]+
/* RGX_ASCII    [`-=~!@#$%^&*()_+[]\;',./{}|:"<>?a-zA-Z0-9]+ */
RGX_STR         ([^"\\\n]|\\[^"\n]|\\\")*
RGX_CHR         ([^'\\\n]|\\[^'\n]|\\')*
RGX_CMT         .|\n
%{
%}

%%

<S0>{RGX_NL}            { ; }
<S2>{RGX_NL}            { BEGIN(S0); }
<S0>"/*"                { BEGIN(S1); }
<S1>"*/"                { BEGIN(S0); }
<S1>{RGX_CMT}           { ; }
<S2>{RGX_CMT}           { ; }
<S0>"//"                { BEGIN(S2); }
<S0>{RGX_INCL}          { return (TOK_INCL); }
<S0>{RGX_DEF}           { return (TOK_DEF); }
<S0>{RGX_UNDEF}         { return (TOK_UNDEF); }
<S0>{RGX_MACIF}         { return (TOK_MACIF); }
<S0>{RGX_MACELSE}       { return (TOK_MACELSE); }
<S0>{RGX_MACELIF}       { return (TOK_MACELIF); }
<S0>{RGX_MACENDIF}      { return (TOK_MACENDIF); }
<S0>{RGX_IFDEF}         { return (TOK_IFDEF); }
<S0>{RGX_IFNDEF}        { return (TOK_IFNDEF); }
<S0>{RGX_PRAGMA}        { return (TOK_PRAGMA); }
<S0>{RGX_WARNING}       { return (TOK_WARNING); }
<S0>{RGX_ERROR}         { return (TOK_ERROR); }
<S0>{RGX_LINE}          { return (TOK_LINE); }
<S0>{RGX_LINCONT}       { return (TOK_LINCONT); }
<S0>\<{RGX_PATH}\>      { return (TOK_INCLIB); }
<S0>\"{RGX_PATH}\"      { return (TOK_INCUSR); }
<S0>{RGX_DIGIT}         { return (TOK_DIGIT); }
<S0>{RGX_INT}           { return (TOK_INT); }
<S0>{RGX_FLOAT}         { return (TOK_FLOAT); }
<S0>{RGX_HEX}           { return (TOK_HEX); }
<S0>{RGX_TCHAR}         { return (TOK_TCHAR); }
<S0>{RGX_TINT}          { return (TOK_TINT); }
<S0>{RGX_TFLOAT}        { return (TOK_TFLOAT); }
<S0>{RGX_TDBL}          { return (TOK_TDBL); }
<S0>{RGX_TLONG}         { return (TOK_TLONG); }
<S0>{RGX_TSHORT}        { return (TOK_TSHORT); }
<S0>{RGX_SIGNED}        { return (TOK_SIGNED); }
<S0>{RGX_UNSIGNED}      { return (TOK_UNSIGNED); }
<S0>{RGX_EXT}           { return (TOK_EXT); }
<S0>{RGX_STAT}          { return (TOK_STAT); }
<S0>{RGX_TYPEDEF}       { return (TOK_TYPEDEF); }
<S0>{RGX_STRUCT}        { return (TOK_STRUCT); }
<S0>{RGX_ENUM}          { return (TOK_ENUM); }
<S0>{RGX_UNI}           { return (TOK_UNI); }
<S0>{RGX_SO}            { return (TOK_SO); }
<S0>"["                 { return (TOK_ARRO); }
<S0>"]"                 { return (TOK_ARRC); }
<S0>{RGX_IF}            { return (TOK_IF); }
<S0>{RGX_THEN}          { return (TOK_THEN); }
<S0>{RGX_ELSE}          { return (TOK_ELSE); }
<S0>{RGX_GOTO}          { return (TOK_GOTO); }
<S0>{RGX_RET}           { return (TOK_RET); }
<S0>{RGX_LABLE}         { return (TOK_LABLE); }
<S0>{RGX_DELIM}         { return (TOK_DELIM); }
<S0>{RGX_BEGIN}         { return (TOK_BEGIN); }
<S0>{RGX_END}           { return (TOK_END); }
<S0>{RGX_PARENO}        { return (TOK_PARENO); }
<S0>{RGX_PARENC}        { return (TOK_PARENC); }
<S0>{RGX_COMMA}         { return (TOK_COMMA); }
<S0>{RGX_MBR}           { return (TOK_MBR); }
<S0>{RGX_PMBR}          { return (TOK_PMBR); }
<S0>{RGX_INC}           { return (TOK_INC); }
<S0>{RGX_DEC}           { return (TOK_DEC); }
<S0>{RGX_MULEQ}         { return (TOK_MULEQ); }
<S0>{RGX_MULDEREF}      { return (TOK_MULDEREF); }
<S0>{RGX_DIVEQ}         { return (TOK_DIVEQ); }
<S0>{RGX_DIV}           { return (TOK_DIV); }
<S0>{RGX_ADDEQ}         { return (TOK_ADDEQ); }
<S0>{RGX_ADD}           { return (TOK_ADD); }
<S0>{RGX_MINEQ}         { return (TOK_MINEQ); }
<S0>{RGX_MIN}           { return (TOK_MIN); }
<S0>{RGX_MODEQ}         { return (TOK_MODEQ); }
<S0>{RGX_MOD}           { return (TOK_MOD); }
<S0>{RGX_TRTHEN}        { return (TOK_TRTHEN); }
<S0>{RGX_TRELSE}        { return (TOK_TRELSE); }
<S0>{RGX_LAND}          { return (TOK_LAND); }
<S0>{RGX_LOR}           { return (TOK_LOR); }
<S0>{RGX_LEQ}           { return (TOK_LEQ); }
<S0>{RGX_NEQ}           { return (TOK_NEQ); }
<S0>{RGX_GTE}           { return (TOK_GTE); }
<S0>{RGX_GT}            { return (TOK_GT); }
<S0>{RGX_LTE}           { return (TOK_LTE); }
<S0>{RGX_LT}            { return (TOK_LT); }
<S0>{RGX_BAND}          { return (TOK_BAND); }
<S0>{RGX_BOR}           { return (TOK_BOR); }
<S0>{RGX_XOR}           { return (TOK_XOR); }
<S0>{RGX_NOT}           { return (TOK_NOT); }
<S0>{RGX_RSH}           { return (TOK_RSH); }
<S0>{RGX_LSH}           { return (TOK_LSH); }
<S0>{RGX_EQ}            { return (TOK_EQ); }
<S0>{RGX_ONECMP}        { return (TOK_ONECMP); }
<S0>{RGX_NAME}          { return (TOK_NAME); }
<S0>{RGX_WORD}          { return (TOK_WORD); }
<S0>\"{RGX_STR}\"       |
<S2>\"{RGX_STR}\"       {
                ;
            }
<S0>\'{RGX_CHR}\'       { ; }
<S0>{RGX_WS}    { ; }
%%
int
main(int ac, char *av[])
{
    ++av; --ac;
    if (ac > 0) {
        yyin = fopen(av[0], "r");
    } else {
        yyin = stdin;
    }
    BEGIN(S0);
    while (!feof(yyin)) {
        yylex();
    }
}

We’ll be using this lexer to lex the C source file that’s been generated from
lex code that specifies how to lex the C language.

First the available probes:

provider lex {
    /* Tracks the lexer's state transitions */
    probe state(int);
    /* Indicates which action was taken */
    probe action(int);
    /* Fires whenever a match occures; arg0 is matched str */
    probe match(char*);
    /* The current character */
    probe ch(char);
    /* Fires whenever the lexer backtracks */
    probe fallback();
    /* Fires whenever we enter a compressed state */
    probe compr_st();
    /* Fires whenever the lexer stops to complete a match */
    probe stop();
    /* Fires whenever the user uses the BEGIN macro */
    probe begin(int, void*);
    /* Fires whenever the user uses the REJECT macro */
    probe reject();
};

As you can see, we probe events such as state transitions, matches,
action-execution, backtracking, and the current character.

So the first thing we will do, is a high-resolution state-by-state,
character-by-character, match-by-match trace of the C lexer lexing itself.

The following script, trace.d is used:

lex$target:::match
{
}

lex$target:::reject
{
}

lex$target:::fallback_st
{
    trace(arg0);
}

lex$target:::state
{
    trace(arg0);
}

lex$target:::begin
{
    trace(arg0);
}

lex$target:::fallback_ch,
lex$target:::ch
/arg0 != '\t' && arg0 != '\n' && arg0 != ''/
{
    printf(": '%c'\n", (char) arg0);
}

lex$target:::fallback_ch,
lex$target:::ch
/arg0 == '\t'/
{
    printf(": '\\t'\n");
}

lex$target:::fallback_ch,
lex$target:::ch
/arg0 == '\n'/
{
    printf(": '\\n'\n");
}

lex$target:::fallback_ch,
lex$target:::ch
/arg0 == ''/
{
    printf(": '\'\n");
}

pid$target::yylex:entry
{

}

pid$target::yylex:return
{
    trace(arg1);
}

However, keep in mind that if you’re not using a BEGIN or REJECT macro in any
of your lex code, you’ll have to commend out the parts of the code that enables
those probes.

Running it on the lexer (the clexer binary):

pfexec dtrace -c './clexer lex.yy.c' -s trace.d

Here are the first few lines of output:

CPU     ID                    FUNCTION:NAME
  3  72077                       main:begin                 2
  3  72136                      yylex:entry
  3  72084                     yylook:state                 3
  3  72079                        yylook:ch : '#'

  3  72082               yylook:fallback_st                 2
  3  72084                     yylook:state                12
  3  72079                        yylook:ch : 'i'

  3  72084                     yylook:state                65
  3  72079                        yylook:ch : 'n'

  3  72084                     yylook:state               122
  3  72079                        yylook:ch : 'c'

  3  72084                     yylook:state               156
  3  72079                        yylook:ch : 'l'

  3  72084                     yylook:state               186
  3  72079                        yylook:ch : 'u'

  3  72084                     yylook:state               208
  3  72079                        yylook:ch : 'd'

  3  72084                     yylook:state               223
  3  72079                        yylook:ch : 'e'

  3  72084                     yylook:state               228
  3  72083                     yylook:match
  3  72138                     yylex:return               260
  3  72136                      yylex:entry
  3  72084                     yylook:state                 2
  3  72079                        yylook:ch : ' '

  3  72084                     yylook:state                 8
  3  72079                        yylook:ch : '<'

  3  72083                     yylook:match
  3  72084                     yylook:state                 2
  3  72079                        yylook:ch : '<'

  3  72084                     yylook:state                28
  3  72079                        yylook:ch : 's'
...

As you can see, we also trace the return value of yylex, which is the integer
that represents a particular token. The definitions can be found in y.tab.h
which was generated from an incomplete yacc grammar.

Full y.tab.h:

# define TOK_SPACES 257
# define TOK_NL 258
# define TOK_TB 259
# define TOK_INCL 260
# define TOK_DEF 261
# define TOK_UNDEF 262
# define TOK_MACIF 263
# define TOK_MACELSE 264
# define TOK_MACELIF 265
# define TOK_MACENDIF 266
# define TOK_IFDEF 267
# define TOK_IFNDEF 268
# define TOK_PRAGMA 269
# define TOK_WARNING 270
# define TOK_ERROR 271
# define TOK_LINE 272
# define TOK_LINCONT 273
# define TOK_INCLIB 274
# define TOK_INCUSR 275
# define TOK_DIGIT 276
# define TOK_INT 277
# define TOK_FLOAT 278
# define TOK_HEX 279
# define TOK_TCHAR 280
# define TOK_TINT 281
# define TOK_TFLOAT 282
# define TOK_TDBL 283
# define TOK_TLONG 284
# define TOK_TSHORT 285
# define TOK_SIGNED 286
# define TOK_UNSIGNED 287
# define TOK_EXT 288
# define TOK_STAT 289
# define TOK_TYPEDEF 290
# define TOK_STRUCT 291
# define TOK_ENUM 292
# define TOK_UNI 293
# define TOK_SO 294
# define TOK_ARRO 295
# define TOK_ARRC 296
# define TOK_IF 297
# define TOK_THEN 298
# define TOK_ELSE 299
# define TOK_GOTO 300
# define TOK_RET 301
# define TOK_LABLE 302
# define TOK_DELIM 303
# define TOK_BEGIN 304
# define TOK_END 305
# define TOK_PARENO 306
# define TOK_PARENC 307
# define TOK_COMMA 308
# define TOK_MBR 309
# define TOK_PMBR 310
# define TOK_INC 311
# define TOK_DEC 312
# define TOK_MULEQ 313
# define TOK_MULDEREF 314
# define TOK_DIVEQ 315
# define TOK_DIV 316
# define TOK_ADDEQ 317
# define TOK_ADD 318
# define TOK_MINEQ 319
# define TOK_MIN 320
# define TOK_MODEQ 321
# define TOK_MOD 322
# define TOK_TRTHEN 323
# define TOK_TRELSE 324
# define TOK_LAND 325
# define TOK_LOR 326
# define TOK_LEQ 327
# define TOK_NEQ 328
# define TOK_GTE 329
# define TOK_GT 330
# define TOK_LTE 331
# define TOK_LT 332
# define TOK_BAND 333
# define TOK_BOR 334
# define TOK_XOR 335
# define TOK_NOT 336
# define TOK_RSH 337
# define TOK_LSH 338
# define TOK_EQ 339
# define TOK_ONECMP 340
# define TOK_NAME 341
# define TOK_WORD 342
# define TOK_CHR 343
# define TOK_STR 344

Now we will do a very similar trace, except we will also print timestamps in
the right-most column for particular events like matches, fallbacks, etc,
using the following script, trace_stamp.d:

lex$target:::match
{
    trace(vtimestamp);
}

lex$target:::reject
{
    trace(vtimestamp);
}

lex$target:::fallback_st
{
    trace(arg0);
    trace(vtimestamp);
}

lex$target:::state
{
    trace(arg0);
    trace(vtimestamp);
}

lex$target:::begin
{
    trace(arg0);
    trace(vtimestamp);
}

lex$target:::fallback_ch,
lex$target:::ch
/arg0 != '\t' && arg0 != '\n' && arg0 != ''/
{
    printf(": '%c'\n", (char) arg0);
}

lex$target:::fallback_ch,
lex$target:::ch
/arg0 == '\t'/
{
    printf(": '\\t'\n");
}

lex$target:::fallback_ch,
lex$target:::ch
/arg0 == '\n'/
{
    printf(": '\\n'\n");
}

lex$target:::fallback_ch,
lex$target:::ch
/arg0 == ''/
{
    printf(": '\'\n");
}

pid$target::yylex:entry
{
    trace(vtimestamp);
}

pid$target::yylex:return
{
    trace(arg1);
    trace(vtimestamp);
}

The commenting out of BEGIN or REJECT probes also applies to this script.

And now, the trace:

CPU     ID                    FUNCTION:NAME
  2  72077                       main:begin                 2      833713
  2  72136                      yylex:entry            835272
  2  72084                     yylook:state                 3      849428
  2  72079                        yylook:ch : '#'

  2  72082               yylook:fallback_st                 2      983644
  2  72084                     yylook:state                12      984585
  2  72079                        yylook:ch : 'i'

  2  72084                     yylook:state                65      985769
  2  72079                        yylook:ch : 'n'

  2  72084                     yylook:state               122      986795
  2  72079                        yylook:ch : 'c'

  2  72084                     yylook:state               156      987847
  2  72079                        yylook:ch : 'l'

  2  72084                     yylook:state               186      988972
  2  72079                        yylook:ch : 'u'

  2  72084                     yylook:state               208      990097
  2  72079                        yylook:ch : 'd'

  2  72084                     yylook:state               223      991058
  2  72079                        yylook:ch : 'e'

  2  72084                     yylook:state               228      992023
  2  72083                     yylook:match            993829
  2  72138                     yylex:return               260      995786
  2  72136                      yylex:entry            996406
  2  72084                     yylook:state                 2      996916
  2  72079                        yylook:ch : ' '

  2  72084                     yylook:state                 8      997841
  2  72079                        yylook:ch : '<'

  2  72083                     yylook:match            999122
  2  72084                     yylook:state                 2      999588
  2  72079                        yylook:ch : '<'

  2  72084                     yylook:state                28     1000575
  2  72079                        yylook:ch : 's'
...

So the last two outputs are very detailed, and can be very useful in debugging
the lexer. However, examining a complete trace from top to bottom is a very
tedious, time-consuming task (the full output for each of the above traces is
195,352 lines long).

As an alternative, one can use DTrace to ask a very specific question about the
lexing process as a whole, without having to manually sift through reams of
output.

For instance, we may wish to see how many times a particular token is matched,
using tok_count.d:

pid$target::yylex:return
{
    @[arg1] = count();
}

The output:

0                  1
269                1
287                1
289                1
294                1
317                1
332                1
329                2
274                3
328                3
331                3
324                4
336                4
260                5
299                5
333                5
300                6
323                6
326                6
288                7
263                8
268                9
325               10
264               12
311               12
312               12
280               13
330               13
291               15
310               16
267               18
327               19
261               24
295               25
296               25
320               32
266               35
297               41
281               44
314               60
339               71
301               93
302              102
272              104
275              109
304              151
305              151
306              307
307              307
303              371
318              531
276             1201
341             1433
277             2186
308             3272

We may also wish to see what average lex time per token-type is, using
tok_time_avg.d:

pid$target::yylex:entry
{
    self->ts = vtimestamp;
}

pid$target::yylex:return
{
    @[arg1] = avg(vtimestamp - self->ts);
}

Output of pfexec dtrace -c './clexer lex.yy.c' -s tok_time_avg.d:

308              509
323              512
295              518
296              524
324              534
303              552
320              554
318              559
310              566
312              569
306              570
336              595
311              606
333              631
330              635
327              637
339              638
314              639
276              644
332              660
325              665
328              673
317              678
329              688
277              694
331              699
326              700
305              718
304              767
280              802
302              831
264              836
268              859
266              861
341              879
294              884
287              894
288              906
281              915
297              941
291              946
299              961
300              966
301              973
272             1050
267             1124
263             1257
275             1433
274             1550
307             1641
261             1879
289             5318
0              18105
260            26437
269            66109

So, on average token 269 takes the longest to lex. Looking a y.tab.h, 269 is
TOK_PRAGMA. 260, the runner up, is TOK_INLC, and so forth.

To see what is so slow about parsing a pragma, we’ll use trace_pragma.d a
modified version of trace_stamp.d. trace_pragma.d uses DTrace’s speculative
tracing facility to display the trace data pragma matching and nothing else.

The output of that trace is 6312 lines long. And it only encompasses one
trace.

Here’s the initial screen of text:

CPU     ID                    FUNCTION:NAME
  2  72076                      yylex:entry          61977649
  2  72086                     yylook:state                 2     61978154
  2  72081                        yylook:ch : '\n'

  2  72086                     yylook:state                 9     61979102
  2  72081                        yylook:ch : '/'

  2  72085                     yylook:match          61980055
  2  72086                     yylook:state                 3     61980550
  2  72081                        yylook:ch : '/'

  2  72084               yylook:fallback_st                 2     61981495
  2  72086                     yylook:state                23     61981973
  2  72081                        yylook:ch : '*'

  2  72086                     yylook:state                80     61982925
  2  72085                     yylook:match          61983512
  2  72080                      yylex:begin                 4     61985065
  2  72086                     yylook:state                 4     61985521
  2  72081                        yylook:ch : '\n'

  2  72086                     yylook:state                53     61986638
  2  72085                     yylook:match          61987130
  2  72086                     yylook:state                 5     61987618
  2  72081                        yylook:ch : ' '

  2  72083               yylook:fallback_ch : '\t'

  2  72084               yylook:fallback_st                 4     61990368
  2  72083               yylook:fallback_ch : '\t'

  2  72086                     yylook:state                53     61991362
  2  72085                     yylook:match          61991838
  2  72086                     yylook:state                 4     61992306
  2  72081                        yylook:ch : '*'

  2  72086                     yylook:state                54     61993324
  2  72081                        yylook:ch : ' '

  2  72085                     yylook:match          61994266
  2  72086                     yylook:state                 4     61994728
  2  72081                        yylook:ch : ' '
...

As you can see we started off by parsing a block comment. We don’t finish
parsing the block comment until line 5879 of the output.

The reason yylex takes so long to return on a pragma, is because it never
returns when matching a comment, it just keeps lexing and matching until it
matches something that it has to return, in this case a pragma token.

If that comment wasn’t there, right above the pragma it may have returned
much faster.

Looking at the bottom of the output, we see that matching a pargma token takes
more than 8 microseconds. However, because of the multitiude of probe enablings
in this script, we have to take overhead into account.

Bottom of output:

...
  2  72085                     yylook:match          64343356
  2  72086                     yylook:state                 3     64343869
  2  72081                        yylook:ch : '#'

  2  72084               yylook:fallback_st                 2     64344798
  2  72086                     yylook:state                12     64345273
  2  72081                        yylook:ch : 'p'

  2  72086                     yylook:state                67     64346255
  2  72081                        yylook:ch : 'r'

  2  72086                     yylook:state               124     64347347
  2  72081                        yylook:ch : 'a'

  2  72086                     yylook:state               158     64348283
  2  72081                        yylook:ch : 'g'

  2  72086                     yylook:state               188     64349246
  2  72081                        yylook:ch : 'm'

  2  72086                     yylook:state               209     64350288
  2  72081                        yylook:ch : 'a'

  2  72086                     yylook:state               224     64351204
  2  72085                     yylook:match          64351727
  2  72138                     yylex:return               269     64352499

A better way to see how long lex takes to match a pragma is trace the time
between matches. We will use the final state that occurs before a pragma is
matched (224), to determine that this is the match that we care about.

The script used, match_fst_time_avg.d:

dtrace:::BEGIN
{
    self->prevts = vtimestamp;
    self->curts = 0;
}

lex$target:::state
{
    self->st = arg0;
}

lex$target:::match
{
    self->curts = vtimestamp;
    @[self->st] = avg(self->curts - self->prevts);
    self->prevts = self->curts;
}

The output:

 14             1397
 53             1401
 28             1413
 51             1433
 54             1434
 49             1464
 25             1486
 29             1509
 30             1558
 18             1596
 24             1606
 16             1660
 38             1797
 10             1819
 17             1828
 31             1838
 35             1838
 27             1842
 71             1842
 58             1845
 20             1848
 26             1852
  9             1854
110             1854
 33             1856
 80             1862
 89             1871
 21             1881
 19             1886
 91             1887
 76             1895
111             1897
100             1939
  8             2015
 90             2026
 77             2264
 75             2286
 79             2303
 84             2309
121             2382
 85             2395
137             2505
163             2840
167             2873
161             3090
 72             3318
181             3319
187             3780
 32             3857
213             3860
216             3864
217             3868
218             3883
214             3895
204             3931
206             4094
222             4258
221             4717
224             4898
230             4973
 37             5461
129             6119
116             7140
 60            11786
228           176927

As you can see matching pragmas is actually fast (5 microseconds as opposed to
66).

I rather prefer this script to the tok_time_avg.d because this script’s
measurements reflect the average time between matches, regardless of whether or
not a particular match causes yylex to return.

And the following script associates final states with token types, discarding
the states that have no token type.

tok_fstate.d:

lex$target:::state
{
    self->st = arg0;
}

pid$target::yylex:return
{
    @[arg1, self->st] = count();
}

So, here is the table of tokens and their associated final state:

  0                3                1
269              224                1
287              230                1
289              217                1
294              216                1
317               76                1
332               28                1
329               91                2
274              129                3
328               58                3
331               89                3
324               26                4
336               10                4
260              228                5
299              163                5
333               14                5
300              167                6
323               31                6
326              110                6
288              213                7
263              121                8
268              222                9
325               71               10
264              181               12
311               75               12
312               77               12
280              161               13
330               30               13
291              218               15
341               38               15
310               79               16
267              206               18
327               90               19
261              221               24
295               33               25
296               35               25
320               21               32
266              204               35
297              100               41
281              137               44
314               18               60
339               29               71
301              214               93

And, so as you can see the most expensive match is the token that has final
state 228, which is TOK_INCL (a “#include”).

The cheapest, a bitwise ‘and’ (“&”).

What surpises the hell out of me is that (state 38) TOK_NAME (variable names,
function names, anything that’s not a keyword or operator) gets matched much
faster, on average than TOK_INCL does. TOK_NAME gets parsed in almost 2
microseconds, TOK_INCL gets parsed in 177 microseconds.

Perhaps this is because TOK_INCL‘s get matched at the beginning of the file,
and at that point our clexer binary is just starting execution, and has yet
to have some of its instructions and memory cached. But that’s just a
hypothesis.

Of course, averages get boring after a while, at which point you’d probably
like to move on to using scripts that employ DTrace’s quantize function, like
tok_fst_time_qtz.d:

dtrace:::BEGIN
{
    self->prevts = vtimestamp;
    self->curts = 0;
}

lex$target:::state
{
    self->st = arg0;
}

lex$target:::match
{
    self->curts = vtimestamp;
    @[self->st] = quantize(self->curts - self->prevts);
    self->prevts = self->curts;
}

Tail of output:

...
           84
       value  ------------- Distribution ------------- count
         512 |                                         0
        1024 |@@@@@@@@@@@@@@@@@@@@@@@@@@@              1461
        2048 |@@@@@@@@@@@@@                            723
        4096 |                                         0
        8192 |                                         1
       16384 |                                         0
       32768 |                                         0
       65536 |                                         0
      131072 |                                         1
      262144 |                                         0        

           20
       value  ------------- Distribution ------------- count
         512 |                                         0
        1024 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 3270
        2048 |                                         1
        4096 |                                         1
        8192 |                                         0        

            8
       value  ------------- Distribution ------------- count
         512 |                                         0
        1024 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@     2888
        2048 |@@@@                                     299
        4096 |                                         11
        8192 |                                         2
       16384 |                                         0
       32768 |                                         0
       65536 |                                         0
      131072 |                                         1
      262144 |                                         0        

           32
       value  ------------- Distribution ------------- count
         512 |                                         0
        1024 |@                                        35
        2048 |@@@@@@@@@@@@@@@@@@@                      624
        4096 |@@@@@@@@@@@@@@@@@@@                      619
        8192 |                                         2
       16384 |                                         0
       32768 |                                         0
       65536 |                                         0
      131072 |                                         1
      262144 |                                         0

As far as I know, that’s a bit of dynamic introspection that the lexing process
never had! Cool. Now compiler writers, on Illumos, can shine a much brighter
flashlight at their lexers, illuminating some of the depths of this process.
And this is just the beginning, I hope to add more instrumentation in the
future. For example, instead of having to use final states and returned token
id’s, I’d like to be able to aggregate on the regular expressions defined in
the lex source file.

Though, it just occurred to me that I could use DTrace’s built-in hash table
construct to associate final states with token-types, but oh well. I’ll leave
that as an exercise to the reader.

As for how lex works, it’s quite simple but I’m not into hand-holding, as
mentioned earlier so, if lexing turns you on, check out the code. It’s
relatively straight-foreward, and could even use a few enhancements. Such as
generating code that is prettier to read, removing some of the deficiencies in
the pattern matcher (there are always deficiencies ), and so forth.

Also, there’s a repository with the DTrace scripts I used, available here.

So probing lex was a nice little warm-up. An appetizer. Foreplay, even. Now
for the main course: yacc.

Stay tuned, for part 1.5.

INSTALL NOTES:
* use bldenv to set up some environment variables
* run make
* copy the generated lex binary into wherever you keep your custom bins
* copy lex_probes.d and dtrace_ncform from common to /usr/share/lib/ccs/

Back in my day, I would probe by hand. Now you can get software
that does the job for you. -Kevin Mitnick

My name is Nick and I am a DTrace addict.

Hi Nick

I’m not sure when it started (it’s hard to remember when you’re having fun),
but it was either near the end of high school or the beginning of college, when
I transitioned from Haskell to C, having realized that I had something of a
fetish for systems programming. I had OpenSolaris installed and was aching to
rip into the intricate, gritty, and perversely beautiful mechanisms that make it
sane to use a computer for pretty much everything important.

Initially, I set out to read the source code for various utilities that I use,
ranging from the simple (ls, cp, mv, etc) to the complex (X11, firefox, jvm,
etc). Then I planned on transitioning to infrastructure (libc, libumem, the
kernel, etc).

Needless to say, I figured that just reading the source wasn’t enough to gather
a true understanding of the system in a timely manner. I was also afraid that I
might misunderstand something, and have a completely false intuition of how the
system works, which I find to be a far worse affliction than not knowing at
all.

So, having already chanced upon Bryan Cantrill’s DTrace Review
(Google Tech Talk), I did some extensive research, and decided that DTrace was
going to be an indispensable companion, offering guidance and illuminating the
darkest of tunnels and infrastructural perplexities.

After reading the manual, following along with examples on the blogs of the
creators of DTrace and other DTrace addicts, and field testing the
software on some class projects, I took time away to understand the operation
of the utilities that I use, keeping the source code nearby.

And so, curious reader, the journey began, as I descended into the dark depths
of a gaping abyss, not unlike a potholing version of Indiana Jones. Stopping,
frequently, to turn over rocks and poke anything that looks strange or alive.

So, I started out light, deciding to check out the ls utility. How
complicated could it be?

I had multiple vectors from which I could have started jabbing ls.

I chose to use the syscall provider to get a high-level view of what ls was
asking the system to do.

Here is the one-liner I used on a directory full of source files (in
particular, the root directory of the source for the ls command in Illumos rev 147):

pfexec dtrace -c 'ls' -n 'syscall:::entry /pid == $target/
    {@[probefunc] = count();}'

Here are the results:

  fcntl                                                             1
  getpid                                                            1
  getrlimit                                                         1
  openat                                                            1
  rexit                                                             1
  sysi86                                                            1
  write                                                             1
  getdents64                                                        2
  memcntl                                                           2
  mmap                                                              2
  mmapobj                                                           2
  open                                                              2
  resolvepath                                                       2
  setcontext                                                        2
  ioctl                                                             3
  stat64                                                            3
  fstat64                                                           4
  close                                                             5
  brk                                                               6

So, one can see that brk was run most frequently, followed by close, and so on.

But what parts of the code were calling on the system to do its bidding? The
following one-liner gives the stack-trace of all the userland functions that
were invoking the system calls:

pfexec dtrace -c 'ls' -n 'syscall:::entry /pid == $target/
    {@[probefunc,ustack()] = count();}'

The output of which looks like this:

  brk
          libc.so.1`_brk_unlocked+0x15
          libc.so.1`sbrk+0x35
          libc.so.1`_morecore+0xfc
          libc.so.1`_malloc_unlocked+0x17f
          libc.so.1`malloc+0x35
          ls`xmalloc+0x14
          ls`main+0x35e
          ls`_start+0x7d
        1
  brk
          libc.so.1`_brk_unlocked+0x15
          libc.so.1`sbrk+0x35
          libc.so.1`_morecore+0xfc
          libc.so.1`_malloc_unlocked+0x17f
          libc.so.1`_smalloc+0x50
          libc.so.1`_malloc_unlocked+0x209
          libc.so.1`malloc+0x35
          libc.so.1`strdup+0x26
          libc.so.1`expand_locale_name+0x469
          libc.so.1`setlocale+0x8ab
          ls`main+0x28
          ls`_start+0x7d
        1
  brk
          libc.so.1`_brk_unlocked+0x15
          libc.so.1`sbrk+0x35
          libc.so.1`_morecore+0xfc
          libc.so.1`_malloc_unlocked+0x17f
          libc.so.1`_smalloc+0x50
          libc.so.1`_malloc_unlocked+0x209
          libc.so.1`malloc+0x35
          ls`xmalloc+0x14
          ls`xmemdup+0x16
          ls`xstrdup+0x21
          ls`gobble_file+0xa3c
          ls`print_dir+0x2d0
          ls`main+0x526
          ls`_start+0x7d
        1
  brk
          libc.so.1`_brk_unlocked+0x15
          libc.so.1`sbrk+0x35
          libc.so.1`_morecore+0x2e
          libc.so.1`_malloc_unlocked+0x17f
          libc.so.1`malloc+0x35
          ls`xmalloc+0x14
          ls`main+0x35e
          ls`_start+0x7d
        1
  brk
          libc.so.1`_brk_unlocked+0x15
          libc.so.1`sbrk+0x35
          libc.so.1`_morecore+0x2e
          libc.so.1`_malloc_unlocked+0x17f
          libc.so.1`_smalloc+0x50
          libc.so.1`_malloc_unlocked+0x209
          libc.so.1`malloc+0x35
          libc.so.1`strdup+0x26
          libc.so.1`expand_locale_name+0x469
          libc.so.1`setlocale+0x8ab
          ls`main+0x28
          ls`_start+0x7d
        1
  brk
          libc.so.1`_brk_unlocked+0x15
          libc.so.1`sbrk+0x35
          libc.so.1`_morecore+0x2e
          libc.so.1`_malloc_unlocked+0x17f
          libc.so.1`_smalloc+0x50
          libc.so.1`_malloc_unlocked+0x209
          libc.so.1`malloc+0x35
          ls`xmalloc+0x14
          ls`xmemdup+0x16
          ls`xstrdup+0x21
          ls`gobble_file+0xa3c
          ls`print_dir+0x2d0
          ls`main+0x526
          ls`_start+0x7d
        1
  close
          ld.so.1`__close+0x7
          ld.so.1`file_open+0x3f9
          ld.so.1`find_path+0x155
          ld.so.1`load_so+0xdd
          ld.so.1`load_path+0x52
          ld.so.1`load_one+0x1e7
          ld.so.1`dlmopen_core+0x134
          ld.so.1`dlmopen_intn+0xe0
          ld.so.1`dlmopen_check+0xf1
          ld.so.1`dlopen+0x4a
          libc.so.1`load_locale+0x195
          libc.so.1`setlocale+0x93f
          ls`main+0x28
          ls`_start+0x7d
        1
  close
          ld.so.1`__close+0x7
          ld.so.1`file_open+0x3f9
          ld.so.1`_find_file+0x110
          ld.so.1`find_file+0x1f7
          ld.so.1`load_so+0x340
          ld.so.1`load_path+0x52
          ld.so.1`load_one+0x1e7
          ld.so.1`elf_needed+0x1b1
          ld.so.1`analyze_lmc+0xd3
          ld.so.1`dlmopen_core+0x1d0
          ld.so.1`dlmopen_intn+0xe0
          ld.so.1`dlmopen_check+0xf1
          ld.so.1`dlopen+0x4a
          libc.so.1`load_locale+0x195
          libc.so.1`setlocale+0x93f
          ls`main+0x28
          ls`_start+0x7d
        1
  close
          libc.so.1`__close+0x15
          libc.so.1`fclose+0xaa
          ls`close_stream+0x2d
          ls`close_stdout+0xb0
          libc.so.1`_exithandle+0x63
          libc.so.1`exit+0x12
          ls`_start+0x7d
        1
  close
          libc.so.1`__close+0x15
          libc.so.1`fclose+0xaa
          ls`close_stream+0x2d
          ls`close_stdout+0x15
          libc.so.1`_exithandle+0x63
          libc.so.1`exit+0x12
          ls`_start+0x7d
        1
  close
          libc.so.1`__close+0x15
          libc.so.1`closedir+0x4d
          ls`print_dir+0x327
          ls`main+0x526
          ls`_start+0x7d
        1
  fcntl
          libc.so.1`syscall+0x13
          libc.so.1`fcntl+0x104
          libc.so.1`fdopendir+0x5c
          libc.so.1`opendir+0x3f
          ls`print_dir+0x23
          ls`main+0x526
          ls`_start+0x7d
        1
  fstat64
          libc.so.1`syscall+0x13
          libc.so.1`_findbuf+0x9c
          libc.so.1`_wrtchk+0x5d
          libc.so.1`_fwrite_unlocked+0x4a
          libc.so.1`fwrite+0x53
          ls`quote_name+0x107
          ls`print_name_with_quoting+0x151
          ls`print_file_name_and_frills+0x10a
          ls`print_current_files+0xb3
          ls`print_dir+0x47e
          ls`main+0x526
          ls`_start+0x7d
        1
  fstat64
          libc.so.1`syscall+0x13
          libc.so.1`fdopendir+0x78
          libc.so.1`opendir+0x3f
          ls`print_dir+0x23
          ls`main+0x526
          ls`_start+0x7d
        1
  fstat64
          libc.so.1`syscall+0x13
          libc.so.1`isseekable+0x4b
          libc.so.1`_setbufend+0x36
          libc.so.1`_findbuf+0x148
          libc.so.1`_wrtchk+0x5d
          libc.so.1`_fwrite_unlocked+0x4a
          libc.so.1`fwrite+0x53
          ls`quote_name+0x107
          ls`print_name_with_quoting+0x151
          ls`print_file_name_and_frills+0x10a
          ls`print_current_files+0xb3
          ls`print_dir+0x47e
          ls`main+0x526
          ls`_start+0x7d
        1
  fstat64
          libc.so.1`syscall+0x13
          libc.so.1`isptsfd+0x40
          libc.so.1`xpg4_fixup+0x2e
          libc.so.1`__openat+0x32
          libc.so.1`openat+0x83
          libc.so.1`opendir+0x29
          ls`print_dir+0x23
          ls`main+0x526
          ls`_start+0x7d
        1
  getpid
          libc.so.1`getpid+0x15
          ld.so.1`rt_thr_init+0x40
          ld.so.1`setup+0x15c2
          ld.so.1`_setup+0x310
          ld.so.1`_rt_boot+0x56
          0x8047b48
        1
  getrlimit
          libc.so.1`getrlimit+0x15
          ld.so.1`rt_thr_init+0x40
          ld.so.1`setup+0x15c2
          ld.so.1`_setup+0x310
          ld.so.1`_rt_boot+0x56
          0x8047b48
        1
  ioctl
          libc.so.1`ioctl+0x15
          libc.so.1`_findbuf+0x61
          libc.so.1`_wrtchk+0x5d
          libc.so.1`_fwrite_unlocked+0x4a
          libc.so.1`fwrite+0x53
          ls`quote_name+0x107
          ls`print_name_with_quoting+0x151
          ls`print_file_name_and_frills+0x10a
          ls`print_current_files+0xb3
          ls`print_dir+0x47e
          ls`main+0x526
          ls`_start+0x7d
        1
  ioctl
          libc.so.1`ioctl+0x15
          ls`main+0x97
          ls`_start+0x7d
        1
  ioctl
          libc.so.1`ioctl+0x15
          ls`decode_switches+0x29
          ls`main+0x97
          ls`_start+0x7d
        1
  memcntl
          ld.so.1`memcntl+0x7
          ld.so.1`elf_new_lmp+0x10bb
          ld.so.1`load_file+0x170
          ld.so.1`load_so+0x491
          ld.so.1`load_path+0x52
          ld.so.1`load_one+0x1e7
          ld.so.1`elf_needed+0x1b1
          ld.so.1`analyze_lmc+0xd3
          ld.so.1`dlmopen_core+0x1d0
          ld.so.1`dlmopen_intn+0xe0
          ld.so.1`dlmopen_check+0xf1
          ld.so.1`dlopen+0x4a
          libc.so.1`load_locale+0x195
          libc.so.1`setlocale+0x93f
          ls`main+0x28
          ls`_start+0x7d
        1
  memcntl
          ld.so.1`memcntl+0x7
          ld.so.1`elf_new_lmp+0x10bb
          ld.so.1`load_file+0x170
          ld.so.1`load_so+0x491
          ld.so.1`load_path+0x52
          ld.so.1`load_one+0x1e7
          ld.so.1`dlmopen_core+0x134
          ld.so.1`dlmopen_intn+0xe0
          ld.so.1`dlmopen_check+0xf1
          ld.so.1`dlopen+0x4a
          libc.so.1`load_locale+0x195
          libc.so.1`setlocale+0x93f
          ls`main+0x28
          ls`_start+0x7d
        1
  mmap
          libc.so.1`__systemcall+0x6
          libc.so.1`lmalloc+0xd8
          libc.so.1`__tls_static_mods+0xb1
          ld.so.1`tls_statmod+0x1ca
          ld.so.1`setup+0x15a0
          ld.so.1`_setup+0x310
          ld.so.1`_rt_boot+0x56
          0x8047b48
        1
  mmap
          ld.so.1`mmap+0x7
          ld.so.1`malloc+0x7e
          ld.so.1`calloc+0x25
          ld.so.1`elf_new_lmp+0x284
          ld.so.1`load_file+0x170
          ld.so.1`load_so+0x491
          ld.so.1`load_path+0x52
          ld.so.1`load_one+0x1e7
          ld.so.1`dlmopen_core+0x134
          ld.so.1`dlmopen_intn+0xe0
          ld.so.1`dlmopen_check+0xf1
          ld.so.1`dlopen+0x4a
          libc.so.1`load_locale+0x195
          libc.so.1`setlocale+0x93f
          ls`main+0x28
          ls`_start+0x7d
        1
  mmapobj
          ld.so.1`mmapobj+0x7
          ld.so.1`file_open+0x3e1
          ld.so.1`find_path+0x155
          ld.so.1`load_so+0xdd
          ld.so.1`load_path+0x52
          ld.so.1`load_one+0x1e7
          ld.so.1`dlmopen_core+0x134
          ld.so.1`dlmopen_intn+0xe0
          ld.so.1`dlmopen_check+0xf1
          ld.so.1`dlopen+0x4a
          libc.so.1`load_locale+0x195
          libc.so.1`setlocale+0x93f
          ls`main+0x28
          ls`_start+0x7d
        1
  mmapobj
          ld.so.1`mmapobj+0x7
          ld.so.1`file_open+0x3e1
          ld.so.1`_find_file+0x110
          ld.so.1`find_file+0x1f7
          ld.so.1`load_so+0x340
          ld.so.1`load_path+0x52
          ld.so.1`load_one+0x1e7
          ld.so.1`elf_needed+0x1b1
          ld.so.1`analyze_lmc+0xd3
          ld.so.1`dlmopen_core+0x1d0
          ld.so.1`dlmopen_intn+0xe0
          ld.so.1`dlmopen_check+0xf1
          ld.so.1`dlopen+0x4a
          libc.so.1`load_locale+0x195
          libc.so.1`setlocale+0x93f
          ls`main+0x28
          ls`_start+0x7d
        1
  open
          ld.so.1`syscall+0x5
          ld.so.1`open+0x25
          ld.so.1`file_open+0x3be
          ld.so.1`find_path+0x155
          ld.so.1`load_so+0xdd
          ld.so.1`load_path+0x52
          ld.so.1`load_one+0x1e7
          ld.so.1`dlmopen_core+0x134
          ld.so.1`dlmopen_intn+0xe0
          ld.so.1`dlmopen_check+0xf1
          ld.so.1`dlopen+0x4a
          libc.so.1`load_locale+0x195
          libc.so.1`setlocale+0x93f
          ls`main+0x28
          ls`_start+0x7d
        1
  open
          ld.so.1`syscall+0x5
          ld.so.1`open+0x25
          ld.so.1`file_open+0x3be
          ld.so.1`_find_file+0x110
          ld.so.1`find_file+0x1f7
          ld.so.1`load_so+0x340
          ld.so.1`load_path+0x52
          ld.so.1`load_one+0x1e7
          ld.so.1`elf_needed+0x1b1
          ld.so.1`analyze_lmc+0xd3
          ld.so.1`dlmopen_core+0x1d0
          ld.so.1`dlmopen_intn+0xe0
          ld.so.1`dlmopen_check+0xf1
          ld.so.1`dlopen+0x4a
          libc.so.1`load_locale+0x195
          libc.so.1`setlocale+0x93f
          ls`main+0x28
          ls`_start+0x7d
        1
  openat
          libc.so.1`syscall+0x13
          libc.so.1`openat+0x83
          libc.so.1`opendir+0x29
          ls`print_dir+0x23
          ls`main+0x526
          ls`_start+0x7d
        1
  resolvepath
          ld.so.1`resolvepath+0x7
          ld.so.1`find_path+0x155
          ld.so.1`load_so+0xdd
          ld.so.1`load_path+0x52
          ld.so.1`load_one+0x1e7
          ld.so.1`dlmopen_core+0x134
          ld.so.1`dlmopen_intn+0xe0
          ld.so.1`dlmopen_check+0xf1
          ld.so.1`dlopen+0x4a
          libc.so.1`load_locale+0x195
          libc.so.1`setlocale+0x93f
          ls`main+0x28
          ls`_start+0x7d
        1
  resolvepath
          ld.so.1`resolvepath+0x7
          ld.so.1`_find_file+0x110
          ld.so.1`find_file+0x1f7
          ld.so.1`load_so+0x340
          ld.so.1`load_path+0x52
          ld.so.1`load_one+0x1e7
          ld.so.1`elf_needed+0x1b1
          ld.so.1`analyze_lmc+0xd3
          ld.so.1`dlmopen_core+0x1d0
          ld.so.1`dlmopen_intn+0xe0
          ld.so.1`dlmopen_check+0xf1
          ld.so.1`dlopen+0x4a
          libc.so.1`load_locale+0x195
          libc.so.1`setlocale+0x93f
          ls`main+0x28
          ls`_start+0x7d
        1
  rexit
          libc.so.1`0xfee334e8
          ls`_start+0x7d
        1
  setcontext
          libc.so.1`syscall+0x13
          libc.so.1`libc_init+0x3c8
          ld.so.1`rt_thr_init+0x40
          ld.so.1`setup+0x15c2
          ld.so.1`_setup+0x310
          ld.so.1`_rt_boot+0x56
          0x8047b48
        1
  setcontext
          libc.so.1`__getcontext+0x19
          ld.so.1`rt_thr_init+0x40
          ld.so.1`setup+0x15c2
          ld.so.1`_setup+0x310
          ld.so.1`_rt_boot+0x56
          0x8047b48
        1
  stat64
          ld.so.1`syscall+0x5
          ld.so.1`rtld_stat+0x29
          ld.so.1`file_open+0x100
          ld.so.1`find_path+0x155
          ld.so.1`load_so+0xdd
          ld.so.1`load_path+0x52
          ld.so.1`load_one+0x1e7
          ld.so.1`dlmopen_core+0x134
          ld.so.1`dlmopen_intn+0xe0
          ld.so.1`dlmopen_check+0xf1
          ld.so.1`dlopen+0x4a
          libc.so.1`load_locale+0x195
          libc.so.1`setlocale+0x93f
          ls`main+0x28
          ls`_start+0x7d
        1
  sysi86
          libc.so.1`sysi86+0x15
          ls`_start+0x6e
        1
  write
          libc.so.1`__write+0x15
          libc.so.1`_xflsbuf+0xb4
          libc.so.1`_fflush_u+0x37
          libc.so.1`fclose+0x80
          ls`close_stream+0x2d
          ls`close_stdout+0x15
          libc.so.1`_exithandle+0x63
          libc.so.1`exit+0x12
          ls`_start+0x7d
        1
  getdents64
          libc.so.1`getdents64+0x15
          ls`print_dir+0x2a2
          ls`main+0x526
          ls`_start+0x7d
        2
  stat64
          ld.so.1`syscall+0x5
          ld.so.1`rtld_stat+0x29
          ld.so.1`file_open+0x100
          ld.so.1`_find_file+0x110
          ld.so.1`find_file+0x1f7
          ld.so.1`load_so+0x340
          ld.so.1`load_path+0x52
          ld.so.1`load_one+0x1e7
          ld.so.1`elf_needed+0x1b1
          ld.so.1`analyze_lmc+0xd3
          ld.so.1`dlmopen_core+0x1d0
          ld.so.1`dlmopen_intn+0xe0
          ld.so.1`dlmopen_check+0xf1
          ld.so.1`dlopen+0x4a
          libc.so.1`load_locale+0x195
          libc.so.1`setlocale+0x93f
          ls`main+0x28
          ls`_start+0x7d
        2

This output is pretty cool. Actually I take it back. This is downright awesome!
Within the time frame of a few minutes I was able to see the code-paths taken
to execute all the system calls needed to list a directory. This would have
taken weeks to fully understand, assuming I didn’t give up before then. It is
this level of introspection that makes DTrace a programmer’s wet dream.

Of course it would help to know in what order these system calls are happening.
And it is only a single line of code away:

pfexec dtrace -F -c 'ls' -n 'syscall::: /pid == $target/ {}'

The “=>” indicates that the system entered the system call and “<=” indicated
that the system exited it.

CPU FUNCTION
  0  => mmap
  0  <= mmap
  0  => setcontext
  0  <= setcontext
  0  => getrlimit
  0  <= getrlimit
  0  => getpid
  0  <= getpid
  0  => setcontext
  0  <= setcontext
  0  => sysi86
  0  <= sysi86
  0  => brk
  0  <= brk
  0  => brk
  0  <= brk
  0  => stat64
  0  <= stat64
  0  => resolvepath
  0  <= resolvepath
  0  => open
  0  <= open
  0  => mmapobj
  0  <= mmapobj
  0  => close
  0  <= close
  0  => mmap
  0  <= mmap
  0  => memcntl
  0  <= memcntl
  0  => stat64
  0  <= stat64
  0  => stat64
  0  <= stat64
  0  => resolvepath
  0  <= resolvepath
  0  => open
  0  <= open
  0  => mmapobj
  0  <= mmapobj
  0  => close
  0  <= close
  0  => memcntl
  0  <= memcntl
  0  => ioctl
  0  <= ioctl
  0  => ioctl
  0  <= ioctl
  0  => brk
  0  <= brk
  0  => brk
  0  <= brk
  0  => openat
  0  <= openat
  0  => fstat64
  0  <= fstat64
  0  => fcntl
  0  <= fcntl
  0  => fstat64
  0  <= fstat64
  0  => getdents64
  0  <= getdents64
  0  => brk
  0  <= brk
  0  => brk
  0  <= brk
  0  => getdents64
  0  <= getdents64
  0  => close
  0  <= close
  0  => ioctl
  0  <= ioctl
  0  => fstat64
  0  <= fstat64
  0  => fstat64
  0  <= fstat64
  0  => write
  0  <= write
  0  => close
  0  <= close
  0  => close
  0  <= close
  0  => rexit

So, there are multiple occurrences of some system calls (mmap, brk, etc), and
it would be nice to see the stack traces in chronological order.

Another one-liner to the rescue:

pfexec dtrace -c 'ls' -n 'syscall:::entry /pid == $target/
    {trace(probefunc); ustack();}'

So the first thing ls does is call mmap. Likely the one in the second stack
trace from the top, which is a result of calling setlocale. Which begs the
question, what the heck is setlocale? Luckily it has its own manpage. As much
as I’d hate to admit it, if I hadn’t done this trace, it would have likely been
a long, long time before I even heard of setlocale, let alone use it.
Needless to say I studied the manpage to see if setlocale was of any interest
to me.

So my shiny new toy now makes certain system facilities more discoverable than
they were previously: merely a few command lines away, as opposed to involved
studying of the source, and having to create the stack traces mentally (which
isn’t difficult, if one has a brain, but is time consuming). Not that having
the source isn’t just as cool…

Looking at that stack trace also reveals something else of interest: ld.so.1,
which is used to load an ELF file has its own implementation of malloc and
calloc, based on mmap. It doesn’t use the libc version.

So, I asked my self, now what? Well, it may be a good idea to keep going system
call by system call, and take note of other implementation details of ls. Or
I can go deeper, rappel into the rabit hole, and see what’s up with libc.so
and ld.so.

After all, ls isn’t that interesting: it merely lists directory contents.
The ld shared object and the standard C library are where I saw the first
glimmer of magic. I should dig into mmap and see the kernel magic in all of its
coruscating brilliance.

So, ls patches into the kernel via mmap, after it calls setlocale. It would
be cool to see what functions the kernel executes to carry out an mmap. The
command to carry this out is slightly more involved, and is thus rolled up into
a script, mmap_fbt.d. We trace all the kernel functions triggered by
setlocale‘s call to mmap.

file mmap_fbt.d:

pid$target::setlocale:entry
{
    self->follow = 1;
}

syscall::mmap:entry
/self->follow == 1/
{
    self->follow = 2;
}

syscall::mmap:return
/self->follow == 2/
{
    self->follow = 0;
    exit(0);
}

fbt:::entry,
fbt:::return
/self->follow == 2/
{
    trace(probemod);
}

executing said file:

pfexec dtrace -c 'ls' -s mmap_fbt.d

The output of the above command is a rather detailed map of the kernel code
taken to execute mmap in this particular instance. At this point it may be
useful to pull up a copy of the Illumos source code. I did this on Illumos 147.

CPU FUNCTION
  2  -> smmap32                                 genunix
  2    -> smmap_common                          genunix
  2      -> as_rangelock                        genunix
  2      <- as_rangelock                        genunix
  2      -> zmap                                genunix
  2        -> choose_addr                       genunix
  2          -> map_addr                        unix
  2            -> map_addr_proc                 unix
  2              -> as_gap_aligned              genunix
  2                -> avl_first                 genunix
  2                <- avl_first                 genunix
  2                -> as_findseg                genunix
  2                  -> avl_find                genunix
  2                    -> as_segcompar          genunix
  2                    <- as_segcompar          genunix
  2                  <- avl_find                genunix
  2                <- as_findseg                genunix
  2                -> avl_walk                  genunix
  2                <- avl_walk                  genunix
  2                -> avl_walk                  genunix
  2                <- avl_walk                  genunix
  2                -> valid_va_range_aligned    unix
  2                <- valid_va_range_aligned    unix
  2              <- as_gap_aligned              genunix
  2            <- map_addr_proc                 unix
  2          <- map_addr                        unix
  2        <- choose_addr                       genunix
  2        -> as_map                            genunix
  2          -> as_map_locked                   genunix
  2            -> gethrestime                   genunix
  2              -> pc_gethrestime              unix
  2                -> gethrtime                 genunix
  2                  -> tsc_gethrtime           unix
  2                  <- tsc_gethrtime           unix
  2                <- gethrtime                 genunix
  2              <- pc_gethrestime              unix
  2            <- gethrestime                   genunix
  2            -> as_map_ansegs                 genunix
  2              -> map_pgszcvec                unix
  2                -> map_szcvec                unix
  2                <- map_szcvec                unix
  2              <- map_pgszcvec                unix
  2              -> as_map_segvn_segs           genunix
  2                -> seg_alloc                 genunix
  2                  -> valid_va_range          unix
  2                    -> valid_va_range_aligned   unix
  2                    <- valid_va_range_aligned   unix
  2                  <- valid_va_range          unix
  2                  -> valid_usr_range         unix
  2                  <- valid_usr_range         unix
  2                  -> kmem_cache_alloc        genunix
  2                  <- kmem_cache_alloc        genunix
  2                  -> mutex_init              unix
  2                  <- mutex_init              unix
  2                  -> seg_attach              genunix
  2                    -> as_addseg             genunix
  2                      -> gethrestime         genunix
  2                        -> pc_gethrestime    unix
  2                          -> gethrtime       genunix
  2                            -> tsc_gethrtime   unix
  2                            <- tsc_gethrtime   unix
  2                          <- gethrtime       genunix
  2                        <- pc_gethrestime    unix
  2                      <- gethrestime         genunix
  2                      -> avl_walk            genunix
  2                      <- avl_walk            genunix
  2                      -> avl_insert_here     genunix
  2                        -> avl_insert        genunix
  2                        <- avl_insert        genunix
  2                      <- avl_insert_here     genunix
  2                    <- as_addseg             genunix
  2                  <- seg_attach              genunix
  2                <- seg_alloc                 genunix
  2                -> segvn_create              genunix
  2                  -> anon_resvmem            genunix
  2                    -> rctl_incr_swap        genunix
  2                    <- rctl_incr_swap        genunix
  2                  <- anon_resvmem            genunix
  2                  -> hat_map                 unix
  2                  <- hat_map                 unix
  2                  -> crhold                  genunix
  2                  <- crhold                  genunix
  2                  -> avl_walk                genunix
  2                  <- avl_walk                genunix
  2                  -> avl_walk                genunix
  2                  <- avl_walk                genunix
  2                  -> kmem_cache_alloc        genunix
  2                  <- kmem_cache_alloc        genunix
  2                  -> lgrp_privm_policy_set   unix
  2                    -> lgrp_mem_policy_default   unix
  2                    <- lgrp_mem_policy_default   unix
  2                  <- lgrp_privm_policy_set   unix
  2                <- segvn_create              genunix
  2              <- as_map_segvn_segs           genunix
  2            <- as_map_ansegs                 genunix
  2            -> as_setwatch                   genunix
  2              -> avl_numnodes                genunix
  2              <- avl_numnodes                genunix
  2            <- as_setwatch                   genunix
  2          <- as_map_locked                   genunix
  2        <- as_map                            genunix
  2      <- zmap                                genunix
  2      -> as_rangeunlock                      genunix
  2        -> cv_signal                         genunix
  2        <- cv_signal                         genunix
  2      <- as_rangeunlock                      genunix
  2    <- smmap_common                          genunix
  2  <- smmap32                                 genunix

It’s also a very good idea to generate an index of all the source files like so:

ls -R illumos_gate > ig-ix

In the above output, the kernel is alternating between two modules: unix and
genunix. Genunix contains all the functions that are platform independent,
while unix contains functions that are platform specific. For example,
pc_gethrestime is probably unique to the x86 version of the kernel. As can be
seen, generic code calls non-generic code, and vice versa, all the time. I
think this is an important point. Most undergrad curicula present the design of
software as the layering of one body of code on top of another, as opposed to a
web of functions or modules calling each other at various points of execution.

Tangent 1: Dogmatically layering abstractions on top of each other, because
it’s The Right Way(tm), is a very bad habit to get into, and is an example of
defensive programming. Layering often results in needlessly deep call-stacks,
and code that is not very cache coherent. The kernel needs to be portable, but
it also needs to be fast. The kernel engineers accomplished this by using
conditional compilation, instead of aggressive layering. In fact, in the last
decade or so, the kernel team has been collapsing layers to both improve
performance and relialibility; witness ZFS. Also, Zones are a method of improving the perfomance of virtualization by
removing unneccessary layers (like the CPU emulator).

Tangent 2: To go on a tanget to this tangent, I did some stack traces on
Mercurial, a python application. In this particular instance, I ran
hg log and saved the stack traces to a file. You’ll notice that the default
stack depth of dtrace’s ustack action is too small, and I had to expand it to
150. These stacks are very, very deep. We’re monitoring an application that is
layered on top of the Python VM. Which goes to show that layering can lead to
the problems mentioned above. Not that the Mercurial code is bad; in fact, it
is a very high-quality and one of my favourite revision control systems. It already
performs quite well. I’m just demonstrating that layering results in deep call
stacks which are potentially problematic for some programs.

Here’s the one-liner used on hg log:

pfexec dtrace -c 'hg log' -n 'syscall:::entry /pid == $target/
    {@[ustack(150)] = count();}'

Here’s one of the smaller stack-traces from hg log:

          libc.so.1`__read+0x15
          libc.so.1`_filbuf+0xd3
          libc.so.1`fread+0x118
          libpython2.6.so.1.0`Py_UniversalNewlineFread+0x160
          libpython2.6.so.1.0`file_read+0xeb
          libpython2.6.so.1.0`PyCFunction_Call+0x19d
          libpython2.6.so.1.0`call_function+0x3e6
          libpython2.6.so.1.0`PyEval_EvalFrameEx+0x2cdb
          libpython2.6.so.1.0`fast_function+0x10b
          libpython2.6.so.1.0`call_function+0xe4
          libpython2.6.so.1.0`PyEval_EvalFrameEx+0x2cdb
          libpython2.6.so.1.0`fast_function+0x10b
          libpython2.6.so.1.0`call_function+0xe4
          libpython2.6.so.1.0`PyEval_EvalFrameEx+0x2cdb
          libpython2.6.so.1.0`fast_function+0x10b
          libpython2.6.so.1.0`call_function+0xe4
          libpython2.6.so.1.0`PyEval_EvalFrameEx+0x2cdb
          libpython2.6.so.1.0`fast_function+0x10b
          libpython2.6.so.1.0`call_function+0xe4
          libpython2.6.so.1.0`PyEval_EvalFrameEx+0x2cdb
          libpython2.6.so.1.0`fast_function+0x10b
          libpython2.6.so.1.0`call_function+0xe4
          libpython2.6.so.1.0`PyEval_EvalFrameEx+0x2cdb
          libpython2.6.so.1.0`fast_function+0x10b
          libpython2.6.so.1.0`call_function+0xe4
          libpython2.6.so.1.0`PyEval_EvalFrameEx+0x2cdb
          libpython2.6.so.1.0`PyEval_EvalCodeEx+0x8cb
          libpython2.6.so.1.0`function_call+0x190
          libpython2.6.so.1.0`PyObject_Call+0x67
          libpython2.6.so.1.0`PyObject_CallFunctionObjArgs+0x3d
          libpython2.6.so.1.0`slot_tp_descr_get+0x93
          libpython2.6.so.1.0`PyObject_GenericGetAttr+0x1e7
          libpython2.6.so.1.0`PyObject_GetAttr+0x96
          libpython2.6.so.1.0`PyEval_EvalFrameEx+0x27a4
          libpython2.6.so.1.0`fast_function+0x10b
          libpython2.6.so.1.0`call_function+0xe4
          libpython2.6.so.1.0`PyEval_EvalFrameEx+0x2cdb
          libpython2.6.so.1.0`fast_function+0x10b
          libpython2.6.so.1.0`call_function+0xe4
          libpython2.6.so.1.0`PyEval_EvalFrameEx+0x2cdb
          libpython2.6.so.1.0`PyEval_EvalCodeEx+0x8cb
          libpython2.6.so.1.0`fast_function+0x174
          libpython2.6.so.1.0`call_function+0xe4
          libpython2.6.so.1.0`PyEval_EvalFrameEx+0x2cdb
          libpython2.6.so.1.0`PyEval_EvalCodeEx+0x8cb
          libpython2.6.so.1.0`fast_function+0x174
          libpython2.6.so.1.0`call_function+0xe4
          libpython2.6.so.1.0`PyEval_EvalFrameEx+0x2cdb
          libpython2.6.so.1.0`gen_send_ex+0xbf
          libpython2.6.so.1.0`gen_iternext+0x18
          libpython2.6.so.1.0`PyEval_EvalFrameEx+0x1fec
          libpython2.6.so.1.0`PyEval_EvalCodeEx+0x8cb
          libpython2.6.so.1.0`function_call+0x13d
          libpython2.6.so.1.0`PyObject_Call+0x67
          libpython2.6.so.1.0`ext_do_call+0x156
          libpython2.6.so.1.0`PyEval_EvalFrameEx+0x2d88
          libpython2.6.so.1.0`PyEval_EvalCodeEx+0x8cb
          libpython2.6.so.1.0`function_call+0x13d
          libpython2.6.so.1.0`PyObject_Call+0x67
          libpython2.6.so.1.0`ext_do_call+0x156
          libpython2.6.so.1.0`PyEval_EvalFrameEx+0x2d88
          libpython2.6.so.1.0`PyEval_EvalCodeEx+0x8cb
          libpython2.6.so.1.0`fast_function+0x174
          libpython2.6.so.1.0`call_function+0xe4
          libpython2.6.so.1.0`PyEval_EvalFrameEx+0x2cdb
          libpython2.6.so.1.0`PyEval_EvalCodeEx+0x8cb
          libpython2.6.so.1.0`fast_function+0x174
          libpython2.6.so.1.0`call_function+0xe4
          libpython2.6.so.1.0`PyEval_EvalFrameEx+0x2cdb
          libpython2.6.so.1.0`PyEval_EvalCodeEx+0x8cb
          libpython2.6.so.1.0`fast_function+0x174
          libpython2.6.so.1.0`call_function+0xe4
          libpython2.6.so.1.0`PyEval_EvalFrameEx+0x2cdb
          libpython2.6.so.1.0`fast_function+0x10b
          libpython2.6.so.1.0`call_function+0xe4
          libpython2.6.so.1.0`PyEval_EvalFrameEx+0x2cdb
          libpython2.6.so.1.0`PyEval_EvalCodeEx+0x8cb
          libpython2.6.so.1.0`fast_function+0x174
          libpython2.6.so.1.0`call_function+0xe4
          libpython2.6.so.1.0`PyEval_EvalFrameEx+0x2cdb
          libpython2.6.so.1.0`fast_function+0x10b
          libpython2.6.so.1.0`call_function+0xe4
          libpython2.6.so.1.0`PyEval_EvalFrameEx+0x2cdb
          libpython2.6.so.1.0`fast_function+0x10b
          libpython2.6.so.1.0`call_function+0xe4
          libpython2.6.so.1.0`PyEval_EvalFrameEx+0x2cdb
          libpython2.6.so.1.0`fast_function+0x10b
          libpython2.6.so.1.0`call_function+0xe4
          libpython2.6.so.1.0`PyEval_EvalFrameEx+0x2cdb
          libpython2.6.so.1.0`fast_function+0x10b
          libpython2.6.so.1.0`call_function+0xe4
          libpython2.6.so.1.0`PyEval_EvalFrameEx+0x2cdb
          libpython2.6.so.1.0`PyEval_EvalCodeEx+0x8cb
          libpython2.6.so.1.0`PyEval_EvalCode+0x35
          libpython2.6.so.1.0`run_mod+0x3d
          libpython2.6.so.1.0`PyRun_FileExFlags+0x6e
          libpython2.6.so.1.0`PyRun_SimpleFileExFlags+0x18a
          libpython2.6.so.1.0`PyRun_AnyFileExFlags+0x71
          libpython2.6.so.1.0`Py_Main+0xa9a
          isapython2.6`main+0x66
          isapython2.6`_start+0x7d
           77

Anyway, apologies for the tangents, but it’s a topic I feel strongly about.
Back to the main plot.

Where does one go from here? Study the AVL tree implementation of the kernel?
Read the assembly responsible for memory mapping? Dive into gethrestime,
which would probably lead us into the cyclics subsystem? Perhaps, one should
dig into the kernel memory allocator, entered through kmem_cache_alloc? There
are many starting points, for the budding Illumos kernel engineers out there.

The main contention was whether or not I ought to study the mechanics of mmap
via synthesis or decomposition. Seeing as how I tend to write code,
predominantly, in a bottom-up style, it seems most natural to explore mmap via
synthesis. Also, the way these blog posts are written is an example of
synthesis (translation: I’m winging them).

The inner-most functions in the stack trace are, in chronological order:

• as_rangelock
• avl_first
• as_segcompar
• avl_walk
• valid_va_range_aligned
• tsc_gethrtime
• map_szcvec
• valid_usr_range
• kmem_cache_alloc
• mutex_init
• avl_insert
• rctl_incr_swap
• hat_map
• crhold
• lgrp_mem_policy_default
• avl_numnodes
• cv_signal

It seems that all functions that are prefixed with as_ are functions related
to address space management. The block comments in as.h are good introductory
material to the structures used by these functions (and should be read prior to
continuing). Starting with as_rangelock. It’s implementation is in
vm_as.c:

/*
 * Serialize all searches for holes in an address space to
 * prevent two or more threads from allocating the same virtual
 * address range.  The address space must not be "read/write"
 * locked by the caller since we may block.
 */
void
as_rangelock(struct as *as)
{
    mutex_enter(&as->a_contents);
    while (AS_ISCLAIMGAP(as))
        cv_wait(&as->a_cv, &as->a_contents);
    AS_SETCLAIMGAP(as);
    mutex_exit(&as->a_contents);
}

The first thing I noticed was that mutex_enter and mutex_exit, didn’t
appear in the dtrace output. I suspect this is due to function inlining.
There’s a quick way to check, using mdb, the modular debugger.

Run this:

pfexec mdb -k

A command prompt should pop up.

This disassembles the function.

> as_rangelock::dis
as_rangelock:                   pushq  %rbp
as_rangelock+1:                 movq   %rsp,%rbp
as_rangelock+4:                 subq   $0x8,%rsp
as_rangelock+8:                 movq   %rdi,-0x8(%rbp)
as_rangelock+0xc:               pushq  %r12
as_rangelock+0xe:               pushq  %r13
as_rangelock+0x10:              subq   $0x8,%rsp
as_rangelock+0x14:              movq   %rdi,%r13
as_rangelock+0x17:              call   -0x2e46b4        <mutex_enter>
as_rangelock+0x1c:              movzbl 0x8(%r13),%eax
as_rangelock+0x21:              testl  $0x40,%eax
as_rangelock+0x26:              je     +0x1b    <as_rangelock+0x43>
as_rangelock+0x28:              leaq   0xa(%r13),%r12
as_rangelock+0x2c:              movq   %r12,%rdi
as_rangelock+0x2f:              movq   %r13,%rsi
as_rangelock+0x32:              call   -0x175667        <cv_wait>
as_rangelock+0x37:              movzbl 0x8(%r13),%eax
as_rangelock+0x3c:              testl  $0x40,%eax
as_rangelock+0x41:              jne    -0x17    <as_rangelock+0x2c>
as_rangelock+0x43:              orl    $0x40,%eax
as_rangelock+0x46:              movb   %al,0x8(%r13)
as_rangelock+0x4a:              movq   %r13,%rdi
as_rangelock+0x4d:              call   -0x2e45ca        <mutex_exit>
as_rangelock+0x52:              addq   $0x8,%rsp
as_rangelock+0x56:              popq   %r13
as_rangelock+0x58:              popq   %r12
as_rangelock+0x5a:              leave
as_rangelock+0x5b:              ret
> ::quit

So, suprisingly, the calls and symbols are there. It’s just that DTrace doesn’t
trace them. It turns out, that while DTrace does match mutex_enter and
mutex_exit as valid fbt probes on Illumos 147, it doesn’t trace them
because the lock primitives aren’t instrumented.

Though, it would appear, that the DTrace creators didn’t intend for the probe
to be matched, as indicated in chapter 3, page 64 of the dtrace book.

Also, for those who don’t typically read x86 assembly, reading the above output
would be made easier with an introductory article to x86 assembly and a listing of all x86 instructions.

So mutex_enter claims a mutex, and mutex_exit releases it. Claiming a mutex
creates a mutual exclusion over some part of memory, so that
only one thread can write to that part of memory, while other threads wait
(“block” is a common synonym for wait).

When we do the mutex_enter we are locking down some of the fields of the
address space structure. Then as_rangelock proceeds make a claim over the
address space, if it isn’t already claimed. If it is already claimed,
as_rangelock waits (via cv_wait) for the claim to be withdrawn, and
then claims it, but releases the original mutex. As can be seen in the DTrace
output, we didn’t have to wait.

As is indicated in as.h the address-space structures and meta-data are stored
in an avl tree, and this is visible in the dtrace output.
Some as_ functions make calls to avl_ functions. What’s even more
interesting is that avl_find makes a call to as_segcompar; avl_find
probably takes a callback that it uses for comparison purposes. We’ll find out
later.

The only other as_ function that’s an innermost function is as_segcompar
(according to DTrace).

/*
 * compar segments (or just an address) by segment address range
 */
static int
as_segcompar(const void *x, const void *y)
{
    struct seg *a = (struct seg *)x;
    struct seg *b = (struct seg *)y;

    if (a->s_base < b->s_base)
        return (-1);
    if (a->s_base >= b->s_base + b->s_size)
        return (1);
    return (0);
}

And this turns out to be true. Though I’m not sure why ‘compare’ is misspelled.
I suspect that this is historical, and that the developers of the original unix
were motivited by similar conditions that motivated the unfortunate misspelling
of the creat system call. I’m sure they had really good reasons to do this –
but typeing and reading creat still irks the crap out of me.

So it’s just a simple comparison function.

That’s all for the inner-most as_ functions. Onto avl_ functions.
avl_first is located in usr/src/common/avl.c. Before continuing, give
avl.h a glance.

/*
 * Return the lowest valued node in a tree or NULL.
 * (leftmost child from root of tree)
 */
void *
avl_first(avl_tree_t *tree)
{
    avl_node_t *node;
    avl_node_t *prev = NULL;
    size_t off = tree->avl_offset;

    for (node = tree->avl_root;
        node != NULL;
        node = node->avl_child[0])
        prev = node;

    if (prev != NULL)
        return (AVL_NODE2DATA(prev, off));
    return (NULL);
}

Once one understands the structures and macros in avl.h, the above should be
self explanatory.

Next up, avl_walk.

/*
 * Walk from one node to the previous valued node (ie. an infix walk
 * towards the left). At any given node we do one of 2 things:
 *
 * - If there is a left child, go to it, then to it's rightmost
 * descendant.
 *
 * - otherwise we return thru parent nodes until we've come from a
 * right child.
 *
 * Return Value:
 * NULL - if at the end of the nodes
 * otherwise next node
 */
void *
avl_walk(avl_tree_t *tree, void *oldnode, int left)
{
    size_t off = tree->avl_offset;
    avl_node_t *node = AVL_DATA2NODE(oldnode, off);
    int right = 1 - left;
    int was_child;

    /*
     * nowhere to walk to if tree is empty
     */
    if (node == NULL)
        return (NULL);

    /*
     * Visit the previous valued node. There are two possibilities:
     *
     * If this node has a left child, go down one left, then all
     * the way right.
     */
    if (node->avl_child[left] != NULL) {
        for (node = node->avl_child[left];
            node->avl_child[right] != NULL;
            node = node->avl_child[right])
            ;
    /*
     * Otherwise, return thru left children as far as we can.
     */
    } else {
        for (;;) {
            was_child = AVL_XCHILD(node);
            node = AVL_XPARENT(node);
            if (node == NULL)
                return (NULL);
            if (was_child == right)
                break;
        }
    }

    return (AVL_NODE2DATA(node, off));
}

So this is slightly more involved, though it should be intuitive to those
familiar with AVL Trees. Basically avl_walk attempts to find the biggest
value that is less than the current node we’re in (i.e. void *oldnode).

Continueing with avl_insert. Quickly glossing over the code reveals that
avl_insert isn’t an inner-most function as it might call avl_rotation.

Either way, avl_rotation rotates and rebalances the tree, using new_balance
to calculate how to rotate it. avl_insert merely adds data to the tree as a
new leaf, and then call avl_rotation if a rebalancing is required.

avl_numnodes merely returns the number of nodes in the tree. Accordingly,
it’s a very short function.

/*
 * Return the number of nodes in an AVL tree.
 */
ulong_t
avl_numnodes(avl_tree_t *tree)
{
    ASSERT(tree);
    return (tree->avl_numnodes);
}

So that covers the innermost avl_ and as_ functions. Let’s move up one
layer of functions and have a look at avl_find.

/*
 * Search for the node which contains "value".  The algorithm is a
 * simple binary tree search.
 *
 * return value:
 *      NULL: the value is not in the AVL tree
 *              *where (if not NULL)  is set to indicate the insertion
 *              point
 *      "void *"  of the found tree node
 */
void *
avl_find(avl_tree_t *tree, const void *value, avl_index_t *where)
{
    avl_node_t *node;
    avl_node_t *prev = NULL;
    int child = 0;
    int diff;
    size_t off = tree->avl_offset;

    for (node = tree->avl_root; node != NULL;
        node = node->avl_child[child]) {

        prev = node;

        diff = tree->avl_compar(value, AVL_NODE2DATA(node, off));
        ASSERT(-1 <= diff && diff <= 1);
        if (diff == 0) {
#ifdef DEBUG
            if (where != NULL)
                *where = 0;
#endif
            return (AVL_NODE2DATA(node, off));
        }
        child = avl_balance2child[1 + diff];

    }

    if (where != NULL)
        *where = AVL_MKINDEX(prev, child);

    return (NULL);
}

As can be seen inside the for-loop, we do have a user-defined comparison
function, stored in the tree structure. In the case of the as_ functions,
it’s as_segcompar. Also, this misspelling of ‘compare’ seems to be
pathologically effecting the Illumos code base. Sad day.

And on and on it goes. DTrace a syscall, find the bottom-level functions,
understand them, move up one level, synthesize, repeat.

That’s my modus operandi whenever I want to understand a system call, some part
of the kernel, or even a user land application. It’s not enough to ask what a
function does. I care about how various components interact with each other.

Most people seem content starting from main and working their way down to the
depths of the kernel/app. That’s only because main provides a convenient
starting point in the absence of DTrace (which apparently works wonderfully as
a code-mapper). DTrace gives the developer the ability to see the interplay
between functions, between modules, and between processes.

Sythesizing how a program works is more efficient (IMHO), because one has less
mental state to keep track of and does less mental context switching between
functions. Try it some time.

I’m sure most people think DTrace is exclusively a performance analysis tool,
which is what it was meant to be used as, but I hope I’ve demonstrated that it
is also extremely well suited for edificatory purposes.

Now, go forth and DTrace.