# Motivation

The idea behind libevtr and evtranalyze is to build a flexible analysis framework
built on the ktr(9) lightweight tracing framework. The best parts about ktr(9) are
that a) it's really non-intrusive when it's on and b) adding new tracepoints is a
low-overhead task. So the idea behind evtr is to preserve and embrace those features
in providing a framework that allows for robust data analysis that doesn't rely on
parsing the output of ktrdump(8). It also needs to provide convenience functions
for common tasks.

# Design

We want to keep the tracing overhead (that might also affect system behaviour) to a
minimum. So we don't add any code to ktr(9) at all. We just consider ktr format
strings as statements in a language that we interpret at postprocessing time. This
language can use variables and data structures (it must; useful statements have
side effects). Our postprocessing code runs after every event is processed and has
access to those variables and data structures defined and modified by the ktr
events.

# Examples

Blah, blah. Let me make this more concrete.

* Let's say there is a ktr event with the format string
    "foo = %d"

then libevtr expands it to an assignment with a concrete value for every event,
keeps track of the value of the integer variable 'foo' and allows us to register
a callback that will be run every time 'foo' is modified. So at the libevtr level,
there's a stream of events, we can register interest in variables and we have access
to all defined variables when our callback code runs after an event is processed.

* Now, a variable is defined just by assigning to it, at which point it also gets its
type. However, complex assignments work too, and implicitly instantiate the objects
they refer to. Example:

    "devices[%p].status = %d"

This defines 'devices' as a hash (if it is not already defined. If it is, its type
must match). Hash keys can be integers or strings. The syntax bar.baz is the same
as bar["baz"], so in the example above, devices[%p] (for a concrete value of %p)
is also a hash. And of course, devices[%p].status is an integer.

* There should be many predefined 'anchor' variables that we can rely on to get things
done. (Most of these aren't implemented ATM, but should be trivial)

At thread switch time, we do:

    cpu[%d].curthread = %p

at thread creation time, we do:

    threads[%p].name = %s

So in our processing code, we can trivially get data about the current thread.

Currently, ktrdump creates pseudo-events for us at the start of the stream for most of
the devices, though this needs to be done in a better way. he current pseudo-event
format is

    "devicenames[%s] = %lx"

(so that the user can use device names to index the devices[] hash).

# Values

We just saw that variables can be assigned values of integer and string types. A lot
of events don't fit that model, however. Consider e.g. interrupts. In this case, we can
use arbitrary structural types (can't be nested atm, but could easily be extended to
allow that). So

    devices[%p].intr = BeginIntr
    devices[%p].intr = EndIntr

{Begin,End}Intr are arbitrary identifiers (inspired by haskell data constructors, hence
I'm refering to them as 'constructors'). They can also have arguments. E.g. for a
write to a device, we could have

    devices[%p].io BeginWrite %p %llu %lu
    devices[%p].io EndWrite %p %llu %lu

where the second argument is the offset and the third is the length of the write request.
The first argument is the address of the request itself. You should be able to specify
which are the significant arguments for pairing constructors. In this case it would be the
first argument (%p).

# Querying

The query interface is what has given me the most trouble here. Stuff I want to support:

* for an integer variable, calculate avg, stddev, plot the data, produce the values over
  time so that they can be easily priocessed with other tools
* for a completion variable, i.e. a variable that is assigned only two constructors, a
  "begin" and an "end" one, calculate avg, min, max, stddev. Plot them. Print info about
  unpaired constructors. This is a very common pattern. Should apply to interrupts, wakeups,
  disk/net I/O requests, etc.
* possibly consider variables that get assigned a set of constructor values. E.g. a device
  status changing between DeviceInitializing, DeviceRunning, DeviceSuspended etc.

The current possibilities for querying are described in evtranalyze(1). I'm thinking it might
make sense to embed lua, use it for the callbacks and expose the variables as lua variables.