Derived Trackables

A derived trackable looks like any Trackable on every screen, but has no entries of its own: its value each day is worked out when it is read, from a formula over other trackables, and never stored. Change an entry it depends on and it changes with it; there is nothing to recompute or invalidate in storage. It is the derived kind, Derived in the app. It can be a member of tags, and it cannot hold members. How one is made in the form is Derived Authoring.

Seeded examples: Exercised, Any over everything tagged Fitness; Daily Calories, the Sum of what Diet’s entries contained; Hydration, water over body weight; Rest days, 1 minus whether a Set was logged; Good day, in Weight’s Healthy zone and Water’s goal on the same day; Main sleep, the longest of each day’s sleeps.

The formula

The formula is stored as a tree, not as text (DerivedSpec in packages/core/src/derived-spec.ts: { version: 1, root }, with an optional period that can only be day); a text syntax could be laid over it without changing a stored formula. Its nodes:

• const: a number, the same on every day.
• ref: a source read into a daily series by a reduce, which is a Reading. The source is a trackable, read through list steps and a map over its entries; or the members of a tag (members, the whole subtree by default, recursive: false for direct members only).
• binop (+ − × ÷) and neg: arithmetic over aligned series. Two series join on the day, inner: a day missing either side has no value. A series against a number applies it to every day. A result that is not finite is dropped, so ÷0 leaves a gap rather than an infinity.
• window: each day’s value is its operand reduced over a trailing span, of calendar days or of days with a value (below).
• zone: 1 on days a zone’s trackable sat inside the zone, 0 on days it was read and did not (below).

A derived trackable’s own Reading is named by its formula, worked out on every read rather than stored (formulaReduce): a formula of one ref reads by that ref’s reduce (Daily Calories is a Sum, Exercised an Any), a zone is an Any, and arithmetic, which has one value a day and is no sum of anything, reads as Last.

Every node is kept expressible in SQL (a reduce as a GROUP BY aggregate, arithmetic as operators, a ref as a per-day subquery, a window as a window frame or a date spine), so a compiler from the tree to SQL could take over evaluation without changing a stored formula. Evaluation runs in memory (below).

Reading a trackable

A ref to a trackable is the same pipeline as any Reading: list steps over its entries (join intervals, filter, pick one a day), a map of what each entry becomes, then the reduce. The steps, the map, the reduces and what answers a day are in Readings. A map can be a path into the entry (calories, what an amount of an item contained; duration), arithmetic over paths (volume = sets × reps × weight), or an instant read as a time of day. A record entry whose arithmetic does not come to a finite number is skipped, not counted as 0.

Reading through. A derived trackable that is one ref over another trackable’s entries, a view such as Main sleep, can be read through: a ref to it that asks for more than its daily value (steps or a map) reads the underlying entries, the view’s steps first and the asking ref’s after. A day step of the asking ref takes the place of the view’s, keeping the view’s conditions when it names none, and its map gives way to the asking ref’s (readThrough). So Bedtime, the start of Main sleep, and Other sleep, every sleep but Main sleep’s pick, follow Main sleep’s rule when it changes.

Over a derived trackable, a ref reads its computed value, one value a day: Any is a value that is not 0 (any value at all, for a time of day), Count is 1, and every other reduce is the value itself.

Over a tag, the members’ entries read as one list, as Readings sets out; the derived trackable that owns the ref and the tag’s nominated summaries are left out of the members it reads, so a summary inside its tag never reads itself.

Any

Any is the reduce that turns activity into a yes or no per day: whether there was at least one. Over entries it is an entry that is not a Nothing, a No or a zero, so an entry with no value, a moment logged at midnight, counts. Exercised is Any over Fitness’s members: did I log anything under Fitness today, worked out and never logged.

Any is sparse on days nobody answered: 1 − Any(X) has no value on a day with no entry unless an empty day means none for X or a Nothing answers it. A one-day calendar window with Sum fills those days, which is how Rest days reads: 1 − (a Set logged, over a window of one day).

Windows

A window reduces its operand over the period ending each day (windowSeries in packages/core/src/derived-evaluator.ts):

• by days: the last N calendar days. It runs through today and answers on a day anything at all was logged, or a day its operand has a value. A day you logged nothing anywhere is on hold, as it is for a streak, so an app left unopened for a month is not a month of rest. On an answered day with nothing in the window, a Sum or a Count is 0 and any other reduce has no value.
• by values: the last N days that have a value, skipping gaps, so it has a value only on those days.

requireFull holds the window back until it is full, for a series that feeds a record, where a part-full window’s mean could beat a full one by having less data. The mean of the last seven weigh-ins is a window by values with Mean over Weight; “good days in the last 30” is a window by days with Sum over a yes/no.

Zones in a formula

{ node: "zone", zone_id } is 1 on days the zone’s own trackable sat inside it, 0 on days it was read and did not, and absent on days with no reading. The trackable is read by its own Reading, so “in the healthy range” means the same number as on its own chart.

That is what makes conditions over several trackables expressible: a mul of two zone nodes is a logical and, since the join is inner (a day only one side measured is dropped) and 0 absorbs. NOT is 1 − x. There is no OR over zones: the arithmetic has no min or max.

The node names a zone rather than carrying a bound, and takes no operand. A general comparison (value, operator, bound) would put a bare threshold inside the formula; naming the zone means moving its bound moves every formula over it, and the formula reads as “Healthy” rather than 78. A zone belongs to one trackable, so “in this zone” has one subject. A time is compared the same way: Snoozed is a zone on Waketime from 06:30 (Time of Day).

Three things a zone node has to get right, each of which would otherwise fail silently:

• Units. Bounds are kept in the unit they were typed in, and the series is in its dimension’s base unit, so zoneInBase converts the bounds before zoneContains compares, once, beside the rule. It reads the shared unit registry and not the number’s own extra units (Zones).
• Cycles. A formula can reach a trackable through a zone or through a tag’s members. trackableRefsOf is a pure walk and resolves neither, so readsOf (packages/core/src/derived-reads.ts), which has the database, adds the zones’ trackables and the members each tag reference reads (membersRead, the rule the evaluator reads them by) to the cycle check.
• Missing zones. A formula is JSON, so no foreign key protects the zone it names. A write is refused when the zone does not exist; on read, a zone that has since been deleted gives an empty series, as a ref to a deleted trackable does.

Clock readings

A trackable ref can read when instead of what: its map names an instant, at (when it was logged) or an interval’s start or end, and the series is a time. A reading is seconds from the midnight of the day the entry counts for, so a bedtime the evening before is negative and a mean of bedtimes lands where bedtimes are; the frame, and how a typed time is placed in it, are in Time of Day. An open interval reads nothing, the same rule that keeps its length out of every total.

A formula is refused on write, rather than left to evaluate to an empty series that reads as no data, when it asks for: start or end of anything but an interval; at of an interval, whose logged time is its start while open and its end once closed; a clock reading or a time window on a trackable that is not one you log, or that keeps its day alone; or picking among the entries of anything but an interval (checkClockRefs in packages/core/src/trackable-repository.ts, checked through a view to what it reads). Over times, Min is the earliest and Max the latest; Sum is not offered.

Evaluation

createDerivedEvaluator reads the trackables and the profile’s entries once per evaluation, handed in by a page that already holds them or read in one query, and works out every day in memory over entry objects: through the app’s SQL bridge a query costs more than all the arithmetic. A ref to another derived trackable recurses into its formula; derived can read derived as deep as it goes.

A cycle is refused by the two changes that can close one (assertNoFormulaCycle): saving a formula over an existing trackable, and tagging (adding a member, or setting a tag’s members), since a formula reading a tag’s members reads a new member too. Both walk what every formula reads, by name, through zones and through tags. A brand-new trackable cannot close one, since nothing reads it or contains it yet. If one is ever read anyway, the evaluator’s visiting set throws, and the screen leaves only that formula’s chart empty.

A day’s value can carry a sign beside it (Readings): ≥ when a value behind it is unknown or At least, ≤ when one is At most, ~ when one is About. Every node passes its inputs’ signs on to its own days, combined (combineSigns), and what is taken away, negated or divided by is flipped first (flipSign): 2000 minus a day that is at least 500 is at most 1,500.

In the renderer, useReadingSeriesMap (apps/renderer/src/hooks/useTrackableSeriesMap.ts) works out the derived series a screen needs in one query, keyed on the readings and on fingerprints of what they read: the entries’ values (an entry’s routine among them) and the trackables’ formulas and zones. So adding, deleting or editing an entry, moving it into a routine, or saving a formula or a zone recomputes them; changing a tag’s members or a catalog item invalidates them too.

Limits of the engine

• Unit-blind. The engine composes stored base values whatever their dimension; a sum over a tag adds its members’ numbers as they are. The builder names what an aggregate produces in its source’s unit (Derived Authoring), but a Combine of two dimensions is a plain number: unit algebra is not done.
• A day is the grain. period is day; there is no formula over hours.
• Shared time counts twice. A Sum over a tag whose members are intervals adds each length as it is, so an hour of reading while at work is an hour of each, and a day’s total can pass 24 hours. Nothing yet decides which trackable gets the time two intervals share.
• No change-over-time reduces. A delta or a rate of change is not a reduce; a moving average is a window.
• No whole-history value. A reduction of a whole series to one number (lifetime volume) is a record summary over the entries (Presentation Layer), not a formula.

Related

Trackable · Derived Authoring · Readings · Tags · Zones · Time of Day · Value Types · Editor for Several Trackables