Variables
Variables reference the data Mangrove evaluates the expression against. They are wrapped in double curly braces and always carry a prefix naming what kind of thing they point at.{{node.*}}, and a Batch Calculation rule cannot reference {{data-point.*}}. Mismatches are rejected when you save.
What the editor offers follows from that:
- Data Points rules use data point slugs. These rules apply across the whole project, so autocomplete offers every data point type that is captured on an event, and the rule evaluates against any event whose data points satisfy the condition. One rule reads a single event type: a condition referencing types from two of them is rejected when you save, and autocomplete does not narrow the list to stop you writing one. A calculated data point type, one a model computes rather than an event captures, is not offered, and a condition that names one is rejected when you save. Test a computed value with a Batch Calculation rule and a
{{node.<name>}}reference instead. - Batch Calculation rules use nodes in the selected model.
{{ in the rule editor to open autocomplete. Each suggestion shows the variable name and its description, value type, or unit when available, and inserts the full prefixed form for you.
Comparison operators
Logical operators
Combine conditions withAND, OR, and NOT. Use parentheses to group. NOT binds tightest, then AND, then OR.
Range and set operators
A data point with no value never triggers
NOT BETWEEN, NOT IN, or NOT in front of a single check: with no value the check has no answer, and NOT does not give it one. A bracketed group is different. NOT (a OR b) fires on an empty value, because the group as a whole is false. To catch an empty value with a single check, add IS MISSING: {{data-point.ph}} not between 6.0 and 8.0 OR {{data-point.ph}} IS MISSING.
Text operators
String literals use single quotes.
MATCHES patterns use Ruby regular expression syntax, and each match attempt is bounded by a one-second timeout, so keep patterns anchored and avoid ones that backtrack heavily. A pattern without ^ and $ matches anywhere in the text.
Text operators are case-sensitive: contains 'override' does not match Override. To ignore case in a MATCHES pattern, start it with (?i): {{data-point.tracking_id}} matches '(?i)^tmp-'.
There is no NOT MATCHES operator, and the same goes for the other text operators. Negate them with the NOT prefix instead: NOT {{data-point.lot_code}} matches '[A-Z]{2}-[0-9]+'. Like NOT BETWEEN, this does not fire on a data point with no value.
Presence operators
Use these to check whether a value was recorded.false behaves the same way. Test for those with = '' and = false instead.
A value that was never submitted has no data point for the rule to evaluate, so IS MISSING stays out of its reach.
Aggregates
Aggregate functions roll a set of values into a single number. Use one as the left side of a comparison.SUM, AVG, MIN, MAX and ZSCORE need a numeric data point type and are rejected when you save otherwise. COUNT works on any type.
COUNT counts every data point, including one with no value. SUM, AVG, MIN and MAX skip data points with no value, and when none has a value they return nothing and the comparison is false.
By default an aggregate covers only the data points on the event being evaluated. To ask a question that spans events, add a window.
In a Batch Calculation rule, an aggregate over a {{node.*}} reference combines every node in the model that the reference matches. Where several nodes share a name, add the data point type, as in SUM({{node.boiler.steam_flow}}) > 5000. A bare name shared by several nodes is rejected as ambiguous when you save.
To compare an aggregate against another measurement, or to hold two measurements to a tolerance band, see Catching drift between two data points with RATIO.
Windowed aggregates
Add aWITHIN clause inside the aggregate’s parentheses to widen the set from one event to many. Use it for rules about how often something happens, or about a total across a period.
ROLLING
A sliding window centred on the data point being evaluated, reaching the full duration in both directions. For a value captured at a single point in time,ROLLING 7.days therefore spans 14 days: the seven days before it and the seven after. Where the event carries a start and an end, the window reaches out from both, so the span is the event’s own duration plus the window twice. Use it for proximity and anomaly questions, such as flagging a cluster of data points around one point in time.
A
ROLLING duration has to be greater than zero. ROLLING 0.days is rejected when you save.
TRAILING
A window that ends at the data point being evaluated and reaches back the duration, so it never reads a value captured later.TRAILING 1.hour on a value captured at 14:00 covers 13:00 to 14:00, and both ends count, so a value captured at 13:00 is in the window. On a meter reporting every 15 minutes the window holds five values, not four. Where the event carries a start and an end, the window reaches back from the start and ends at the end, so it holds the value’s own span as well as the duration. On hourly totals, TRAILING 1.hour holds two values and TRAILING 3.hours holds four: one more than the hours you wrote. Use it for conditions that only the history up to that point can answer, such as a sensor that has stopped changing.
TRAILING takes the same duration forms as ROLLING, and the duration has to be greater than zero in the same way.
CALENDAR
The calendar bucket the evaluated data point falls in, in your account’s timezone. Use it for cohort and reporting-period questions, such as “no more than one of these per calendar month”.SECOND, MINUTE, HOUR, DAY, WEEK, MONTH, and YEAR. Weeks start on Monday. The sub-hour units suit duplicate detection at a known cadence, such as one data point per clock minute.
Calendar buckets are absolute. February 27 and March 2 sit in different MONTH buckets and never pair, however close together they are. Two data points two seconds apart at 12:00:59 and 12:01:01 fall in different MINUTE buckets for the same reason. Use ROLLING when you care about elapsed time in either direction, TRAILING when only earlier values may count, and CALENDAR when you care about the clock bucket.
What to know before you use one
- The evaluated data point is in the set when the window names its own type.
COUNT(...) = 1is therefore true for a data point with no neighbours, so writeCOUNT(...) > 1to mean “more than just this one”. In a rule that references several types, a value is not a member of a window over a different type. - The preview evaluates windowed aggregates. It resolves each window the same way a saved rule does, evaluating once per data point rather than once per event, so what you see in the preview is what the rule will do. Because a windowed condition costs more to resolve, the preview works through it in smaller batches, which means the sampling note in Create a Data Rule appears on a smaller project than an unwindowed rule would need.
- Windows only work on
{{data-point.*}}variables. AWITHINclause on a{{node.*}}reference is rejected when you save, so Batch Calculation rules cannot use one. - A value covering a span of time is in the window only if it overlaps it. A data point that ends exactly as the window opens, or starts exactly as it closes, is left out. For a meter that reports a total every 15 minutes, a
CALENDAR HOURbucket holds the four totals inside that hour and not the one that ended as it began. A value recorded at a single moment on the window’s edge is in it. A span that crosses a bucket edge overlaps both buckets, so it counts in full in each: a data point from 22:00 on March 31 to 02:00 on April 1 adds its whole value to both March and April. A substitution method’s window follows the same boundary rule. - The rule’s effective dates clamp the window. A
CALENDAR MONTHbucket is cut short if the rule only became effective partway through that month. - You can use more than one windowed aggregate in a rule. Each resolves its own set independently.
- A windowed rule can act on the whole window, not only the value that fired. That is a setting on the rule’s Then step rather than part of the expression. See Acting on the whole window.
Catching a stuck sensor
A sensor frozen on one value, a flatline, keeps reporting on schedule, so nothing about any single value looks wrong. The check is that the lowest and highest values over a trailing window are the same, with a count beside it saying how many identical values it takes to call the sensor stuck.COUNT is required. MIN = MAX is true of a single value, so without it the rule would fire on the first value after every gap. A MIN = MAX test with no windowed COUNT on the same data point is rejected when you save, and the preview rejects it too. The window sets how long the flatline has to last before the rule fires, and the COUNT stops it firing on a lone value after a gap in the data. Size the count against the meter’s reporting interval. Both ends of the window count, so a meter reporting every 15 minutes puts five values in a trailing hour, and the rule first fires when five values in a row are the same: a meter that freezes at 10:30 is first flagged at 11:30. COUNT >= 3 lets that hour be missing up to two values and still count as flat. Raise it to 5 to require every value in the hour.
The rule fires on the first value that has a flat hour behind it, and on each later value that does, for as long as the sensor stays stuck. By default only those values are flagged. To reach the rest of the run (the run is the stretch of repeated values), set What to flag on the Then step, or What to correct on a substitute rule. Repeats in the window takes every value in a fired window that repeats the one before it, and leaves the first value of the run alone, because the sensor was still working when it reported it. The whole run takes every value in the run, including the first and any outside the window. It is offered only when the condition tests MIN = MAX on each data point the rule flags, and a value another rule has already corrected stays with that rule. See Acting on the whole window, and repair a stuck sensor with a substitute rule over the API for the same rule written as a request.
The COUNT may use a longer window than the MIN and MAX, to count values over a day while testing flatness over an hour. The rule then acts only on the data points inside both windows.
Values that alternate, such as 412.6 and 412.7, never trip it, and a flat run of zeros is detected like any other value. Equipment that is legitimately off, such as a flare idle overnight, is also a flat run of zeros, so exclude it with a clause like AND {{data-point.flow_rate}} != 0, or use a window longer than the longest idle stretch you expect.
Spotting an outlier with ZSCORE
ZSCORE returns the z-score of the evaluated value: the number of standard deviations it sits from the average of the window, negative when the value is below the average. It asks how unusual a value is for that data point type. A methane content that is inside its permitted range but two standard deviations away from the month’s typical value is the kind of thing it catches.
ZSCORE always needs a window. Without one it would have only the evaluated value to compare against, so a bare ZSCORE(...) is rejected when you save. Any of the three window modes works.
When the window holds fewer than two values, or every value in it is identical, there is no spread to measure. ZSCORE then returns nothing and the comparison is false, so the rule never fires on a lone value or a flat run.
The evaluated value counts in its own window, which caps how far it can sit from the average. With n values in the window, the z-score can be at most (n − 1) / √n, so > 2 needs at least six values to ever fire, and > 3 needs at least eleven.
A CALENDAR MONTH window reads the whole month, including values captured later. A value that arrives on the second day is first judged against the one value before it, and Mangrove re-checks it as the month fills in, so its result can change after the fact. To require a minimum number of values in the window, add a COUNT over the same window, ZSCORE({{data-point.methane_content}} WITHIN CALENDAR MONTH) NOT BETWEEN -2 AND 2 AND COUNT({{data-point.methane_content}} WITHIN CALENDAR MONTH) >= 10, or use WITHIN TRAILING 30.days, which always holds a full month of history. For a substitute rule built on ZSCORE, see replace an outlier and hold it for approval.
Catching drift between two data points with RATIO
RATIO divides one measurement by another, so a rule can test a data point against a second stream or against its own history. Use it to catch two meters that should agree drifting apart, or a week’s average sliding away from the year’s.
RATIO returns nothing when either side has no value, and when the denominator is zero. The comparison is false in both cases, so a rule built on RATIO does not fire on a gap.
Data capture rate
DATA_CAPTURE_RATE asks how much of the data you expected actually arrived, so a rule can flag a period that came in short rather than a value that looks wrong. It reads a fact about the data point type itself rather than aggregating values, so it goes on the left of a comparison.
DATA_CAPTURE_RATE returns a fraction rather than a percentage, so compare against 0.95, not 95.
The WITHIN clause is required, because a capture rate with no period behind it means nothing. Only two windows resolve, CALENDAR MONTH and CALENDAR YEAR, and CALENDAR YEAR reads the year to date, which is all that exists before the data year closes. ROLLING windows are not accepted.
Time not captured
TIME_NOT_CAPTURED returns how long a period had no data for a data point, as a span of time rather than the fraction DATA_CAPTURE_RATE returns. It is the total time with no data in the period, whether or not the gaps are consecutive. Use it when a limit is written as time, such as “no more than 10 days of missing data a month”.
10.days, 36.hours or 1.week. The units are seconds, minutes, hours, days and weeks, singular or plural. A plain number has no unit, so it is rejected, and a duration can only be compared with TIME_NOT_CAPTURED. It takes WITHIN CALENDAR MONTH or WITHIN CALENDAR YEAR, as DATA_CAPTURE_RATE does, and goes on the left of the comparison.
A slot holding a substituted value counts as not captured, the same as an empty one.
When a capture rate rule is rejected
The data point type’s event type must declare a cadence, since the cadence is what supplies the denominator. A rule referencing a data point type whose event type has no cadence will not save. See Data capture rate for declaring one. Once a rule is saved, a period with no capture rate worked out yet reads as no value rather than as zero, and the comparison is false. A rule never fires merely because the figure is missing. A rule is evaluated when data arrives, so this condition reports on a period once the stream delivers into it. A rule set to run only on demand reports on a period when you evaluate it. To watch a stream that is quiet right now, and to see which slots are missing rather than how many, use the Capture Rate view. These conditions are rejected when you save, and previewing rejects them too. Each applies toTIME_NOT_CAPTURED as well:
Booleans and numbers
- Booleans are written
trueandfalse(case-insensitive). - Numbers are written without quotes. Both integers and decimals work:
0,42,3.14,0.005. Negative numbers are allowed. - Durations are written as a whole number, a dot and a unit, such as
10.days, and are compared only withTIME_NOT_CAPTURED.
Common patterns
These examples assume your project has data points with the slugs shown. Substitute slugs that exist in your project. For these patterns as complete rules, grouped by the failure each one catches and carrying the Then-step settings, see Commonly used rules.Tips for writing good rules
- Phrase the rule as the failure case. A rule with the expression
{{data-point.ph}} not between 6.0 and 8.0triggers when pH is out of range, which is the alert you want. - Validate before saving. Validating previews the matches from the last 90 days. Use it to catch over-eager rules before they generate noise. When nothing matches, a range filter appears with the result so you can look further back.
- Cover every shape of an empty value.
IS MISSINGcatches a data point with no value stored. It misses a text data point holding an empty string and a boolean holdingfalse, so a rule that has to catch those needs each test spelled out with its own variable:{{data-point.lab_sample_id}} IS MISSING OR {{data-point.lab_sample_id}} = ''. - Say why in the rule name. Expressions are terse and a rule carries no separate description field, so the name and the alert message are where you explain what the rule is for.