An example scenario
An ambient temperature probe at one site drops out intermittently, so readings arrive as
0. A season of production data is affected, so correcting each value by hand is not practical, and the dropouts will keep happening. You want every affected reading replaced with the average of the preceding week, falling back to the last good reading when that week has nothing to average.Two ways to write a correction
Both write to the same correction history on the data point, and both leave the original reading readable. Use the single-value endpoint when you know the right answer for a specific reading. Use a substitute rule when the same class of problem recurs.
Substitutions are enabled per account. Ask your Mangrove account team to switch them on. Until they are, the value endpoints return
403 and creating a substitute rule returns 422.Anatomy of a substitute rule
A substitute rule is an ordinary data rule with two extra fields. It works on readings that already exist on an event, including ones recorded without a value. A reading ingestion never created at all has no row for a rule to act on, so use the single-value endpoint for those. The condition identifies the readings that need correcting, and the method chain says what to replace them with.
Note three constraints on
subject_slugs:
- It takes exactly one slug. A rule corrects one value, even when its condition reads several.
- The slug has to appear in
rule_text. - Its data point type has to be numeric.
alert_message is optional on a substitute rule. When you leave it out, Mangrove supplies one, used on the readings the chain could not resolve.
Substitution methods
params: value
Replaces the reading with a fixed number. Always resolves, so no method after it can run. Where the condition reads a single value, Mangrove rejects a constant that would itself match it. On a condition reading several values it cannot make that check, so pick the constant with care.
params: slug
Falls back to another data point type (
dpt), copying its reading from the same event. The target has to be a number type, and defined on the same event type as the value being corrected. Declines when the event carries more than one reading of that type.no params
Uses the nearest earlier valid reading of the same type.
no params
Uses the nearest later valid reading of the same type.
params: window
Averages the same type’s readings across the window, rounded to five decimal places.
params: window
Takes the lowest reading of the same type in the window.
params: window
Takes the highest reading of the same type in the window.
trailing_min, trailing_max and the two fills copy an existing reading exactly. trailing_average is the only method that computes a new number. It rounds to five decimal places, using banker’s rounding, so a tie goes to the even digit.
Windows
The trailing methods take awindow of { kind, value, unit }.
A rolling window ends at the reading being corrected, so it only draws on data that already existed when the reading was taken. A calendar window covers its whole period, including the part after the reading. backward_fill reads forward by design. Where a correction has to be defensible as history, use a rolling window and leave backward_fill out of the chain.
How a chain resolves
Methods run in the order you list them and the first one to produce a value wins. A method that cannot produce one is skipped and the next is tried. A value that would match the rule’s own condition is refused, and the chain carries on to the next method. When no method resolves, nothing is written and the reading is flagged with an alert instead. This is the outcomealert_message covers.
Mangrove rejects a chain at creation when it repeats a method with the same parameters, or when a method follows a constant.
Writing the rule
1
Prerequisites
Before starting, make sure you have:
- A Mangrove API token with Data Rules write permission, and Data Collection read permission to read back what changed. Reverting a value needs Data Collection write as well.
- Corrections enabled on your account
- The project ID, and the slug of the data point type you are correcting
2
Create the rule
Post the condition, the subject slug, and the method chain together.A successful create returns
curl
201 and the saved rule, including the friendly ID you will use to read its results.Mangrove queues an evaluation as soon as the rule is saved, so the corrections land shortly after the call returns rather than during it. A rule with no effective_from or effective_to covers the project’s whole history and everything that arrives afterwards. Set either one to bound it, for example to start at a methodology change.Track a long backfill by re-reading the rule’s stats until evaluated stops climbing.3
Check what it corrected
The rule’s stats give you the count without paging through every record.On a substitute rule,
curl
corrected counts the readings the chain resolved. measured counts everything else it evaluated, which includes both the readings that passed the condition and the ones the chain could not resolve, so it is not a clean success-versus-failure split.4
List what it could not correct
On a substitute rule the two result statuses do not mean what they mean on an alert rule, and they do not mean what the app’s filters of the same name mean either.Each of those carries a reason: the chain resolved nothing, the reading sits in a batch in a published report, someone reverted the correction, the write failed, or two substitute rules fired on the same reading.To find the corrections instead, read the results and filter on the per-record
A fired correction is not an alert, so a successful one is recorded as a pass.
status=fail is therefore the readings that need attention:curl
corrected flag, which is true where this version wrote a replacement. There is no query parameter for it, so page through and filter client side. For the count alone, use /stats.A record can be both corrected and fail: the rule corrected it once, a later edit put it back in breach, and a version only ever writes one correction per reading. Widening the window or adding a further method to the chain usually clears most of the unresolved ones. Editing the rule mints a new version and re-evaluates.5
Inspect one corrected value
Each correction is appended to its data point’s history, which you can read per value.A rule-written correction comes back with
curl
source: rule and the rule named in the rule object, alongside the from_value it replaced and the method that won the chain.Reading corrected values back
Every data point read carries its correction state, so you do not have to fetch the history to tell whether a value was changed.
On the correction itself,
origin reads substituted where a reading was changed, imputed where one was filled in, and reverted where a correction was undone. The distinction between the first two matters because some programs cap how much of a reported period may be filled in.
A correction does not recalculate anything downstream. A batch that was already generated keeps the value it was generated with. Data rules do re-evaluate against the corrected value, so a correction can clear an alert or raise a new one.
What is only in the app
Three parts of the substitution surface have no API equivalent:- Rule runs. A rule’s Runs tab lists every evaluation, Evaluate a period re-runs a date range on demand, and Undo run takes back what one run wrote. All three are in Project Settings → Data Rules, on the rule’s detail page.
- Derivation evidence. The stage-by-stage record of which methods were tried and why each was skipped shows in the event drawer. The API returns the winning method, not the full trace.
- Previewing a substitution. The preview endpoint dry-runs the condition and reports which readings would match. It does not compute what the chain would produce.
Undoing a correction
DELETE on the value endpoint reverts the most recent correction and appends a revert entry to the history.
curl