> ## Documentation Index
> Fetch the complete documentation index at: https://docs.mangrovesystems.com/llms.txt
> Use this file to discover all available pages before exploring further.
# Correcting many values at once
> Write a substitute rule through the Mangrove API to correct bad or missing readings automatically, and read back what it changed.
This cookbook explains how to correct data at scale. You write a rule that describes which readings are wrong, and the methods Mangrove should try in order to work out a replacement. The rule then corrects every matching reading in its effective period and keeps correcting new ones as they arrive.
An ambient temperature probe at one site drops out intermittently, so readings arrive as `0`. A season of production data is affected, so correcting each value by hand is not practical, and the dropouts will keep happening. You want every affected reading replaced with the average of the preceding week, falling back to the last good reading when that week has nothing to average.
## Two ways to write a correction
| | Single value | Substitute rule |
| ------------------- | ------------------------------------------------------------------ | ----------------------------------------------------- |
| Endpoint | [Correct a data point value](/api-reference/corrections/set-value) | [Create a rule](/api-reference/data-rules/create) |
| Scope | One data point on one event | Every reading matching a condition |
| Applies to new data | No | Yes, while the rule is in its effective period |
| Replacement value | You supply it | Mangrove derives it from a method chain |
| Recorded as | `source: api` | `source: rule`, with the rule named on the correction |
Both write to the same correction history on the data point, and both leave the original reading readable. Use the single-value endpoint when you know the right answer for a specific reading. Use a substitute rule when the same class of problem recurs.
Substitutions are enabled per account. Ask your Mangrove account team to switch them on. Until they are, the value endpoints return `403` and creating a substitute rule returns `422`.
## Anatomy of a substitute rule
A substitute rule is an ordinary data rule with two extra fields. It works on readings that already exist on an event, including ones recorded without a value. A reading ingestion never created at all has no row for a rule to act on, so use the single-value endpoint for those. The condition identifies the readings that need correcting, and the method chain says what to replace them with.
| Field | What it does |
| --------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- |
| `rule_text` | The condition, in the [rule expression language](/data-rules/rule-expressions). A reading that matches is a reading Mangrove will try to correct. |
| `action` | `substitute`. The default is `alert`, which flags the reading and changes nothing. |
| `subject_slugs` | Exactly one data point type slug, naming which value the rule corrects. |
| `method_chain` | The ordered list of methods to try. |
| `target_type` | `data_point`. Batch calculation rules cannot substitute. |
Note three constraints on `subject_slugs`:
* It takes exactly one slug. A rule corrects one value, even when its condition reads several.
* The slug has to appear in `rule_text`.
* Its data point type has to be numeric.
`alert_message` is optional on a substitute rule. When you leave it out, Mangrove supplies one, used on the readings the chain could not resolve.
## Substitution methods
Replaces the reading with a fixed number. Always resolves, so no method after it can run. Where the condition reads a single value, Mangrove rejects a constant that would itself match it. On a condition reading several values it cannot make that check, so pick the constant with care.
Falls back to another data point type (`dpt`), copying its reading from the same event. The target has to be a number type, and defined on the same event type as the value being corrected. Declines when the event carries more than one reading of that type.
Uses the nearest earlier valid reading of the same type.
Uses the nearest later valid reading of the same type.
Averages the same type's readings across the window, rounded to five decimal places.
Takes the lowest reading of the same type in the window.
Takes the highest reading of the same type in the window.
The three trailing methods and both fills draw only on readings that pass the rule and have not themselves been corrected, so a chain never builds a value out of other substitutions. Both fills also decline where several readings tie at the winning timestamp, rather than let the correction depend on the order the readings were recorded in.
They also draw only on readings carrying the **same tracking ID** as the one being corrected. On a project that captures a tracking ID per delivery or per ticket, that usually leaves no candidates at all, and the rule alerts instead of correcting. If a rule corrects nothing and you expected it to, check whether tracking IDs are set on the events it reads.
Two caps bound the source data: a window draws on at most 1000 readings nearest the one being corrected, and a fill looks back or forward at most 100 readings. A very wide window over high-frequency data is averaged over the nearest 1000 readings rather than everything in the period. `trailing_min`, `trailing_max` and the two fills copy an existing reading exactly. `trailing_average` is the only method that computes a new number. It rounds to five decimal places, using banker's rounding, so a tie goes to the even digit.
### Windows
The trailing methods take a `window` of `{ kind, value, unit }`.
A `rolling` window ends at the reading being corrected, so it only draws on data that already existed when the reading was taken. A `calendar` window covers its whole period, including the part after the reading. `backward_fill` reads forward by design. Where a correction has to be defensible as history, use a rolling window and leave `backward_fill` out of the chain.
| `kind` | Meaning | Units |
| ---------- | ------------------------------------------------------------------ | ---------------------------------------------------------- |
| `rolling` | The span immediately before the reading. Needs a positive `value`. | `second`, `minute`, `hour`, `day`, `week`, `month`, `year` |
| `calendar` | The reading's own calendar period. Takes no `value`. | `day`, `week`, `month`, `quarter`, `year` |
### How a chain resolves
Methods run in the order you list them and the first one to produce a value wins. A method that cannot produce one is skipped and the next is tried. A value that would match the rule's own condition is refused, and the chain carries on to the next method.
When no method resolves, nothing is written and the reading is flagged with an alert instead. This is the outcome `alert_message` covers.
Mangrove rejects a chain at creation when it repeats a method with the same parameters, or when a method follows a `constant`.
Two substitute rules matching the same reading correct nothing. Mangrove will not choose between competing rules, so where several fire on one reading it applies none of them and both report the reading as unresolved. The alert names the rules that collided, so start there. Before adding a rule, check that no existing rule already covers the same values, and if a rule's corrected count comes in lower than you expect, an overlap is the first thing to look for.
## Writing the rule
Before starting, make sure you have:
* A Mangrove API token with **Data Rules** write permission, and **Data Collection** read permission to read back what changed. Reverting a value needs **Data Collection** write as well.
* Corrections enabled on your account
* The project ID, and the slug of the data point type you are correcting
Post the condition, the subject slug, and the method chain together.
```bash curl theme={null}
curl -X POST "https://app.gomangrove.com/api/v1/projects/prj_W55PYc6xVeXb2iuT/rules" \
-H "Authorization: YOUR_API_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"rule": {
"name": "Fill ambient temperature dropouts",
"target_type": "data_point",
"action": "substitute",
"rule_text": "{{data-point.ambient-temperature-c}} = 0",
"subject_slugs": ["ambient-temperature-c"],
"method_chain": [
{
"method": "trailing_average",
"params": { "window": { "kind": "rolling", "value": 7, "unit": "day" } }
},
{ "method": "forward_fill" }
]
}
}'
```
A successful create returns `201` and the saved rule, including the friendly ID you will use to read its results.
Mangrove queues an evaluation as soon as the rule is saved, so the corrections land shortly after the call returns rather than during it. A rule with no `effective_from` or `effective_to` covers the project's whole history and everything that arrives afterwards. Set either one to bound it, for example to start at a methodology change.
Track a long backfill by re-reading the rule's stats until `evaluated` stops climbing.
The rule's stats give you the count without paging through every record.
```bash curl theme={null}
curl "https://app.gomangrove.com/api/v1/projects/prj_W55PYc6xVeXb2iuT/rules/rule_Qb3k9TzdL0pWnXyz/stats" \
-H "Authorization: YOUR_API_TOKEN"
```
On a substitute rule, `corrected` counts the readings the chain resolved. `measured` counts everything else it evaluated, which includes both the readings that passed the condition and the ones the chain could not resolve, so it is not a clean success-versus-failure split.
On a substitute rule the two result statuses do not mean what they mean on an alert rule, and they do not mean what the app's filters of the same name mean either.
| `status` | On an alert rule | On a substitute rule |
| -------- | ----------------------- | ------------------------------------------------------------------------------------------ |
| `fail` | The records it flagged | The records it fired on and did **not** correct, where the value still trips the condition |
| `pass` | The records that passed | Everything else: the records that never matched, and the ones it corrected |
A fired correction is not an alert, so a successful one is recorded as a pass. `status=fail` is therefore the readings that need attention:
```bash curl theme={null}
curl "https://app.gomangrove.com/api/v1/projects/prj_W55PYc6xVeXb2iuT/rules/rule_Qb3k9TzdL0pWnXyz/results?status=fail&page_size=100" \
-H "Authorization: YOUR_API_TOKEN"
```
Each of those carries a reason: the chain resolved nothing, the reading sits in a batch in a published report, someone reverted the correction, the write failed, or two substitute rules fired on the same reading.
To find the corrections instead, read the results and filter on the per-record `corrected` flag, which is true where this version wrote a replacement. There is no query parameter for it, so page through and filter client side. For the count alone, use `/stats`.
The app's **Evaluated Records** filter relabels the same two statuses: **Substituted** maps to `fail` and **Measured** to `pass`, the opposite way round from this endpoint. A count taken from the app will not match a count taken from the API unless you account for that.
A record can be both `corrected` and `fail`: the rule corrected it once, a later edit put it back in breach, and a version only ever writes one correction per reading. Widening the window or adding a further method to the chain usually clears most of the unresolved ones. Editing the rule mints a new version and re-evaluates.
Each correction is appended to its data point's history, which you can read per value.
```bash curl theme={null}
curl "https://app.gomangrove.com/api/v1/projects/prj_W55PYc6xVeXb2iuT/events/evt_DLnYvzbjSujNAvXE/data_points/ambient-temperature-c/corrections" \
-H "Authorization: YOUR_API_TOKEN"
```
A rule-written correction comes back with `source: rule` and the rule named in the `rule` object, alongside the `from_value` it replaced and the `method` that won the chain.
## Reading corrected values back
Every data point read carries its correction state, so you do not have to fetch the history to tell whether a value was changed.
| Field | Meaning |
| -------------- | --------------------------------------------------- |
| `substituted` | The value in effect is a correction. |
| `imputed` | The correction filled a reading that was missing. |
| `raw_value` | The original reading. Null where none was recorded. |
| `substitution` | The correction now in effect, or null. |
On the correction itself, `origin` reads `substituted` where a reading was changed, `imputed` where one was filled in, and `reverted` where a correction was undone. The distinction between the first two matters because some programs cap how much of a reported period may be filled in.
A rule never corrects a reading that has gone into a batch in a published report. Those are skipped and reported as skipped, so a rule run over a period you have already reported changes less than its match count suggests.
A correction does not recalculate anything downstream. A batch that was already generated keeps the value it was generated with. Data rules do re-evaluate against the corrected value, so a correction can clear an alert or raise a new one.
## What is only in the app
Three parts of the substitution surface have no API equivalent:
* **Rule runs.** A rule's **Runs** tab lists every evaluation, **Evaluate a period** re-runs a date range on demand, and **Undo run** takes back what one run wrote. All three are in **Project Settings → Data Rules**, on the rule's detail page.
* **Derivation evidence.** The stage-by-stage record of which methods were tried and why each was skipped shows in the event drawer. The API returns the winning method, not the full trace.
* **Previewing a substitution.** The [preview endpoint](/api-reference/data-rules/preview) dry-runs the condition and reports which readings would match. It does not compute what the chain would produce.
## Undoing a correction
`DELETE` on the value endpoint reverts the most recent correction and appends a `revert` entry to the history.
```bash curl theme={null}
curl -X DELETE "https://app.gomangrove.com/api/v1/projects/prj_W55PYc6xVeXb2iuT/events/evt_DLnYvzbjSujNAvXE/data_points/ambient-temperature-c/value" \
-H "Authorization: YOUR_API_TOKEN"
```
Only the latest correction can be reverted, and a revert cannot itself be reverted. Reverting a value by hand also stops rules from correcting that value again.
Deleting the rule does not undo anything it wrote. Corrections outlive the rule that made them by design, and a rule that has written one cannot be deleted.
The error you get says the corrections must be reversed first. Reverting them does not release the rule: a revert appends a new entry and leaves the original correction in place, so the rule stays undeletable. Treat a substitute rule that has run as permanent, and stop it by setting its **Effective to** date instead.
To put individual values back, revert them one at a time as above.
## Troubleshooting
Validation messages come back prefixed with the field they are about.
| Response | Meaning | What to do |
| ----------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------- |
| `422 Method chain must list at least one method for substitute rules` | `action` is `substitute` with no methods. | Add at least one method, or set `action` to `alert`. |
| `422 Subject slugs must name exactly one data point to correct (got 2)` | More than one slug, or none. | Name the single value the rule corrects. |
| `422 Subject slugs must be data points the rule references (ambient-temperature-c)` | The slug does not appear in `rule_text`. | Correct one of the values the condition reads. |
| `422 Subject slugs must be number data points to be corrected (transport-mode)` | The slug points at text, a date, or a boolean. | Substitution derives numbers. Use an alert rule instead. |
| `422 Action substitute rules must target data points` | `target_type` is `model`. | Batch calculation rules can only alert. |
| `422` naming corrections as not enabled | Substitutions are switched off on the account. The rules endpoints report this with an internal setting name in the text; the value endpoints report it as `403 Corrections are not enabled`. | Ask your Mangrove account team to enable them. |
| `422` naming an existing rule | Another active rule has the same condition over an overlapping period. | Edit the existing rule rather than adding a second. |
| `403 You are not authorized to perform this action` | The token lacks the permission the endpoint needs. | Tick **Data Rules** for the rules endpoints, and **Data Collection** for the value endpoints, in **Account Admin → API**. |