Data connectors

Data connectors bring records from your database or API into a data feed, where transformations turn them into events. Add a connector as a source, then click Sync to fetch on demand or turn on daily fetching. Testing a connection or query and previewing a response read from your system without creating events or moving the saved cursor. They do send real requests, so point them at endpoints where reading has no side effects.

​

Choose a connector

​

Choosing what each run fetches

Under What each run fetches, choose one of two options:

• Everything, every run imports the full result each time, adding every record again; earlier imports are not updated or deduplicated.
• Only what’s new picks up where the previous run stopped. Use it whenever your database or API can select records after a value or within a time range.

With Only what’s new, the variables in your SQL query or HTTP request decide how Mangrove keeps its place. Use one approach or the other, not both. Everything, every run uses neither. If fetching fails, the saved cursor stays where it was: fix the cause and click Sync again. If a later step fails, such as a transformation, Mangrove has already stored the fetched data and moved the cursor. Open Data Inputs > Data Feed Runs and click Retry on that run instead of Sync, so its records are processed rather than skipped.

​

Continue from the last value

A cursor marks where the previous fetch stopped. Set Cursor column to the returned column holding this value, and Initial cursor to the value before the first record you want. HTTP API sources also need a Cursor type. Request records after the cursor and return them in ascending cursor order. After a successful fetch, Mangrove saves the cursor column’s value from the last row. An empty result leaves the cursor unchanged. Use a unique, stable value, such as an increasing record ID. If a result ends partway through records sharing a timestamp, the next request can skip the rest of that group. Records added later at or below the saved cursor are also missed.

​

Fetch the next time window

Set Cursor type to a timestamp or Unix epoch type, and Initial cursor to the inclusive start of the first window. For timestamps, include Z or an explicit offset, for example 2026-01-01T00:00:00Z. Each run starts at the saved cursor and ends at the earlier of:

• The current time minus Settle lag (seconds). The default is 300 seconds: a run at noon fetches records up to 11:55.
• The window start plus Maximum window (seconds), if set. Leave it empty for no limit.

Your query or API must include the start, exclude the end and return every record in the window. Mangrove supplies UTC bounds but cannot check how your source filters them. After a successful fetch, the cursor moves to the window’s end, even if the window had no records. Maximum window caps how much each run fetches, not how far a scheduled source gets: when a run of a scheduled source is cut short by the maximum and completes, Mangrove fetches the next window straight away and repeats until it catches up. Each window is transformed and published before the next starts, so catching up is no faster than one uncapped run. A failed run pauses the catch-up; retrying it successfully, the next scheduled fetch, or a Sync starts it again. A manual source fetches one window per Sync. Allow enough settle lag for records to become available. Records arriving after their window was fetched are not picked up automatically.

​

Cursor types

Use the representation your source expects. Time windows offer only the timestamp and epoch types; epoch values must be whole numbers in the selected unit. Opaque text is a value from a returned row, not a pagination token from a response envelope.

​

Scheduling

Turn on Fetch automatically in the source’s Schedule section to fetch daily. Automatic fetches run at 11:59 pm Eastern Time; the section shows that time, and the next run, in your account’s time zone. Click Sync in the Sources table to fetch on demand at any time. A data feed can have at most one scheduled source. If another source holds the schedule, the section names it. Turning on Fetch automatically here and saving moves the schedule to this source, and the other source becomes manual. New HTTP API sources and sources set to Everything, every run start manual. A new SQL source set to Only what’s new takes the schedule if no other source on the feed has it. Check Schedule before saving.

​

Changing fetch modes

Editing Initial cursor does not reset an existing source’s saved progress. When you switch an existing source to time windows, or change its window cursor type, the form asks for Start the next window at (inclusive) and shows the current cursor. Reusing a last-row cursor as an inclusive start can import those records again; a later start can skip records. Enter the boundary in the selected cursor type. Other changes keep or reset progress without asking:

• Switching from time windows back to {{cursor_value}} restarts a SQL source from Initial cursor, which can import history again. An HTTP source continues from the last window’s end, as a timestamp or epoch value.
• Changing Cursor column keeps the saved cursor, which may not suit the new column.

Before syncing after a change like this, check Initial cursor and the Cursor column in the Sources table.

​

Checking progress

The Sources table shows each source’s Last run and its Cursor, the point the next fetch starts from. Open Data Inputs > Data Feed Runs for each run’s details and events. If an import went wrong, click Revert on the run in Data Feed Runs to remove its events. Reverting doesn’t move the cursor back, so with Only what’s new the next sync won’t fetch those records again. With Everything, every run, the next sync imports the full result again, so fix the query or request first. Mangrove keeps the fetched data, though: once you’ve fixed the cause, click Restart on the reverted run to process it again. If a run’s events are already used in a model run, Revert is unavailable and the run says why.

​

Cloning a project

Cloned sources copy the configuration without credentials, saved cursors or schedules. Add any required credentials, review the initial cursor, and run Sync before turning on the schedule. The first fetch starts from the initial cursor, so it may import historical data again.

​

Troubleshooting