Before you begin
Prepare your query and choose the event types to create in Mangrove. You’ll also need connection details for your database:
For direct connections, check that your database’s network rules allow access from Mangrove. Contact support for the current IP allowlist.
Add a SQL source
1
Choose your database
In your project, go to Data Inputs > Data Feeds. Click Add Data Feed and select SQL Database, or choose Add source on an existing data feed. For a new feed, name it on Configure Data Feed, then continue to the source configuration.Choose your database under SQL Database. For Microsoft SQL Server, select MSFT SQL Server.
2
Connect
Enter the connection details for your database type, then click Test connection to verify access.
3
Choose what each run fetches
Under What each run fetches, choose:
- Everything, every run to import the full query result each time. Existing records are not replaced or deduplicated.
- Only what’s new to pick up where the previous run stopped. Your query’s variables decide how:
{{cursor_value}}continues after the last value received, and{{window_start}}with{{window_finish}}fetches the next time window. Fill in the cursor fields that appear once the query uses them.
4
Write and test your query
Enter the Data query with the variables for your choice, then click Test query to see up to 100 rows. Check the columns and values; a sync runs the query without the 100-row limit.Click Add a pre-query to look up values the query needs, such as a table name. Pre-queries cannot be combined with time windows. With time windows, Settle lag (seconds) and Maximum window (seconds) appear under Settings.
5
Review the schedule
Before leaving the source configuration, check Schedule. New sources set to Everything, every run start manual. A new source set to Only what’s new is scheduled by default if no other source on the feed is. Turn off Fetch automatically if you’d like to check your first import with a manual sync.
6
Finish setup
For a new feed, upload an example CSV or spreadsheet matching your query’s columns on the Schema step. On Configure Result, select the event types to create, then save the feed. For an existing feed, save the source and check that its columns match the schema and transformations.
7
Run your first sync
Click Sync beside the source in the Sources table, then check its Last run, the run in Data Inputs > Data Feed Runs, and the resulting events. If the import isn’t what you expected, click Revert on the run to remove its events; see Checking progress.To fetch daily, edit the source and turn on Fetch automatically. See Scheduling.
Writing SQL queries
Use double curly braces to insert variables into your data query:
Variables are inserted as plain text. Quote string and timestamp values using your database’s SQL syntax. Use either
{{cursor_value}} or both window variables, never both, and neither with Everything, every run.
Continue from the last value
For an increasing integer ID, set Cursor column toreading_id and use:
0 for IDs starting at 1. The cursor column must be returned by the query, in ascending order; see Continue from the last value for choosing a safe cursor.
Fetch the next time window
Include the start with>= and exclude the end with <:
Using a pre-query
A pre-query’s first row supplies variables to the data query, useful for table names that change over time. For a pre-query returningtable_name and company_id:
Troubleshooting
Connection failed
Connection failed
- Check the connection details and credentials for the selected database type.
- Check that network rules allow Mangrove to reach the database.
- For Secure Share, check that the shared database and schema are accessible to Mangrove.
No rows in query tests
No rows in query tests
- Confirm that the table contains rows matching your query’s filters and initial cursor or window.
- Check that the database user has permission to read the queried tables.
- If you use a pre-query, test it and check the values supplied to the data query.
Cannot save the source
Cannot save the source
- Check the highlighted fields and match the query variables to What each run fetches.
- For time windows, use both window variables, remove any pre-query, and enter an initial cursor with the selected timestamp or epoch format.
- When switching an existing source to windows, choose an explicit starting boundary. See Changing fetch modes.