← All capabilities New
ETL Pipelines
DSN is scaling from analysis-on-upload to a full extract-transform-load layer. Teams keep the warehouses and ETL tools they already run, and still land clean datasets where agents, models, and reports can use them.
Native pipelines ingest from a DSN source or a live database, then apply restricted SQL (SELECT … FROM data) and pandas-like Python transforms before saving a new dataset, training a model, or generating a report.
External pipelines store encrypted connector credentials, trigger ADF, Glue, Dataflow, or SnapLogic, wait on a webhook or poll, and ingest the output URL. Cron schedules run in UTC without a separate orchestrator.
What this does for the business
- Stop copying warehouse extracts by hand before every analysis.
- Reuse ADF, Glue, Dataflow, or SnapLogic jobs and still feed DSN agents.
- Schedule nightly clean-and-aggregate jobs that land as queryable sources.
API
POST /api/pipelines POST /api/pipelines/:id/run POST /api/pipelines/connectors Full reference: api.dsnresearch.com/docs