Integrating data

Integrating data makes source systems available to datasets, pipelines, dashboards, reports, agents, and jobs.

Use Datasets for simple file uploads, Connections for reusable source configuration, Secrets for credentials, and Pipelines when ingestion should be repeatable or combined with transformations.

Ways to bring in data

TrueState supports three main ingestion patterns:

• File upload: upload CSV or Excel files into Datasets.
• Connections and integration steps: connect to cloud storage, databases, business applications, sheets, ad platforms, outreach tools, or REST APIs.
• Dataset API: upsert rows into a dataset from an external system.

For file upload details, see Loading Data.

Supported integration sources

Current pipeline and connection configuration supports:

Source	Typical use
AWS S3	Files and data lake exports
Google Cloud Storage	Files and data lake exports
Azure Blob Storage	Files and cloud exports
Google Sheets	Spreadsheet-backed operational data
BigQuery	Query results from BigQuery
Snowflake	Warehouse queries or tables
Microsoft SQL Server	Operational database queries or tables
PostgreSQL	Operational database queries or tables
Microsoft Fabric Warehouse	Fabric SQL warehouse queries
Salesforce	SOQL queries for CRM data
SharePoint	Document or file ingestion workflows
REST API	Custom API extraction
LinkedIn	Campaign and account performance data
Taboola	Campaign performance data
Outbrain	Campaign performance data
Dynamics 365 Business Central	Business Central entities
Instantly	Outreach performance data
Heyreach	Outreach performance data

Available sources can vary by organisation permissions and environment.

Credentials and Secrets

Most integrations require credentials. Store them in Secrets rather than pasting credentials into prompts or instructions.

Common secret types include:

• Username and password
• API token
• OAuth client ID and client secret
• Google service account JSON
• AWS access keys
• Azure storage account credentials
• Google Sheets OAuth
• LinkedIn access token

Use clear names such as snowflake-prod, gcs-marketing-exports, or salesforce-sandbox so other builders can select the right secret later.

Connections

Use Connections when a source should be reusable across workflows. A connection captures the source type, credentials, and source-specific configuration.

Connections are useful when:

• Multiple pipelines need the same source.
• The agent needs to retrieve metadata or inspect available tables/files.
• You want to centralise source configuration rather than duplicate it in every pipeline.

Pipelines

Use a Pipeline when ingestion should be repeatable, transformed, enriched, or delivered as a dataset.

Common patterns:

• Read from S3, GCS, Azure Blob, or Google Sheets and publish a cleaned dataset.
• Query Snowflake, BigQuery, Postgres, SQL Server, or Fabric Warehouse and transform the result.
• Pull campaign data from an ad platform and combine it with CRM data.
• Call a REST API, normalise the response, and store it for dashboarding.

Dataset API

The Dataset API can upsert JSON rows into a dataset.

PUT https://platform-api.truestate.io/datasets/upsert/
api-key: <truestate-api-key>

Example request body:

{
  "dataset_name": "customers",
  "row_unique_identifier": "id",
  "data": [
    { "id": "1", "name": "Acme", "status": "active" },
    { "id": "2", "name": "Beacon", "status": "prospect" }
  ],
  "data_schema": {
    "id": "string",
    "name": "string",
    "status": "string"
  }
}

Fields:

• dataset_name: dataset to create or update.
• row_unique_identifier: unique column used to update existing rows.
• data: array of JSON objects, one object per row.
• data_schema: optional top-level column type mapping.

Valid schema types include INTEGER, FLOAT, STRING, BOOLEAN, and JSON.

Best practices

• Start with file upload for one-off exploration.
• Use Connections and Pipelines for repeated ingestion.
• Store credentials in Secrets.
• Keep source queries as narrow as practical.
• Give output datasets clear names and descriptions.
• Document important source assumptions in Agent → Context.

Next steps