BigQuery Data Transfer Service: Examples, Security and Limits

BigQuery Data Transfer Service (DTS) is Google Cloud's managed way to load data into BigQuery on a schedule: from Google services such as Google Ads and YouTube, from Cloud Storage, Amazon S3 and Azure Blob Storage, from databases and warehouses such as Amazon Redshift, Snowflake and Teradata, and from SaaS platforms such as Salesforce. This guide gives working bq examples for Cloud Storage and Amazon S3, the permissions and service accounts a transfer needs, the limits that break transfers in production, and the point where a custom pipeline is the better tool.

Key Takeaways

• DTS automates scheduled, managed loads into BigQuery. It also runs scheduled queries and dataset copies, but it cannot move data out of BigQuery.
• A transfer runs every 24 hours by default. Cloud Storage transfers can run as often as every 15 minutes or on Pub/Sub events; Amazon S3 transfers at most once an hour.
• People who create transfers need BigQuery Admin or a custom role with seven permissions, and the DTS service agent's role must never be revoked.
• A Cloud Storage transfer fails when a file's schema differs from the destination table, and it takes at most 10,000 files per run.
• Once data lands, standard BigQuery storage and query pricing applies.
• Use DTS when a supported connector can load a source as it is. Build a pipeline when you need transformation before the load, a source DTS cannot reach, or fresher data than its schedule allows.

What Is BigQuery Data Transfer Service?

In Google's words, the service "lets you automate scheduled, managed data transfers into BigQuery". You configure a transfer once, with a source, a destination dataset and a schedule, and DTS runs it and keeps a history of its runs. The same service starts backfills for past dates, copies datasets and runs scheduled queries. It only loads data in: it cannot transfer data out of BigQuery.

A transfer configuration lives in the same location as its destination dataset. DTS can load from any region your data is stored in to any location where the destination dataset is, though location affects the price of Cloud Storage transfers (see the cost section below).

Which Sources Does It Support?

| Group | Sources | |---|---| | Cloud storage | Amazon S3, Azure Blob Storage, Cloud Storage | | Databases and warehouses | Amazon Redshift, Apache Hive Metastore, Microsoft SQL Server (preview), MySQL, Oracle, PostgreSQL, Snowflake, Teradata | | Google services | Campaign Manager, Display & Video 360, Google Ad Manager, Google Ads, Google Analytics 4, Google Play, Search Ads 360, YouTube Channel, YouTube Content Owner; Google Merchant Center and CSS Center (preview) | | Marketing and payments | Facebook Ads; HubSpot, Klaviyo, Mailchimp, PayPal, Shopify and Stripe (preview) | | SaaS platforms | Salesforce, Salesforce Marketing Cloud, ServiceNow |

A preview connector can change before it is generally available, so read its page before a production pipeline depends on it.

BigQuery Data Transfer Service Example: Cloud Storage to BigQuery

The bq command-line tool creates a transfer with bq mk --transfer_config. This one loads CSV files from a bucket into an existing orders table:

bq mk \
  --transfer_config \
  --project_id=my-project \
  --data_source=google_cloud_storage \
  --display_name="Daily orders from Cloud Storage" \
  --target_dataset=raw_orders \
  --service_account_name=dts-loader@my-project.iam.gserviceaccount.com \
  --params='{
    "data_path_template": "gs://my-bucket/orders/*.csv",
    "destination_table_name_template": "orders",
    "file_format": "CSV",
    "write_disposition": "APPEND",
    "skip_leading_rows": "1"
  }'

What each part does:

• --data_source=google_cloud_storage picks the connector. The other required flags are --transfer_config, --display_name, --target_dataset and --params; --project_id, --service_account_name and --destination_kms_key are optional.
• data_path_template can hold wildcards anywhere in the object name, but a wildcard does not expand a directory.
• file_format takes CSV (the default), JSON, AVRO, PARQUET or ORC.
• write_disposition is APPEND or MIRROR. With APPEND, an unmodified file is loaded only once, and a file whose last modification time changes is loaded again. MIRROR overwrites the table on every run.
• The destination table has to exist before you create the transfer.

The bq tool cannot set a schedule or notifications. A transfer created this way runs on the default, every 24 hours, until you change its schedule in the Google Cloud console.

To load past dates, start a backfill with bq mk --transfer_run, giving either --run_time or a --start_time and --end_time, and the transfer's resource name. One backfill request covers at most 180 days, and requests cannot run at the same time, so split a long history into 180-day requests and start each when the last one finishes.

Example: Amazon S3 to BigQuery

bq mk \
  --transfer_config \
  --project_id=my-project \
  --data_source=amazon_s3 \
  --display_name="Daily events from S3" \
  --target_dataset=raw_events \
  --params='{
    "data_path": "s3://my-bucket/events/*.parquet",
    "destination_table_name_template": "events",
    "file_format": "PARQUET",
    "access_key_id": "AWS_ACCESS_KEY_ID",
    "secret_access_key": "AWS_SECRET_ACCESS_KEY"
  }'

The S3 connector differs from Cloud Storage in four ways:

• It takes an AWS access key ID and secret access key in its parameters, and its path key is data_path, not data_path_template.
• Amazon S3 matches prefixes, not wildcards, so files outside your pattern can still be picked up, which Google warns "could result in excess Amazon S3 outbound data transfer costs". Give each transfer's files a prefix of their own.
• Google asks for, at a minimum, the AWS managed policy AmazonS3ReadOnlyAccess on the source data. A dedicated IAM user that can read only the transfer's bucket keeps the key's reach small; rotate its keys as AWS recommends.
• It runs at most once an hour, cannot read from AWS GovCloud (us-gov) regions, cannot load into BigQuery Omni locations, and the bucket part of its URI cannot be parameterized.

Scheduling, Backfills and Run Notifications

A recurring transfer runs every 24 hours unless you choose otherwise. The console offers hours, days, weeks, months, a custom schedule or on-demand only, starting now or at a set time. Custom schedules use the cron-style syntax Google documents for App Engine, such as every 3 hours or every monday 09:00.

The shortest interval is 15 minutes for Cloud Storage and one hour for Amazon S3. Cloud Storage transfers can also be event-driven, started by Pub/Sub notifications, for data that should land soon after it arrives.

Three operating details matter more than they look:

• Run history is kept for 90 days. Older runs are deleted from it, so keep your own record if you need a longer trail.
• Failure emails go to one person. They go to the transfer's administrator, the account that set it up, and you cannot change the recipient or the content.
• Pub/Sub notifications cover every finished run. They fire for SUCCEEDED, FAILED and CANCELLED runs, with the event type TRANSFER_RUN_FINISHED. Google recommends them when a transfer runs as a service account, whose email nobody reads. Setting them up needs pubsub.topics.getIamPolicy and pubsub.topics.setIamPolicy, and Pub/Sub bills for them.

Disabling a transfer stops new runs and backfills; runs already in progress finish.

BigQuery Data Transfer Service Security: IAM, Service Accounts and Keys

Who can do what. Enabling the service takes the Owner role on the project. Google recommends the BigQuery Admin role (roles/bigquery.admin) for the people who create transfers; when that is too broad, a custom role needs:

• bigquery.transfers.update and bigquery.transfers.get
• bigquery.datasets.get, bigquery.datasets.update, bigquery.datasets.getIamPolicy and bigquery.datasets.setIamPolicy
• bigquery.jobs.create

The two IAM policy permissions have been required since March 17, 2026, so a custom role written before then may now fail. Viewing transfers takes BigQuery User (roles/bigquery.user), reading their logs Logs Viewer (roles/logging.viewer), and changing, disabling or deleting them BigQuery Admin.

The service agent. DTS acts through a service agent, service-PROJECT_NUMBER@gcp-sa-bigquerydatatransfer.iam.gserviceaccount.com, which holds roles/bigquerydatatransfer.serviceAgent. Google creates it and grants the role the first time the API is used after the service is enabled; if you create it yourself with gcloud, the API or Terraform, grant the role yourself. Never revoke it: the service stops working. When a transfer runs as a service account from another project, grant the service agent roles/iam.serviceAccountTokenCreator on that account.

Run transfers as a service account. By default a transfer runs with the credentials of the person who created it, which ties a production load to one person's account. For Cloud Storage and Amazon S3 transfers, a service account is required when you sign in with a federated identity and optional with a Google account. The service account should belong to the transfer's project and needs bigquery.datasets.get and bigquery.datasets.update on the destination dataset, plus read access to the source. Whoever attaches it needs bigquery.transfers.update and the Service Account User role on it.

Encryption keys. Cloud Storage transfers and scheduled queries can use a customer-managed encryption key (--destination_kms_key), which DTS also applies to its intermediate on-disk cache of the data. Set it when you create the transfer: a key cannot be added to a transfer created without one.

Limits to Plan Around

| Limit | Applies to | What to do | |---|---|---| | 10,000 files and 15 TB per run | Cloud Storage | Spread large drops across prefixes or runs | | No minimum file age: files are picked up at once | Cloud Storage | Let writers finish a file before it matches the pattern | | Every matching file must fit the table's schema; a schema change between runs fails the transfer | Cloud Storage | Change schemas deliberately, then update the table | | Archived objects are not loaded | Cloud Storage | Keep source files live until they are loaded | | Prefix matching only; the bucket cannot be parameterized | Amazon S3 | One prefix per transfer | | 180 days per backfill request, one request at a time | All sources | Backfill in chunks | | Run history kept 90 days | All sources | Keep your own log of runs | | Load jobs count toward BigQuery's load job quotas | All sources | Watch quotas when many transfers run at once |

What Does BigQuery Data Transfer Service Cost?

Billing has to be enabled on the project for every transfer, and Google bills nothing for the transfers it lists as free. Not every connector is free, so check BigQuery's pricing page for the one you use before you plan volume. The rest of the cost comes from around the transfer:

• BigQuery itself. Once data is transferred, standard BigQuery storage and query pricing applies.
• Location. For a Cloud Storage transfer across locations, the price depends on where the bucket and the destination dataset are.
• Notifications. Pub/Sub notifications are billed by Pub/Sub.
• Amazon S3 egress. Amazon bills outbound data transfer from S3, and prefix matching can read more files than you meant to load.

When DTS Is Enough, and When You Need a Pipeline

| Situation | BigQuery Data Transfer Service | Custom pipeline (Airflow, dbt, code) | |---|---|---| | A supported source, loaded as it is | The simplest option | More to build and run | | Transformations before the data lands | Load raw, then transform in BigQuery with scheduled queries or dbt | Transform in flight | | Freshness under 15 minutes from Cloud Storage, or under an hour from S3 | Not possible on a schedule; event-driven Cloud Storage transfers help | Streaming ingestion | | A source DTS does not support, or an internal API | Not possible | Required | | Sending data out of BigQuery | Not possible | Required | | Schemas that change between runs | Cloud Storage transfers fail on a schema change | Handle the change in code |

A common split works well: DTS loads each supported source into a raw dataset, transformations run inside BigQuery, and custom ingestion covers only what DTS cannot reach. For data that has to land within seconds, a broker belongs in front of the warehouse; our guide to real-time data pipelines with Apache Kafka covers when that is worth its cost.