External Engines Can Read, Write, and Create Iceberg Tables in Unity Catalog

Managed Iceberg tables, any engine. A standalone reference with credential-vending mechanics and Snowflake Horizon comparison.

A managed Iceberg table in Unity Catalog is not locked to Databricks compute. As of May 28, 2026, external Iceberg-compatible engines -- Spark, Flink, Trino, DuckDB, and others -- read, write, and create these tables directly, using short-lived credentials the catalog vends on request [1]. This is not a read-only export. It is full catalog participation by any engine that speaks the Iceberg REST Catalog protocol.

How credential vending actually works

The interaction follows the Apache Iceberg REST Catalog (IRC) specification [2]. The flow has four steps:

1. Authentication. The external engine authenticates to Unity Catalog using a Personal Access Token (PAT) or an OAuth2 token (including Azure Entra service principal tokens). The token is passed via `spark.sql.catalog.<name>.token` in the engine's configuration [3].
2. Catalog endpoint resolution. The engine connects to Unity Catalog's IRC endpoint at `https://<workspace-url>/api/2.1/unity-catalog/iceberg-rest`. The `warehouse` parameter maps to a Unity Catalog catalog name. The engine calls standard IRC endpoints -- `GET /v1/{prefix}/namespaces`, `GET /v1/{prefix}/namespaces/{namespace}/tables/{table}`, `POST /v1/{prefix}/namespaces/{namespace}/tables` -- and Unity Catalog handles metadata resolution [3].
3. Credential issuance. Unity Catalog validates the requesting principal's permissions, then issues short-lived cloud storage credentials scoped to the specific table's storage path. These are STS tokens (AWS), SAS tokens (Azure), or OAuth2-based tokens (GCP), with a one-hour TTL [4]. The scoping is table-level, not account-level: the credential grants access only to the storage paths required for that table, inheriting the privileges of the Databricks principal used to configure the integration [4].
4. Direct storage access. The external engine uses the vended credentials to read or write data files directly in cloud storage. Unity Catalog is not in the data path -- it issued the credential and stepped aside. The engine interacts with S3, ADLS, or GCS directly [4].

Two modes exist: table credential vending (access to registered Unity Catalog metastore data) and path credential vending*= (access to external locations) [4]. Both are Generally Available.

Cross-engine Attribute-Based Access Control (ABAC) -- which enforces row filters and column masks via IRC Scan APIs -- is in Beta as of May 2026 and requires Iceberg 1.11+ client libraries [1].

What shipped, and when

The timeline reflects a deliberate progression from read-only access to full read-write-create GA:

• 2023: Unity Catalog launched initial read-only support for the Iceberg REST Catalog, making it an early adopter of the protocol [5].
• June 12, 2025: Public Preview of Managed and Foreign Iceberg table support was announced at Data+AI Summit [5].
• May 28, 2026: General Availability of Managed Iceberg, Foreign Iceberg, credential vending for Foreign Iceberg, and Iceberg v3 support, announced in the "Unity Catalog and the Next Era of Apache Iceberg" blog post [1].

At GA, Unity Catalog auto-maintains managed Iceberg tables: snapshot expiry, file cleanup, and Liquid Clustering run automatically [5]. The table-maintenance burden that typically falls on teams operating external Iceberg catalogs is handled by the platform.

What Snowflake offers

Snowflake Horizon implements the Iceberg REST Catalog via an Apache Polaris-powered endpoint at `https://.snowflakecomputing.com/polaris/api/catalog` [6]. Authentication uses Programmatic Access Tokens (PATs) exchanged via the OAuth2 `client_credentials` grant [6]. Credential vending is requested via the `X-Iceberg-Access-Delegation: vended-credentials` header, and Snowflake issues temporary cloud storage credentials (STS, SAS, or OAuth2-based) so the engine can read or write Iceberg data files directly [6].

Snowflake's IRC timeline:

• October 2025: Catalog-Linked Databases (writing to Glue-managed Iceberg tables) reached GA [7].
• January 30, 2026: Bidirectional reads with Azure Fabric reached GA [7].
• February 2026: External engine read access via Horizon IRC reached GA [7].
• March 2026: External engine write access entered Public Preview [7].

Supported engines include Apache Spark, Apache Flink, Trino, DuckDB, Dremio, StarRocks, Doris, and PyIceberg [6]. Spark policy enforcement (row masking, column access) is GA via Snowflake Connector for Spark v3.1.6+ [7].

Billing: Horizon IRC API calls cost 0.5 credits per million calls (Cloud Services), with billing starting in H2 2026 [6]. During the current preview-to-GA transition, API calls are not billed.

A significant detail: Snowflake Horizon external engine writes remain in Public Preview as of September 2026 [7]. Snowflake native reads and writes to its own Iceberg tables are fully GA, but the IRC-based external write path has not yet graduated.

The architectural difference

Both implementations vend short-lived, scoped cloud storage credentials. Both support the IRC specification's standard endpoints. The difference is in what each catalog governs beyond the IRC protocol.

Unity Catalog positions the IRC endpoint as one facet of a broader governance plane. A managed Iceberg table in UC is governed by the same permissions, lineage tracking, and access control that govern Delta tables, ML models, and other catalog objects. When an external Spark job writes to a managed Iceberg table via IRC, it does so under Unity Catalog's unified governance -- same ACLs, same audit log [1][4].

Snowflake Horizon's Polaris-powered IRC endpoint governs Iceberg tables within Snowflake's perimeter. Row filters and column masks are enforced on IRC-served tables. The governance model is Snowflake's RBAC, applied to the Horizon catalog [6][7].

One directional asymmetry applies to Databricks: the IRC standard flows one direction on the platform -- inward to Unity Catalog. Databricks compute does not connect outward to external IRC catalogs (Polaris, Nessie, Glue) as an IRC client [8]. Unity Catalog can serve external engines, but Databricks clusters do not consume from external IRC catalogs via the same protocol. Snowflake's Horizon serves external engines via IRC but also does not natively connect to Databricks UC as a catalog source [7].

We concede

Snowflake's Polaris is a standard IRC implementation. Horizon's Polaris-powered endpoint implements the same Apache Iceberg REST Catalog specification that Unity Catalog implements. Apache Polaris itself is a genuine open-source project in the Apache Incubator, co-created by Snowflake and Dremio [9]. The protocol is not proprietary to either vendor.

Snowflake's read support reached GA first. External engine read access through Horizon IRC went GA in February 2026 [7], three months before Databricks' full read-write-create GA in May 2026. For organizations that needed production-grade external reads in early 2026, Snowflake had the GA milestone first.

Apache Polaris is genuinely open source. The reference IRC implementation that Snowflake's Horizon builds on is available as an Apache Incubator project under the Apache 2.0 license [9]. This is not a vendor-controlled open-core model.

The limits of this argument

Multi-statement transactions that write to managed Iceberg tables remain in Private Preview as of a July 22, 2026 release note -- for every engine, Databricks included [1]. This is a platform-wide limitation, not specific to external access, but it means that transactional write patterns are not yet GA.

Foreign Iceberg tables in Unity Catalog -- tables governed by a catalog other than UC -- are read-only. You cannot write to or create foreign Iceberg tables through UC [8]. The full read-write-create GA applies only to managed Iceberg tables.

Snowflake Horizon external writes remain in Public Preview [7]. Neither platform offers fully GA bidirectional IRC read-write with the other's catalog. All comparative analyses in this piece are sourced from competing vendors; no neutral third-party benchmarks exist for IRC performance or credential-vending latency.

Comparison matrix

The verdict

Both Databricks and Snowflake now vend short-lived credentials to external engines via the Iceberg REST Catalog protocol. The table does not care which engine wrote it -- that statement holds on both platforms. The difference is in maturity at the write boundary: Databricks reached full read-write-create GA for managed Iceberg in May 2026; Snowflake's external write support remains in Public Preview. For organizations that need external engines to write Iceberg tables in production today, that GA distinction matters. For organizations that need cross-platform reads, both platforms deliver.

Sources

[1] Databricks, "Unity Catalog and the Next Era of Apache Iceberg," https://www.databricks.com/blog/unity-catalog-and-next-era-apache-icebergtm

[2] Apache Iceberg, "REST Catalog Open API Specification," https://github.com/apache/iceberg/blob/main/open-api/rest-catalog-open-api.yaml

[3] Databricks Documentation, "Connect to Unity Catalog with Iceberg REST Catalog," https://docs.databricks.com/aws/en/external-access/admin-conn-iceberg

[4] Databricks Documentation, "Credential Vending," https://docs.databricks.com/aws/en/external-access/credential-vending

[5] Databricks, "Announcing Full Apache Iceberg Support in Databricks," https://www.databricks.com/blog/announcing-full-apache-iceberg-support-databricks

[6] Snowflake Chronicles, "Snowflake Apache Iceberg External Write Support Now GA," https://snowflakechronicles.medium.com/snowflake-apache-iceberg-external-write-support-now-ga-b0a99f540a65

[7] Snowflake, "Iceberg Catalog: Snowflake Horizon vs. Unity Catalog," https://www.snowflake.com/en/blog/engineering/iceberg-catalog-snowflake-horizon-vs-unity-catalog/

[8] Community analysis, "Unity Catalog as an Iceberg REST Catalog," sourced from Databricks documentation cross-reference (directional flow limitation documented in admin setup guides)

[9] Apache Polaris (Incubating), https://polaris.apache.org/