S3

S3 providers expose portable object storage to executable plugins. Gestalt models the de facto S3-compatible object API rather than a filesystem API, so the same plugin code can target AWS S3, MinIO, GCS XML interoperability mode, Cloudflare R2, and other compatible backends.

How S3 providers work

Configure one or more named entries under providers.s3, then bind them into executable plugins with plugins.<name>.s3. The host starts one provider process per enabled named entry and only exposes the bindings each plugin is granted.

Plugins still work with ordinary {bucket, key, version_id} references. The host keeps plugin data isolated with an internal namespace prefix, so plugins do not need to invent their own storage prefix scheme.

When a plugin binds exactly one S3 provider, the SDKs let it open the default client directly. When a plugin binds more than one S3 provider, the plugin opens the named binding it wants to use. That keeps plugin code portable while letting the host decide which concrete storage backend sits behind each binding.

The contract stays close to familiar S3 semantics. It supports metadata reads, object reads and writes, deletes, paginated listing, server-side copy, provider-native presigned GET, PUT, DELETE, and HEAD URLs, and host-mediated object access URLs for plugin-scoped browser uploads/downloads.

Configuring the first-party S3 provider

The first-party package lives at github.com/valon-technologies/gestalt-providers/s3/s3:


providers:
  s3:
    assets:
      source: https://artifacts.example.com/s3/s3/v0.0.1-alpha.1/provider-release.yaml
      config:
        region: us-east-1
        endpoint: https://s3.us-east-1.amazonaws.com
        forcePathStyle: false
        accessKeyId: ${AWS_ACCESS_KEY_ID}
        secretAccessKey:
          secret:
            provider: default
            name: aws-secret-access-key
 
plugins:
  media:
    source: ./plugins/media/manifest.yaml
    s3:
      - assets

The main settings are region, which controls request signing, and endpoint, which lets you target S3-compatible systems such as MinIO or GCS XML interoperability instead of AWS itself. forcePathStyle switches between path-style and virtual-host-style requests. If you want to provide static HMAC credentials directly, set accessKeyId, secretAccessKey, and optionally sessionToken. If you leave those out, the provider falls back to the AWS SDK default credential chain.

GCS XML interoperability uses the same provider with a different endpoint and HMAC credentials:


providers:
  s3:
    archive:
      source: https://artifacts.example.com/s3/s3/v0.0.1-alpha.1/provider-release.yaml
      config:
        region: auto
        endpoint: https://storage.googleapis.com
        forcePathStyle: true
        accessKeyId: ${GCS_HMAC_ACCESS_KEY}
        secretAccessKey:
          secret:
            provider: default
            name: gcs-hmac-secret

Using `S3` from a plugin

The SDK exposes an S3 client plus object-handle helpers in every supported language.

Go


import (
  "context"
  "log"
 
  gestalt "github.com/valon-technologies/gestalt/sdk/go"
)
 
ctx := context.Background()
 
s3, err := gestalt.S3()
if err != nil {
  log.Fatal(err)
}
defer s3.Close()
 
avatar := s3.Object("uploads", "avatars/user-123.png")
if _, err := avatar.WriteBytes(ctx, pngBytes, &gestalt.WriteOptions{
  ContentType: "image/png",
}); err != nil {
  log.Fatal(err)
}
 
body, err := avatar.Bytes(ctx, nil)
if err != nil {
  log.Fatal(err)
}
_ = body

Python


import gestalt
 
s3 = gestalt.S3()
avatar = s3.object("uploads", "avatars/user-123.png")
avatar.write_bytes(
    png_bytes,
    gestalt.WriteOptions(content_type="image/png"),
)
body = avatar.bytes()
s3.close()

Rust


use gestalt::{s3::WriteOptions, S3};
 
let s3 = S3::connect().await?;
let mut avatar = s3.object("uploads", "avatars/user-123.png");
 
avatar
    .write_bytes(
        &png_bytes,
        Some(WriteOptions {
            content_type: "image/png".to_string(),
            ..Default::default()
        }),
    )
    .await?;
 
let body = avatar.bytes(None).await?;

TypeScript


import { PresignMethod, S3 } from "@valon-technologies/gestalt";
 
const s3 = new S3();
const avatar = s3.object("uploads", "avatars/user-123.png");
 
await avatar.write(pngBytes, { contentType: "image/png" });
const body = await avatar.bytes();
 
const upload = await avatar.createAccessUrl({
  method: PresignMethod.Put,
  contentType: "image/png",
  headers: { "Content-Length": String(pngBytes.byteLength) },
  expiresSeconds: 900,
});
await fetch(upload.url, {
  method: upload.method,
  headers: { ...upload.headers, "Content-Type": "image/png" },
  body: pngBytes,
});

Use the named helpers when a plugin binds more than one S3 provider: Go and Python use gestalt.S3("archive"), Rust uses gestalt::S3::connect_named("archive"), and TypeScript uses new S3("archive").

Operational notes

Gestalt does not use S3 providers for its own system state, so these bindings are plugin-only. The first-party provider preserves the portable object identity {bucket, key, version_id}, which means buckets are chosen by the caller instead of being fixed in provider config.

For plugin-scoped browser access, use createAccessUrl / CreateAccessURL rather than provider-native presign. Gestalt returns a short-lived URL under /api/v1/s3/object-access/{token}. The token is encrypted by the host, contains the plugin, S3 binding, logical object ref, HTTP method, expiry, and required headers, and the public route streams bytes through the configured S3 provider after applying the plugin namespace prefix. That keeps the S3 provider generic and avoids exposing provider-internal object keys to clients. The compatibility presign methods also return hosted object access URLs for plugin-scoped bindings when server.baseURL and server.encryptionKey are configured.

The first-party provider rejects writes and copies above the single-request S3 limit instead of exposing provider-specific multipart controls through the portable provider contract. Browser uploads that need direct client transport should use hosted object access URLs; product-specific multipart state still belongs in the plugin that owns that workflow. Published S3 provider packages can still be prepared with gestaltd init, just like auth, indexeddb, cache, and secrets providers.

Building your own S3 provider

The provider interface, SDK authoring details, and custom release flow live under Custom Providers > S3.

S3