S3

S3 providers expose portable object storage to executable apps. Gestalt models the de facto S3-compatible object API rather than a filesystem API, so the same app code can target AWS S3, MinIO, GCS XML interoperability mode, Cloudflare R2, and other compatible backends.

Using `S3` from a plugin

The SDK exposes an S3 client plus object-handle helpers in every supported language.

Go


import (
  "context"
  "log"
 
  gestalt "github.com/valon-technologies/gestalt/sdk/go"
)
 
ctx := context.Background()
 
s3, err := gestalt.S3()
if err != nil {
  log.Fatal(err)
}
defer s3.Close()
 
avatar := s3.Object("uploads", "avatars/user-123.png")
if _, err := avatar.WriteBytes(ctx, pngBytes, &gestalt.WriteOptions{
  ContentType: "image/png",
}); err != nil {
  log.Fatal(err)
}
 
body, err := avatar.Bytes(ctx, nil)
if err != nil {
  log.Fatal(err)
}
_ = body

Python


from gestalt import S3, WriteOptions
 
s3 = S3()
avatar = s3.object("uploads", "avatars/user-123.png")
avatar.write_bytes(
    png_bytes,
    WriteOptions(content_type="image/png"),
)
body = avatar.bytes()
s3.close()

Rust


use gestalt::{s3::WriteOptions, S3};
 
let s3 = S3::connect().await?;
let mut avatar = s3.object("uploads", "avatars/user-123.png");
 
avatar
    .write_bytes(
        &png_bytes,
        Some(WriteOptions {
            content_type: "image/png".to_string(),
            ..Default::default()
        }),
    )
    .await?;
 
let body = avatar.bytes(None).await?;

TypeScript


import { S3 } from "@valon-technologies/gestalt";
 
const s3 = new S3();
const avatar = s3.object("uploads", "avatars/user-123.png");
 
await avatar.write(pngBytes, { contentType: "image/png" });
const body = await avatar.bytes();

Building your own S3 provider

Manifest

An S3 provider manifest declares kind: s3:


kind: s3
source: github.com/acme/gestalt-providers/s3/minio
version: 0.0.1-alpha.1
displayName: MinIO S3
description: S3-compatible object storage provider backed by MinIO.
spec:
  configSchemaPath: ./config.schema.json

Provider interface

Implement the SDK’s authored S3Provider surface. The SDK adapts this typed interface to the underlying gRPC service:

Method	Purpose
Head object	Return metadata for one object reference.
Read object	Stream one object body with its metadata.
Write object	Accept a body stream and return committed metadata.
Delete object	Delete one object reference.
List objects	List objects by bucket, prefix, delimiter, and continuation token.
Copy object	Copy an object server-side between object references.
Presign object	Produce a presigned request URL plus required headers.

Go


package s3provider
 
import (
  "bytes"
  "context"
  "io"
  "time"
 
  gestalt "github.com/valon-technologies/gestalt/sdk/go"
)
 
type Provider struct{}
 
func New() *Provider { return &Provider{} }
 
func (p *Provider) Configure(_ context.Context, _ string, config map[string]any) error {
  _ = config
  return nil
}
 
func (p *Provider) HeadObject(ctx context.Context, ref gestalt.ObjectRef) (gestalt.ObjectMeta, error) {
  _ = ctx
  return gestalt.ObjectMeta{
    Ref:          ref,
    ETag:         "",
    Size:         0,
    ContentType:  "application/octet-stream",
    LastModified: time.Now(),
    Metadata:     map[string]string{},
  }, nil
}
 
func (p *Provider) ReadObject(ctx context.Context, ref gestalt.ObjectRef, opts *gestalt.ReadOptions) (gestalt.ObjectMeta, io.ReadCloser, error) {
  _ = opts
  meta, err := p.HeadObject(ctx, ref)
  if err != nil {
    return gestalt.ObjectMeta{}, nil, err
  }
  return meta, io.NopCloser(bytes.NewReader(nil)), nil
}
 
func (p *Provider) WriteObject(ctx context.Context, ref gestalt.ObjectRef, body io.Reader, opts *gestalt.WriteOptions) (gestalt.ObjectMeta, error) {
  _ = body
  _ = opts
  return p.HeadObject(ctx, ref)
}
 
func (p *Provider) DeleteObject(context.Context, gestalt.ObjectRef) error {
  return nil
}
 
func (p *Provider) ListObjects(context.Context, gestalt.ListOptions) (gestalt.ListPage, error) {
  return gestalt.ListPage{}, nil
}
 
func (p *Provider) CopyObject(ctx context.Context, source gestalt.ObjectRef, destination gestalt.ObjectRef, opts *gestalt.CopyOptions) (gestalt.ObjectMeta, error) {
  _ = source
  _ = opts
  return p.HeadObject(ctx, destination)
}
 
func (p *Provider) PresignObject(context.Context, gestalt.ObjectRef, *gestalt.PresignOptions) (gestalt.PresignResult, error) {
  return gestalt.PresignResult{
    URL:       "https://example.invalid/object",
    Method:    gestalt.PresignMethodGet,
    ExpiresAt: time.Now().Add(time.Minute),
    Headers:   map[string]string{},
  }, nil
}

Python


from gestalt import (
    ListPage,
    ObjectMeta,
    PresignMethod,
    PresignResult,
    ProviderReadResult,
    S3Provider,
)
 
class Provider(S3Provider):
    def head_object(self, ref):
        return ObjectMeta(
            ref=ref,
            etag="",
            size=0,
            content_type="application/octet-stream",
            metadata={},
        )
 
    def read_object(self, ref, opts=None):
        return ProviderReadResult(
            meta=self.head_object(ref),
            body=iter(()),
        )
 
    def write_object(self, ref, body, opts=None):
        for _chunk in body:
            pass
        return self.head_object(ref)
 
    def delete_object(self, ref):
        pass
 
    def list_objects(self, opts):
        return ListPage()
 
    def copy_object(self, source, destination, opts=None):
        return self.head_object(destination)
 
    def presign_object(self, ref, opts=None):
        return PresignResult(
            url=f"https://example.invalid/{ref.bucket}/{ref.key}",
            method=opts.method if opts and opts.method else PresignMethod.GET,
            headers=dict(opts.headers) if opts else {},
        )
 
provider = Provider()

Rust


#[derive(Default)]
pub struct MinioS3;
 
#[gestalt::async_trait]
impl gestalt::S3Provider for MinioS3 {
    async fn configure(
        &self,
        _name: &str,
        _config: serde_json::Map<String, serde_json::Value>,
    ) -> gestalt::Result<()> {
        Ok(())
    }
 
    async fn head_object(
        &self,
        _request: gestalt::HeadObjectRequest,
    ) -> gestalt::Result<gestalt::HeadObjectResponse> {
        Err(gestalt::Error::unimplemented("todo"))
    }
 
    async fn read_object(
        &self,
        _request: gestalt::ReadObjectRequest,
    ) -> gestalt::Result<gestalt::S3ReadObjectStream> {
        Err(gestalt::Error::unimplemented("todo"))
    }
 
    async fn write_object(
        &self,
        _request: gestalt::S3WriteObjectStream,
    ) -> gestalt::Result<gestalt::WriteObjectResponse> {
        Err(gestalt::Error::unimplemented("todo"))
    }
 
    async fn delete_object(
        &self,
        _request: gestalt::DeleteObjectRequest,
    ) -> gestalt::Result<()> {
        Err(gestalt::Error::unimplemented("todo"))
    }
 
    async fn list_objects(
        &self,
        _request: gestalt::ListObjectsRequest,
    ) -> gestalt::Result<gestalt::ListObjectsResponse> {
        Err(gestalt::Error::unimplemented("todo"))
    }
 
    async fn copy_object(
        &self,
        _request: gestalt::CopyObjectRequest,
    ) -> gestalt::Result<gestalt::CopyObjectResponse> {
        Err(gestalt::Error::unimplemented("todo"))
    }
 
    async fn presign_object(
        &self,
        _request: gestalt::PresignObjectRequest,
    ) -> gestalt::Result<gestalt::PresignObjectResponse> {
        Err(gestalt::Error::unimplemented("todo"))
    }
}
 
gestalt::export_s3_provider!(constructor = MinioS3::default);

TypeScript


import { PresignMethod, defineS3Provider } from "@valon-technologies/gestalt";
 
export const provider = defineS3Provider({
  async configure(name, config) {
    void name;
    void config.endpoint;
  },
 
  async headObject(ref) {
    return {
      ref,
      etag: "",
      size: 0n,
      contentType: "",
      lastModified: new Date(),
      metadata: {},
      storageClass: "",
    };
  },
 
  async readObject(ref) {
    return {
      meta: await this.headObject(ref),
      body: new Uint8Array(),
    };
  },
 
  async writeObject(ref, body, options) {
    void body;
    void options;
    return await this.headObject(ref);
  },
 
  async deleteObject(ref) {
    void ref;
  },
 
  async listObjects(options) {
    void options;
    return {
      objects: [],
      commonPrefixes: [],
      nextContinuationToken: "",
      hasMore: false,
    };
  },
 
  async copyObject(source, destination, options) {
    void source;
    void options;
    return await this.headObject(destination);
  },
 
  async presignObject(ref, options) {
    return {
      url: `https://example.invalid/${ref.bucket}/${encodeURIComponent(ref.key)}`,
      method: options?.method ?? PresignMethod.Get,
      headers: { ...(options?.headers ?? {}) },
      expiresAt: new Date(Date.now() + 60_000),
    };
  },
});

gestalt.S3Provider adds runtime lifecycle wiring and a serve() helper for languages that expose the server-side S3 provider surface.

App access

Plugin-side S3 usage is documented in Using S3 from a plugin. Provider authors implement the S3 service surface; Gestalt handles configured plugin bindings, SDK sockets, and host-mediated object access URLs.

Deploying the provider

Custom S3 providers are wired into Gestalt the same way as first-party ones:


providers:
  s3:
    assets:
      source: ./providers/s3/custom/manifest.yaml
      config:
        endpoint: http://127.0.0.1:9000
        region: us-east-1
 
apps:
  media:
    source: ./apps/media/manifest.yaml
    s3:
      - assets

For packaging and published release URLs, see Releasing provider packages.

Implementation notes

Treat object references as {bucket, key, version_id} values, not filesystem paths.
Keep reads and writes streaming. Do not require the caller to buffer the full object in memory.
Preserve backend metadata that fits the portable model: etag, size, content_type, last_modified, metadata, and storage_class.
CopyObject conditionals apply to the source object, not the destination object.
Map missing objects to NotFound, conditional failures to FailedPrecondition, and invalid ranges to OutOfRange so SDK error mapping stays portable.
PresignObject should return only caller-required headers. Do not leak transport-generated headers such as Host.
Implement multipart upload and multipart copy internally when the backend needs them, or reject above the single-request S3 limit explicitly. Do not silently rely on backend-specific PutObject / CopyObject failures.

S3

Using S3 from a plugin

Go

Python

Rust

TypeScript

Building your own S3 provider

Manifest

Provider interface

Go

Python

Rust

TypeScript

App access

Deploying the provider

Implementation notes

Using `S3` from a plugin