Iconik Storage Gateway Pro Overview Iconik Storage Gateway Pro Overview
  1. Iconik
  2. Iconik Storage Gateway (ISG) & Edge Transcoder
  3. Iconik Storage Gateway Pro (ISG Pro)

Iconik Storage Gateway Pro Overview

Iconik Storage Gateway Pro (ISG Pro)

This article contains an overview of the Iconik Storage Gateway Pro (ISG Pro)

Roles

These are the Iconik roles that are needed:

  • Owner

Overview

ISG Pro allows horizontal scale by connecting several ISG nodes into a cluster which provides redundancy and increased capacity on the key jobs of checksum calculation, transcoding and transfers.

ISG Pro can have multiple nodes working together in a cluster using a shared network database while connected to the same shared storage to enable ISG job distribution.

Benefits:

  • Horizontal scaling for checksum calculation, transcoding and transfers
  • Application high availability via leader election

Supported platforms:

  • Linux
  • macOS

Windows - not supported!

Concepts & terminology

Storage — a shared storage (NAS) reachable by every node in the cluster.

ISG (standalone) — single application instance, no cluster.

ISG Cluster — a cluster of ISG nodes sharing a network database and one or more storages.

Cluster — the parent entity grouping nodes. Holds shared application settings, db_connection_uri, visibility_timeout, and a linked storage. Created using the web interface in iconik: https://app.iconik.io/admin/isg

Node — an ISG that belongs to a cluster and runs on a dedicated machine/container. Entity can be created using WEB interface inside a cluster view. Linked to ISG by using storage-gateway-id set in local config.ini

Role — a role defines a set of job types that an ISG Node will consume & handle:

  • MAIN - scanner, parts of ingest and everything that wasn't scaled;
  • CHECKSUM - checksumming;
  • TRANSFER - upload/download/archive;
  • TRANSCODE - transcoding jobs;

Main Node — a Node that has MAIN role enabled.

Worker Node — a Node that has MAIN role disabled.

Leader / Follower — within a cluster only one Main Node is Leader at a time and runs MAIN-role jobs; the others are Followers ready to pickup MAIN-role jobs. Election is PostgreSQL-based. Only MAIN-role ISG Nodes taking part in election.

Telemetry — periodic report each worker sends to iconik with status, host info, and (on the Leader) storage stats.

Status — Active, Stopped, Offline, Warning, Error.

Requirements & prerequisites

Supported platforms

  • Linux: Fully supported.
  • macOS: Fully supported.
  • Windows: Not supported.

You can mix Linux and macOS nodes in the same cluster.

Storage

  • Every node mounts the same shared storage at a path it can read and write.
  • Per-node mount-point overrides are supported (useful when nodes have different filesystem layouts).
  • One cluster = one storage.

Pointing multiple nodes at a single shared storage has a performance cost. Horizontal scaling places significantly more load on storage than a single node, and too many concurrent checksum or transcode workers can degrade storage performance. Running many transcode jobs across many workers at once can eventually exhaust storage resources, slowing overall job throughput and affecting other operations on the same storage. Concrete numbers depend on the shared storage in use, the types of jobs running, and other workload factors. If you observe storage performance degradation, reducing the worker count or job concurrency can lessen the load the ISG cluster places on storage.

Network

All nodes must reach:

  • Iconik (cloud)
  • The shared PostgreSQL database (pgBouncer)
  • The shared storage

Database

  • PostgreSQL with pgBouncer in front for connection pooling.
  • Connection string format
    • postgres://[user[:password]@][host][:port][/db][?param=value&...]

    • Examples:

      • postgres://isg:secret@10.0.0.5:5432/isg_cluster (password inline; see the warning below)

      • postgres://isg@db.example.com:5432/isg_cluster (no password)
      • postgres://isg@db.example.com/isg_cluster?sslmode=verify-full (TLS required)

The connection string is saved in Iconik. If that's a concern, leave the password out of the string. Nodes connect through libpq, so they can get the password from anything libpq supports on each node's host:

  • PGPASSWORD environment variable.
  • Password file (~/.pgpass or the path in PGPASSFILE): one line per server in the form hostname:port:database:username:password, readable only by the ISG service user (chmod 600).
  • Password-less authentication: client certificates (sslcert/sslkey) or other methods your PostgreSQL server supports.

More details about authentication methods can be found on the PostgreSQL documentation pages.

PostgreSQL is required by cluster for jobs distribution and communication between the cluster nodes. All nodes in the cluster connect to it with the connection string set on the cluster.

pgBouncer is strongly recommended (not technically required for small clusters, but the application's connection usage is high enough that without pooling, larger clusters will hit PostgreSQL's max connection limit).

A few pgBouncer settings are required:

POOL_MODE=transaction - pool mode must be at least transaction-level. Switching to session-level pooling will break the application.

MAX_CLIENT_CONN=1000 - the limit must stay high. ISG can open many connections at once (this is expected to improve in future releases). When you add worker nodes, raise MAX_CLIENT_CONN accordingly - roughly +50 per worker is a safe starting point, though the exact number depends on worker configuration.

Deployment example using docker compose: https://github.com/iconik-io/iconik-storage-gateway-pro-docker

Iconik account

Each node uses a separate app ID and auth token with admin permissions. Use the same user account for all cluster nodes.

Time sync

Job lease expiration is timestamp-based. Nodes must have synchronized clocks.