This article contains an overview of the Iconik Storage Gateway Pro (ISG Pro)
Roles
These are the Iconik roles that are needed:
- Owner
Overview
ISG Pro allows horizontal scale by connecting several ISG nodes into a cluster which provides redundancy and increased capacity on the key jobs of checksum calculation, transcoding and transfers.
ISG Pro can have multiple nodes working together in a cluster using a shared network database while connected to the same shared storage to enable ISG job distribution.
Benefits:
- Horizontal scaling for checksum calculation, transcoding and transfers
- Application high availability via leader election
Supported platforms:
- Linux
- macOS
Windows - not supported!
Concepts & terminology
Storage — a shared storage (NAS) reachable by every node in the cluster.
ISG (standalone) — single application instance, no cluster.
ISG Cluster — a cluster of ISG nodes sharing a network database and one or more storages.
Cluster — the parent entity grouping nodes. Holds shared application settings, db_connection_uri, visibility_timeout, and a linked storage. Created using the web interface in iconik: https://app.iconik.io/admin/isg
Node — an ISG that belongs to a cluster and runs on a dedicated machine/container. Entity can be created using WEB interface inside a cluster view. Linked to ISG by using storage-gateway-id set in local config.ini
Role — a role defines a set of job types that an ISG Node will consume & handle:
- MAIN - scanner, parts of ingest and everything that wasn't scaled;
- CHECKSUM - checksumming;
- TRANSFER - upload/download/archive;
- TRANSCODE - transcoding jobs;
Main Node — a Node that has MAIN role enabled.
Worker Node — a Node that has MAIN role disabled.
Leader / Follower — within a cluster only one Main Node is Leader at a time and runs MAIN-role jobs; the others are Followers ready to pickup MAIN-role jobs. Election is PostgreSQL-based. Only MAIN-role ISG Nodes taking part in election.
Telemetry — periodic report each worker sends to iconik with status, host info, and (on the Leader) storage stats.
Status — Active, Stopped, Offline, Warning, Error.
Requirements & prerequisites
Supported platforms
- Linux: Fully supported.
- macOS: Fully supported.
- Windows: Not supported.
You can mix Linux and macOS nodes in the same cluster.
Storage
- Every node mounts the same shared storage at a path it can read and write.
- Per-node mount-point overrides are supported (useful when nodes have different filesystem layouts).
- One cluster = one storage.
Network
All nodes must reach:
- Iconik (cloud)
- The shared PostgreSQL database (pgBouncer)
- The shared storage
Database
- PostgreSQL with pgBouncer in front for connection pooling.
- Connection string format example
- postgresql://<user>:<password>@<host>:6432/<db>
- TLS supported via sslmode=require and certificate paths in the connection string. Connection string format example:
- postgresql://<user>:<password>@<host>:6432/<db>?sslmode=require&sslkey=...&sslcert=…
PostgreSQL is required by cluster for jobs distribution and communication between the cluster nodes.
pgBouncer is strongly recommended (not technically required for small clusters, but the application's connection usage is high enough that without pooling, larger clusters will hit PostgreSQL's max connection limit).
A few pgBouncer settings are dictated by how ISG operates and must not be relaxed:
POOL_MODE=transaction - pool mode must be at least transaction-level. Switching to session-level pooling will break the application.
MAX_CLIENT_CONN=1000 - the limit must stay high. ISG can open many connections at once (this is expected to improve in future releases). When you add worker nodes, raise MAX_CLIENT_CONN accordingly - roughly +50 per worker is a safe starting point, though the exact number depends on worker configuration.
Deployment example using docker compose: https://github.com/iconik-io/iconik-storage-gateway-pro-docker
Iconik account
Each node uses a separate app ID and auth token with admin permissions. Use the same user account for all cluster nodes.
Time sync
Job lease expiration is timestamp-based. Nodes must have synchronized clocks.