ISG Pro Installation

This article contains everything you need to know about installing the Iconik Storage Gateway Pro (ISG Pro)

Installing a new ISG Pro Cluster

1. Provision PostgreSQL and pgBouncer and confirm every host can reach the database. A single network database is used by the cluster nodes for jobs distribution and communication. pgBouncer is strongly recommended (not technically required for small clusters, but the application's connection usage is high enough that without pooling, larger clusters will hit PostgreSQL's max connection limit). More details can be found on the overview page. Setup example using docker compose file: https://github.com/iconik-io/iconik-storage-gateway-pro-docker
2. Create ISG storage using the regular ISG storage creation form via iconik web admin panel: https://test.iconik.cloud/admin/storages. You can skip this step if you already have a storage that you want to migrate from standalone ISG to the cluster.
3. Create cluster using iconik web admin panel: https://app.iconik.io/admin/isg . This also creates the first node with all roles enabled.
4. Mount shared storage on each host that will join the cluster.
5. Create a node entry in the Iconik web admin for each host https://app.iconik.io/admin/isg

6. Install ISG on each host. More details about individual ISG installation process can be found on the help pages.

   Important: Instead of storage-id that is used in standalone ISG in the config.ini the cluster node ISG requires that the correct storage-gateway-id is set as the node ID:

   

   The config.ini file should look like this:

   [main]
   
   storage-gateway-id = df23b450-6306-11f1-aa19-f27e47674730
   app-id = 22f34c6e-268e-11e7-ac53-6c4008b85488
   iconik-url = https://app.iconik.io/
   auth-token = eyJhbGciOiJIUzI1NiIsImlhdCI6MTUwMTU3NjAyNCwiZXhwIjoxNTAxNjE5MjI0fQ.eyJpZCI6IjMzMjFkMjIyLTc2OTMtMTFlNy1hN2EyLTAyNDIwYTAyMTAxYSJ9.sgag4OQb19J9qT1IAdKZYAqbpIoai7oW60Mahye7o8s
   sleep-time = 5
   log-filename = /var/log/iconik/iconik_storage_gateway/default.log
   image-magick-config = /etc/iconik/iconik_storage_gateway/image_magick_config
   sqlite-journal-mode = wal

    

7. After all nodes are installed and started enable cluster using cluster action menu.
   
   

Migration from Standalone ISG to ISG Pro Cluster

It is possible to migrate an ISG covered storage to a new cluster. 

Critical: the mount point must stay the same so relative paths remain valid after migration.

Prerequisites: A PostgreSQL instance reachable from the ISG host, and the ISG service stopped.

1. Deploy PostgreSQL with a connection pooler — use the provided docker compose files at https://github.com/iconik-io/iconik-storage-gateway-pro-docker or your own setup.
2. Stop the existing ISG service.

   • Ubuntu: 

     sudo systemctl stop iconik-storage-gateway

   • Enterprise Linux: 

     sudo systemctl stop iconik_storage_gateway

   • Mac OS: use the "Stop iconik Storage Gateway" tray app action.
3. Upgrade ISG to a latest version (confirm that ISG remains stopped after the upgrade). 

   • Ubuntu: 

     sudo apt-get update
     sudo apt-get install iconik-storage-gateway

   • Enterprise Linux:

     sudo dnf install iconik_storage_gateway

   • For Mac OS download the latest package and repeat the installation process: https://downloads.iconik.io/IconikStorageGateway-latest.pkg
4. Create the Cluster entity in the iconik admin panel: https://app.iconik.io/admin/isg. Link it to the existing Storage by selecting it in the storage dropdown field and attach to your empty PostgreSQL database. Make sure connection string for the cluster database works. More details can be found on the overview page.
5. Edit config.ini on the ISG host:
   • Set storage-gateway-id to the cluster's main node ID (copied from the admin panel).
   • Remove storage-id attribute line.
   • You can also remove concurrency attributes from the config.ini to control them from the cluster settings scanner-concurrency-value, checksum-max-workers, file-download-parallel-downloads-num, file-upload-parallel-uploads-num, max-transcoding-jobs

     
     Example of the modified config.ini:

     [main]
     
     storage-gateway-id = df23b450-6306-11f1-aa19-f27e47674730
     app-id = 22f34c6e-268e-11e7-ac53-6c4008b85488
     iconik-url = https://app.iconik.io/
     auth-token = eyJhbGciOiJIUzI1NiIsImlhdCI6MTUwMTU3NjAyNCwiZXhwIjoxNTAxNjE5MjI0fQ.eyJpZCI6IjMzMjFkMjIyLTc2OTMtMTFlNy1hN2EyLTAyNDIwYTAyMTAxYSJ9.sgag4OQb19J9qT1IAdKZYAqbpIoai7oW60Mahye7o8s
     sleep-time = 5
     log-filename = /var/log/iconik/iconik_storage_gateway/default.log
     image-magick-config = /etc/iconik/iconik_storage_gateway/image_magick_config
     sqlite-journal-mode = wal

6. (Optional) Verify with a dry run - counts source rows without writing anything:

   iconik_storage_gateway --migrate-to-postgres --dry-run --data-location <path-to-sqlite-data>
   

7. Run the migration:

   • Linux:

     /opt/iconik/iconik_storage_gateway/iconik_storage_gateway --migrate-to-postgres --data-location <path-to-sqlite-data>
     

   • macOS (arm64):

     "/Applications/Cantemo/IconikStorageGateway/iconik Storage Gateway.app/Contents/Resources/iconik_storage_gateway-onedir-arm64/iconik_storage_gateway" --migrate-to-postgres --data-location <path-to-sqlite-data>
     

   • macOS (x86_64):

     "/Applications/Cantemo/IconikStorageGateway/iconik Storage Gateway.app/Contents/Resources/iconik_storage_gateway-onedir-x86_64/iconik_storage_gateway" --migrate-to-postgres --data-location <path-to-sqlite-data>
     

   The migrator reads the SQLite files under <data-location>/<storage-id>/, applies any pending schema patches to the SQLite copies, then copies records into the PostgreSQL database defined in the cluster settings. Primary-key sequences are advanced automatically, and jobs left in a "started" state (with no live worker lease) are re-queued.

   If --data-location is not specified a default data path will be used.

   If docker is used to run ISG make sure sqlite3 database files are available in the container.

   If the migration fails, drop the PostgreSQL database and re-run; it is single-shot, not resumable.

8. Enable the cluster in the iconik admin panel:
   
9. Start ISG.

Migration command lines overview (dry run):