basic-design: create a datavault-storage abstraction system "data-as-a-service" start-simple: "grow-as-you-go" 1 icat-server (icat-service + postgresql database local vsan) OS: CentOS7 3 resource-servers (with 2 local mounts each) 3 datacenters 2 replica's of data 1 replica in 1 datacenter, other replica in other datacenter - encrypt storage (because cloudstorage) - all servers are esx vm's (rug-cloud) - all storage is vmware datastore (rug-cloud) - all irods-servers/clients connect via SSL - authentication via ldap connection from peregrine to irods-servers is 10 Gb ethernet irods-lingo: icat-server: server containing metadata database irods-resource-server: server with mountpoint storing data provider: icat-server consumer: irods-resource server collections: directories objects: files local-lingo: peregrine: our HPC cluster in Groningen irods installation on centos7 2019: the icat-server: - basic/normal configuration - disable selinux - enable/configure firewall - set/enable ntpd # rpm --import https://packages.irods.org/irods-signing-key.asc # wget -qO - https://packages.irods.org/renci-irods.yum.repo | sudo tee /etc/yum.repos.d/renci-irods.yum.repo # yum install irods-server irods-database-plugin-postgres # yum install postgresql-server # postgresql-setup initdb Initializing database ... OK # systemctl start postgresql # su - postgres Last login: Fri Oct 26 11:30:44 CEST 2018 on pts/0 $ psql psql (9.2.24) Type "help" for help. postgres=# CREATE USER irods WITH PASSWORD 'xxxxx'; CREATE ROLE postgres=# CREATE DATABASE "ICAT"; CREATE DATABASE postgres=# GRANT ALL PRIVILEGES ON DATABASE "ICAT" TO irods; GRANT postgres=# \l List of databases Name | Owner | Encoding | Collate | Ctype | Access privileges -----------+----------+----------+-------------+-------------+----------------------- ICAT | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 | =Tc/postgres + | | | | | postgres=CTc/postgres+ | | | | | irods=CTc/postgres postgres | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 | template0 | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 | =c/postgres + | | | | | postgres=CTc/postgres template1 | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 | =c/postgres + | | | | | postgres=CTc/postgres (4 rows) # vi /var/lib/pgsql/data/pg_hba.conf: host all all 127.0.0.1/32 md5 host all all 129.125.77.0/32 md5 # systemctl restart postgresql # python /var/lib/irods/scripts/setup_irods.py The iRODS service account name needs to be defined. iRODS user [irods]: iRODS group [irods]: +--------------------------------+ | Setting up the service account | +--------------------------------+ Existing Group Detected: irods Existing Account Detected: irods Setting owner of /var/lib/irods to irods:irods Setting owner of /etc/irods to irods:irods iRODS server's role: 1. provider 2. consumer Please select a number or choose 0 to enter a new value [1]: Updating /etc/irods/server_config.json... +-----------------------------------------+ | Configuring the database communications | +-----------------------------------------+ You are configuring an iRODS database plugin. The iRODS server cannot be started until its database has been properly configured. ODBC driver for postgres [PostgreSQL]: Database server's hostname or IP address [localhost]: Database server's port [5432]: Database name [ICAT]: Database username [irods]: ------------------------------------------- Database Type: postgres ODBC Driver: PostgreSQL Database Host: localhost Database Port: 5432 Database Name: ICAT Database User: irods ------------------------------------------- Please confirm [yes]: Database password: Updating /etc/irods/server_config.json... Listing database tables... Salt for passwords stored in the database: Updating /etc/irods/server_config.json... +--------------------------------+ | Configuring the server options | +--------------------------------+ iRODS server's zone name [tempZone]: testZone iRODS server's port [1247]: iRODS port range (begin) [20000]: iRODS port range (end) [20199]: Control Plane port [1248]: Schema Validation Base URI (or off) [file:///var/lib/irods/configuration_schemas]: iRODS server's administrator username [rods]: irods ------------------------------------------- Zone name: testZone iRODS server port: 1247 iRODS port range (begin): 20000 iRODS port range (end): 20199 Control plane port: 1248 Schema validation base URI: file:///var/lib/irods/configuration_schemas iRODS server administrator: irods ------------------------------------------- Please confirm [yes]: yes iRODS server's zone key: Zone key must be at least 1 character in length. iRODS server's zone key: iRODS server's negotiation key (32 characters): Negotiation key must be exactly 32 characters in length. iRODS server's negotiation key (32 characters): Control Plane key (32 characters): Updating /etc/irods/server_config.json... +-----------------------------------+ | Setting up the client environment | +-----------------------------------+ iRODS server's administrator password: Updating /var/lib/irods/.irods/irods_environment.json... +--------------------------+ | Setting up default vault | +--------------------------+ iRODS Vault directory [/var/lib/irods/Vault]: +-------------------------+ | Setting up the database | +-------------------------+ Listing database tables... Creating database tables... +-------------------+ | Starting iRODS... | +-------------------+ Validating [/var/lib/irods/.irods/irods_environment.json]... Success Validating [/var/lib/irods/VERSION.json]... Success Validating [/etc/irods/server_config.json]... Success Validating [/etc/irods/host_access_control_config.json]... Success Validating [/etc/irods/hosts_config.json]... Success Ensuring catalog schema is up-to-date... Updating to schema version 2... Updating to schema version 3... Updating to schema version 4... Updating to schema version 5... Catalog schema is up-to-date. Starting iRODS server... Success +---------------------+ | Attempting test put | +---------------------+ Putting the test file into iRODS... Getting the test file from iRODS... Removing the test file from iRODS... Success. +--------------------------------+ | iRODS is installed and running | +--------------------------------+ installation of irods-resource-server: - disable selinux - enable/configure firewall - set/enable ntpd install irods-repository: # rpm --import https://packages.irods.org/irods-signing-key.asc # wget -qO - https://packages.irods.org/renci-irods.yum.repo | sudo tee /etc/yum.repos.d/renci-irods.yum.repo # yum install epel-release # yum install irods-server # python /var/lib/irods/scripts/setup_irods.py set this server to a consumer (resource-server) provider= icat-server encrypt storage: create keyfile: # echo "some difficult string" >> /etc/keyfile # chmod 600 /etc/keyfile # cryptsetup luksFormat -y -v /dev/sdb --key-file /etc/keyfile # cryptsetup luksFormat -y -v /dev/sdc --key-file /etc/keyfile open encrypted storage: # cryptsetup luksOpen /dev/sdb irods01 --key-file /etc/keyfile # cryptsetup luksOpen /dev/sdc irods02 --key-file /etc/keyfile format storage: # mkfs.xfs /dev/mapper/irods01 # mkfs.xfs /dev/mapper/irods02 mount storage: # mount /dev/mapper/irods01 /mnt/01/ # mount /dev/mapper/irods02 /mnt/02/ create resources: as user irods on whatever irods-server: iadmin mkresc ReplA replication iadmin mkresc ReplB replication iadmin mkresc ReplC replication iadmin mkresc Vol01 rdms-prod-resc0.data.rug.nl:/mnt/01/Vault iadmin mkresc Vol02 rdms-prod-resc0.data.rug.nl:/mnt/02/Vault iadmin mkresc Vol11 rdms-prod-resc1.data.rug.nl:/mnt/11/Vault iadmin mkresc Vol12 rdms-prod-resc1.data.rug.nl:/mnt/12/Vault iadmin mkresc Vol21 rdms-prod-resc2.data.rug.nl:/mnt/21/Vault iadmin mkresc Vol22 rdms-prod-resc2.data.rug.nl:/mnt/22/Vault iadmin addchildtoresc ReplA Vol02 iadmin addchildtoresc ReplA Vol11 iadmin addchildtoresc ReplB Vol01 iadmin addchildtoresc ReplB Vol22 iadmin addchildtoresc ReplC Vol12 iadmin addchildtoresc ReplC Vol21 iadmin mkresc pta passthru iadmin mkresc ptb passthru iadmin mkresc ptc passthru iadmin addchildtoresc pta ReplA iadmin addchildtoresc ptb ReplB iadmin addchildtoresc ptc ReplC iadmin mkresc Randy random iadmin addchildtoresc Randy pta iadmin mkresc pt_top passthru iadmin addchildtoresc pt_top Randy p216149@pg-interactive:~ ilsresc pt_top:passthru └── Randy:random ├── pta:passthru │ └── ReplA:replication │ ├── Vol02:unixfilesystem │ └── Vol11:unixfilesystem ├── ptb:passthru │ └── ReplB:replication │ ├── Vol01:unixfilesystem │ └── Vol22:unixfilesystem └── ptc:passthru └── ReplC:replication ├── Vol12:unixfilesystem └── Vol21:unixfilesystem proof: p216149@pg-interactive:~ ils -l /rug/home/g.j.c.strikwerda@rug.nl: g.j.c.strikw 0 pt_top;Randy;ptb;ReplB;Vol01 515106669 2019-06-13.16:48 & tivo.tar.gz g.j.c.strikw 1 pt_top;Randy;ptb;ReplB;Vol22 515106669 2019-06-13.16:48 & tivo.tar.gz file: tivo.tar.gz is stored on Vol01 and on Vol22 (replicated by ReplB resource) p216149@pg-interactive:~ iput ./package.tar.gz p216149@pg-interactive:~ ils -l /rug/home/g.j.c.strikwerda@rug.nl: g.j.c.strikw 0 pt_top;Randy;pta;ReplA;Vol02 36609 2019-07-03.11:24 & package.tar.gz g.j.c.strikw 1 pt_top;Randy;pta;ReplA;Vol11 36609 2019-07-03.11:24 & package.tar.gz file: package.tar.gz is stored on Vol02 and on Vol11 (replicated by ReplA resource) client-config looks like this: p216149@pg-interactive:.irods cat irods_environment.json { "irods_client_server_negotiation": "request_server_negotiation", "irods_client_server_policy": "CS_NEG_REQUIRE", "irods_connection_pool_refresh_time_in_seconds": 300, "irods_default_hash_scheme": "SHA256", "irods_default_number_of_transfer_threads": 4, "irods_default_resource": "pt_top", "irods_encryption_algorithm": "AES-256-CBC", "irods_encryption_key_size": 32, "irods_encryption_num_hash_rounds": 16, "irods_encryption_salt_size": 8, "irods_host": "rdms-prod-icat.data.rug.nl", "irods_match_hash_policy": "compatible", "irods_maximum_size_for_single_buffer_in_megabytes": 32, "irods_port": 1247, "irods_transfer_buffer_size_for_parallel_transfer_in_megabytes": 4, "irods_user_name": "g.j.c.strikwerda@rug.nl", "irods_zone_name": "rug", "schema_name": "irods_environment", "schema_version": "v3" } backup-strategy: july 2019: - icat: daily dump of pg-database to /var/backups/ daily backup to our tivoli TSM system - resc-servers: daily backup of /mnt/vol/Vault/Trash/ to our tivoli TSM system so we only backup the trash! Which is most of the time the data users want back after error-deletion metalnx webfrontend for irods: checkout software: $ git clone https://github.com/irods-contrib/metalnx-web.git create db for metalnx on postgres: $ (sudo) su - postgres postgres$ psql psql> CREATE USER metalnx WITH PASSWORD ''; psql> CREATE DATABASE "IRODS-EXT"; psql> GRANT ALL PRIVILEGES ON DATABASE "IRODS-EXT" TO metalnx; change config: vi /home/ger/metalnx/metalnx-web/etc/irods-ext/metalnx.properties: $ cat metalnx.properties irods.host= irods.port=1247 irods.zoneName= irods.admin.user=irods irods.admin.password= # metalnx database settings db.driverClassName=org.postgresql.Driver db.url=jdbc:postgresql:// run: $ docker run -d -p 8080:8080 -v /home/ger/metalnx/metalnx-web/etc/irods-ext:/etc/irods-ext --add-host hostcomputer:172.17.0.1 --name metalnx irods/metalnx:latest connect: http://icat-server:8080/metalnx/ Future work: - clean up the trash regularly (script?) - build more irods environments/playgrounds to learn/test/play/fun - set up auditing (ampq with ELK stack backend) - set a performance baseline - find out user needs (budget, storage, performance) - create replication-check-scripts (check/pinpoint/report missing replica's) - do some disaster drills/scenario's - create 2 resource servers in irods on datahandeling nodes (Lustre backend, IB network, direct connected to peregrine) - performance testing (what will be the current bottleneck?) - adding more icat-servers (behind F5 loadbalancer) connected to a separate database(cluster) (icat-scaleing) - create landingzone on peregrine (for irods to pick up files automated) - compute-to-data, data-to-compute testing - irods-hpc-testing: integration metadata BeeGFS, integration metadata Lustre, let iRODS read changelogs@metadata - storage-tiering: tape-archive - test out this new iput-on-steriods for HPC performance testing/differences - test with S3 object store as storage-backends (big-data-not-on-filesytem, but big-data-object-storage)