docs/irods_2019_rug.txt

basic-design:

create a datavault-storage abstraction system "data-as-a-service"

start-simple: "grow-as-you-go"

1 icat-server (icat-service + postgresql database local vsan)
OS: CentOS7
3 resource-servers (with 2 local mounts each)
3 datacenters
2 replica's of data
1 replica in 1 datacenter, other replica in other datacenter
- encrypt storage (because cloudstorage)
- all servers are esx vm's (rug-cloud)
- all storage is vmware datastore (rug-cloud)
- all irods-servers/clients connect via SSL
- authentication via ldap

connection from peregrine to irods-servers is 10 Gb ethernet

irods-lingo:

icat-server: server containing metadata database
irods-resource-server: server with mountpoint storing data
provider: icat-server
consumer: irods-resource server
collections: directories
objects: files

local-lingo:

peregrine: our HPC cluster in Groningen

irods installation on centos7 2019:

the icat-server:

- basic/normal configuration
- disable selinux
- enable/configure firewall
- set/enable ntpd

# rpm --import https://packages.irods.org/irods-signing-key.asc
# wget -qO - https://packages.irods.org/renci-irods.yum.repo | sudo tee /etc/yum.repos.d/renci-irods.yum.repo

# yum install irods-server irods-database-plugin-postgres

# yum install postgresql-server

# postgresql-setup initdb
Initializing database ... OK

# systemctl start postgresql

# su - postgres
Last login: Fri Oct 26 11:30:44 CEST 2018 on pts/0

$ psql
psql (9.2.24)
Type "help" for help.

postgres=# CREATE USER irods WITH PASSWORD 'xxxxx';
CREATE ROLE
postgres=# CREATE DATABASE "ICAT";
CREATE DATABASE
postgres=# GRANT ALL PRIVILEGES ON DATABASE "ICAT" TO irods;
GRANT
postgres=# \l
                                  List of databases
   Name    |  Owner   | Encoding |   Collate   |    Ctype    |   Access privileges   
-----------+----------+----------+-------------+-------------+-----------------------
 ICAT      | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 | =Tc/postgres         +
           |          |          |             |             | postgres=CTc/postgres+
           |          |          |             |             | irods=CTc/postgres
 postgres  | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 | 
 template0 | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 | =c/postgres          +
           |          |          |             |             | postgres=CTc/postgres
 template1 | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 | =c/postgres          +
           |          |          |             |             | postgres=CTc/postgres
(4 rows)


# vi /var/lib/pgsql/data/pg_hba.conf:

host all all 127.0.0.1/32 md5
host all all 129.125.77.0/32 md5

# systemctl restart postgresql

# python /var/lib/irods/scripts/setup_irods.py 
The iRODS service account name needs to be defined.
iRODS user [irods]: 
iRODS group [irods]: 

+--------------------------------+
| Setting up the service account |
+--------------------------------+

Existing Group Detected: irods
Existing Account Detected: irods
Setting owner of /var/lib/irods to irods:irods
Setting owner of /etc/irods to irods:irods
iRODS server's role:
1. provider
2. consumer
Please select a number or choose 0 to enter a new value [1]: 
Updating /etc/irods/server_config.json...

+-----------------------------------------+
| Configuring the database communications |
+-----------------------------------------+

You are configuring an iRODS database plugin. The iRODS server cannot be started until its database has been properly configured.

ODBC driver for postgres [PostgreSQL]: 
Database server's hostname or IP address [localhost]: 
Database server's port [5432]: 
Database name [ICAT]: 
Database username [irods]: 

-------------------------------------------
Database Type: postgres
ODBC Driver:   PostgreSQL
Database Host: localhost
Database Port: 5432
Database Name: ICAT
Database User: irods
-------------------------------------------

Please confirm [yes]: 
Database password: 
Updating /etc/irods/server_config.json...
Listing database tables...
Salt for passwords stored in the database: 
Updating /etc/irods/server_config.json...

+--------------------------------+
| Configuring the server options |
+--------------------------------+

iRODS server's zone name [tempZone]: testZone
iRODS server's port [1247]: 
iRODS port range (begin) [20000]: 
iRODS port range (end) [20199]: 
Control Plane port [1248]: 
Schema Validation Base URI (or off) [file:///var/lib/irods/configuration_schemas]: 
iRODS server's administrator username [rods]: irods

-------------------------------------------
Zone name:                  testZone
iRODS server port:          1247
iRODS port range (begin):   20000
iRODS port range (end):     20199
Control plane port:         1248
Schema validation base URI: file:///var/lib/irods/configuration_schemas
iRODS server administrator: irods
-------------------------------------------

Please confirm [yes]: yes
iRODS server's zone key: 
Zone key must be at least 1 character in length.
iRODS server's zone key: 
iRODS server's negotiation key (32 characters): 
Negotiation key must be exactly 32 characters in length.
iRODS server's negotiation key (32 characters): 
Control Plane key (32 characters): 
Updating /etc/irods/server_config.json...

+-----------------------------------+
| Setting up the client environment |
+-----------------------------------+

iRODS server's administrator password: 

Updating /var/lib/irods/.irods/irods_environment.json...

+--------------------------+
| Setting up default vault |
+--------------------------+

iRODS Vault directory [/var/lib/irods/Vault]: 

+-------------------------+
| Setting up the database |
+-------------------------+

Listing database tables...
Creating database tables...

+-------------------+
| Starting iRODS... |
+-------------------+

Validating [/var/lib/irods/.irods/irods_environment.json]... Success
Validating [/var/lib/irods/VERSION.json]... Success
Validating [/etc/irods/server_config.json]... Success
Validating [/etc/irods/host_access_control_config.json]... Success
Validating [/etc/irods/hosts_config.json]... Success
Ensuring catalog schema is up-to-date...
Updating to schema version 2...
Updating to schema version 3...
Updating to schema version 4...
Updating to schema version 5...
Catalog schema is up-to-date.
Starting iRODS server...
Success

+---------------------+
| Attempting test put |
+---------------------+

Putting the test file into iRODS...
Getting the test file from iRODS...
Removing the test file from iRODS...
Success.

+--------------------------------+
| iRODS is installed and running |
+--------------------------------+

installation of irods-resource-server:

- disable selinux
- enable/configure firewall
- set/enable ntpd

install irods-repository:

# rpm --import https://packages.irods.org/irods-signing-key.asc
# wget -qO - https://packages.irods.org/renci-irods.yum.repo | sudo tee /etc/yum.repos.d/renci-irods.yum.repo
# yum install epel-release
# yum install irods-server
# python /var/lib/irods/scripts/setup_irods.py 

set this server to a consumer (resource-server) provider= icat-server

encrypt storage:

create keyfile:

# echo "some difficult string" >> /etc/keyfile
# chmod 600 /etc/keyfile

# cryptsetup luksFormat -y -v /dev/sdb --key-file /etc/keyfile
# cryptsetup luksFormat -y -v /dev/sdc --key-file /etc/keyfile

open encrypted storage:

# cryptsetup luksOpen /dev/sdb irods01 --key-file /etc/keyfile
# cryptsetup luksOpen /dev/sdc irods02 --key-file /etc/keyfile

format storage:

# mkfs.xfs /dev/mapper/irods01
# mkfs.xfs /dev/mapper/irods02

mount storage:

# mount /dev/mapper/irods01 /mnt/01/
# mount /dev/mapper/irods02 /mnt/02/

create resources:

as user irods on whatever irods-server:

iadmin mkresc ReplA replication
iadmin mkresc ReplB replication
iadmin mkresc ReplC replication

iadmin mkresc Vol01 rdms-prod-resc0.data.rug.nl:/mnt/01/Vault
iadmin mkresc Vol02 rdms-prod-resc0.data.rug.nl:/mnt/02/Vault

iadmin mkresc Vol11 rdms-prod-resc1.data.rug.nl:/mnt/11/Vault
iadmin mkresc Vol12 rdms-prod-resc1.data.rug.nl:/mnt/12/Vault

iadmin mkresc Vol21 rdms-prod-resc2.data.rug.nl:/mnt/21/Vault
iadmin mkresc Vol22 rdms-prod-resc2.data.rug.nl:/mnt/22/Vault

iadmin addchildtoresc ReplA Vol02
iadmin addchildtoresc ReplA Vol11

iadmin addchildtoresc ReplB Vol01
iadmin addchildtoresc ReplB Vol22

iadmin addchildtoresc ReplC Vol12
iadmin addchildtoresc ReplC Vol21

iadmin mkresc pta passthru
iadmin mkresc ptb passthru
iadmin mkresc ptc passthru

iadmin addchildtoresc pta ReplA
iadmin addchildtoresc ptb ReplB
iadmin addchildtoresc ptc ReplC

iadmin mkresc Randy random

iadmin addchildtoresc Randy pta

iadmin mkresc pt_top passthru
iadmin addchildtoresc pt_top Randy

p216149@pg-interactive:~ ilsresc 

pt_top:passthru
└── Randy:random
    ├── pta:passthru
    │   └── ReplA:replication
    │       ├── Vol02:unixfilesystem
    │       └── Vol11:unixfilesystem
    ├── ptb:passthru
    │   └── ReplB:replication
    │       ├── Vol01:unixfilesystem
    │       └── Vol22:unixfilesystem
    └── ptc:passthru
        └── ReplC:replication
            ├── Vol12:unixfilesystem
            └── Vol21:unixfilesystem
            
proof:

p216149@pg-interactive:~ ils -l
/rug/home/g.j.c.strikwerda@rug.nl:
  g.j.c.strikw      0 pt_top;Randy;ptb;ReplB;Vol01    515106669 2019-06-13.16:48 & tivo.tar.gz
  g.j.c.strikw      1 pt_top;Randy;ptb;ReplB;Vol22    515106669 2019-06-13.16:48 & tivo.tar.gz

file: tivo.tar.gz is stored on Vol01 and on Vol22 (replicated by ReplB resource)
  
p216149@pg-interactive:~ iput ./package.tar.gz 
p216149@pg-interactive:~ ils -l
/rug/home/g.j.c.strikwerda@rug.nl:
  g.j.c.strikw      0 pt_top;Randy;pta;ReplA;Vol02        36609 2019-07-03.11:24 & package.tar.gz
  g.j.c.strikw      1 pt_top;Randy;pta;ReplA;Vol11        36609 2019-07-03.11:24 & package.tar.gz

file: package.tar.gz is stored on Vol02 and on Vol11 (replicated by ReplA resource)

client-config looks like this:

p216149@pg-interactive:.irods cat irods_environment.json
{
    "irods_client_server_negotiation": "request_server_negotiation", 
    "irods_client_server_policy": "CS_NEG_REQUIRE",
    "irods_connection_pool_refresh_time_in_seconds": 300, 
    "irods_default_hash_scheme": "SHA256", 
    "irods_default_number_of_transfer_threads": 4, 
    "irods_default_resource": "pt_top", 
    "irods_encryption_algorithm": "AES-256-CBC", 
    "irods_encryption_key_size": 32, 
    "irods_encryption_num_hash_rounds": 16, 
    "irods_encryption_salt_size": 8, 
    "irods_host": "rdms-prod-icat.data.rug.nl", 
    "irods_match_hash_policy": "compatible", 
    "irods_maximum_size_for_single_buffer_in_megabytes": 32, 
    "irods_port": 1247, 
    "irods_transfer_buffer_size_for_parallel_transfer_in_megabytes": 4, 
    "irods_user_name": "g.j.c.strikwerda@rug.nl", 
    "irods_zone_name": "rug", 
    "schema_name": "irods_environment", 
    "schema_version": "v3"
}


backup-strategy:

july 2019:

- icat: daily dump of pg-database to /var/backups/ daily backup to our tivoli TSM system
- resc-servers: daily backup of /mnt/vol<number>/Vault/Trash/ to our tivoli TSM system

so we only backup the trash! Which is most of the time the data users want back after error-deletion

metalnx webfrontend for irods:

checkout software:

$ git clone https://github.com/irods-contrib/metalnx-web.git

create db for metalnx on postgres:

$ (sudo) su - postgres
postgres$ psql
psql> CREATE USER metalnx WITH PASSWORD '<db password metalnx>';
psql> CREATE DATABASE "IRODS-EXT";
psql> GRANT ALL PRIVILEGES ON DATABASE "IRODS-EXT" TO metalnx;

change config:

vi /home/ger/metalnx/metalnx-web/etc/irods-ext/metalnx.properties:

$ cat metalnx.properties
irods.host=<ipaddress icat-server>
irods.port=1247
irods.zoneName=<your zone-name>
irods.admin.user=irods
irods.admin.password=<irods admin pass>

# metalnx database settings

db.driverClassName=org.postgresql.Driver
db.url=jdbc:postgresql://<ip-address icat-server:5432/IRODS-EXT
db.username=metalnx
db.password=<db password metalnx>

run:

$ docker run -d   -p 8080:8080   -v /home/ger/metalnx/metalnx-web/etc/irods-ext:/etc/irods-ext   --add-host hostcomputer:172.17.0.1   --name metalnx   irods/metalnx:latest

connect:

http://icat-server:8080/metalnx/



Future work:

- clean up the trash regularly (script?)
- build more irods environments/playgrounds to learn/test/play/fun
- set up auditing (ampq with ELK stack backend)
- set a performance baseline
- find out user needs (budget, storage, performance)
- create replication-check-scripts (check/pinpoint/report missing replica's)
- do some disaster drills/scenario's 
- create 2 resource servers in irods on datahandeling nodes (Lustre backend, IB network, direct connected to peregrine)
- performance testing (what will be the current bottleneck?)
- adding more icat-servers (behind F5 loadbalancer) connected to a separate database(cluster) (icat-scaleing)
- create landingzone on peregrine (for irods to pick up files automated)
- compute-to-data, data-to-compute testing
- irods-hpc-testing: integration metadata BeeGFS, integration metadata Lustre, let iRODS read changelogs@metadata
- storage-tiering: tape-archive
- test out this new iput-on-steriods for HPC performance testing/differences
- test with S3 object store  as storage-backends (big-data-not-on-filesytem, but big-data-object-storage)