irods rug 2019
This commit is contained in:
parent
f63b475035
commit
797633f42f
|
|
@ -0,0 +1,379 @@
|
|||
|
||||
basic-design:
|
||||
|
||||
create a datavault-storage abstraction system "data-as-a-service"
|
||||
|
||||
start-simple: "grow-as-you-go"
|
||||
|
||||
1 icat-server (icat-service + postgresql database local vsan)
|
||||
OS: CentOS7
|
||||
3 resource-servers (with 2 local mounts each)
|
||||
3 datacenters
|
||||
2 replica's of data
|
||||
1 replica in 1 datacenter, other replica in other datacenter
|
||||
- encrypt storage (because cloudstorage)
|
||||
- all servers are esx vm's (rug-cloud)
|
||||
- all storage is vmware datastore (rug-cloud)
|
||||
- all irods-servers/clients connect via SSL
|
||||
- authentication via ldap
|
||||
|
||||
connection from peregrine to irods-servers is 10 Gb ethernet
|
||||
|
||||
irods-lingo:
|
||||
|
||||
icat-server: server containing metadata database
|
||||
irods-resource-server: server with mountpoint storing data
|
||||
provider: icat-server
|
||||
consumer: irods-resource server
|
||||
collections: directories
|
||||
objects: files
|
||||
|
||||
local-lingo:
|
||||
|
||||
peregrine: our HPC cluster in Groningen
|
||||
|
||||
irods installation on centos7 2019:
|
||||
|
||||
the icat-server:
|
||||
|
||||
- basic/normal configuration
|
||||
- disable selinux
|
||||
- enable/configure firewall
|
||||
- set/enable ntpd
|
||||
|
||||
# rpm --import https://packages.irods.org/irods-signing-key.asc
|
||||
# wget -qO - https://packages.irods.org/renci-irods.yum.repo | sudo tee /etc/yum.repos.d/renci-irods.yum.repo
|
||||
|
||||
# yum install irods-server irods-database-plugin-postgres
|
||||
|
||||
# yum install postgresql-server
|
||||
|
||||
# postgresql-setup initdb
|
||||
Initializing database ... OK
|
||||
|
||||
# systemctl start postgresql
|
||||
|
||||
# su - postgres
|
||||
Last login: Fri Oct 26 11:30:44 CEST 2018 on pts/0
|
||||
|
||||
$ psql
|
||||
psql (9.2.24)
|
||||
Type "help" for help.
|
||||
|
||||
postgres=# CREATE USER irods WITH PASSWORD 'xxxxx';
|
||||
CREATE ROLE
|
||||
postgres=# CREATE DATABASE "ICAT";
|
||||
CREATE DATABASE
|
||||
postgres=# GRANT ALL PRIVILEGES ON DATABASE "ICAT" TO irods;
|
||||
GRANT
|
||||
postgres=# \l
|
||||
List of databases
|
||||
Name | Owner | Encoding | Collate | Ctype | Access privileges
|
||||
-----------+----------+----------+-------------+-------------+-----------------------
|
||||
ICAT | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 | =Tc/postgres +
|
||||
| | | | | postgres=CTc/postgres+
|
||||
| | | | | irods=CTc/postgres
|
||||
postgres | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 |
|
||||
template0 | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 | =c/postgres +
|
||||
| | | | | postgres=CTc/postgres
|
||||
template1 | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 | =c/postgres +
|
||||
| | | | | postgres=CTc/postgres
|
||||
(4 rows)
|
||||
|
||||
|
||||
# vi /var/lib/pgsql/data/pg_hba.conf:
|
||||
|
||||
host all all 127.0.0.1/32 md5
|
||||
host all all 129.125.77.0/32 md5
|
||||
|
||||
# systemctl restart postgresql
|
||||
|
||||
# python /var/lib/irods/scripts/setup_irods.py
|
||||
The iRODS service account name needs to be defined.
|
||||
iRODS user [irods]:
|
||||
iRODS group [irods]:
|
||||
|
||||
+--------------------------------+
|
||||
| Setting up the service account |
|
||||
+--------------------------------+
|
||||
|
||||
Existing Group Detected: irods
|
||||
Existing Account Detected: irods
|
||||
Setting owner of /var/lib/irods to irods:irods
|
||||
Setting owner of /etc/irods to irods:irods
|
||||
iRODS server's role:
|
||||
1. provider
|
||||
2. consumer
|
||||
Please select a number or choose 0 to enter a new value [1]:
|
||||
Updating /etc/irods/server_config.json...
|
||||
|
||||
+-----------------------------------------+
|
||||
| Configuring the database communications |
|
||||
+-----------------------------------------+
|
||||
|
||||
You are configuring an iRODS database plugin. The iRODS server cannot be started until its database has been properly configured.
|
||||
|
||||
ODBC driver for postgres [PostgreSQL]:
|
||||
Database server's hostname or IP address [localhost]:
|
||||
Database server's port [5432]:
|
||||
Database name [ICAT]:
|
||||
Database username [irods]:
|
||||
|
||||
-------------------------------------------
|
||||
Database Type: postgres
|
||||
ODBC Driver: PostgreSQL
|
||||
Database Host: localhost
|
||||
Database Port: 5432
|
||||
Database Name: ICAT
|
||||
Database User: irods
|
||||
-------------------------------------------
|
||||
|
||||
Please confirm [yes]:
|
||||
Database password:
|
||||
Updating /etc/irods/server_config.json...
|
||||
Listing database tables...
|
||||
Salt for passwords stored in the database:
|
||||
Updating /etc/irods/server_config.json...
|
||||
|
||||
+--------------------------------+
|
||||
| Configuring the server options |
|
||||
+--------------------------------+
|
||||
|
||||
iRODS server's zone name [tempZone]: testZone
|
||||
iRODS server's port [1247]:
|
||||
iRODS port range (begin) [20000]:
|
||||
iRODS port range (end) [20199]:
|
||||
Control Plane port [1248]:
|
||||
Schema Validation Base URI (or off) [file:///var/lib/irods/configuration_schemas]:
|
||||
iRODS server's administrator username [rods]: irods
|
||||
|
||||
-------------------------------------------
|
||||
Zone name: testZone
|
||||
iRODS server port: 1247
|
||||
iRODS port range (begin): 20000
|
||||
iRODS port range (end): 20199
|
||||
Control plane port: 1248
|
||||
Schema validation base URI: file:///var/lib/irods/configuration_schemas
|
||||
iRODS server administrator: irods
|
||||
-------------------------------------------
|
||||
|
||||
Please confirm [yes]: yes
|
||||
iRODS server's zone key:
|
||||
Zone key must be at least 1 character in length.
|
||||
iRODS server's zone key:
|
||||
iRODS server's negotiation key (32 characters):
|
||||
Negotiation key must be exactly 32 characters in length.
|
||||
iRODS server's negotiation key (32 characters):
|
||||
Control Plane key (32 characters):
|
||||
Updating /etc/irods/server_config.json...
|
||||
|
||||
+-----------------------------------+
|
||||
| Setting up the client environment |
|
||||
+-----------------------------------+
|
||||
|
||||
iRODS server's administrator password:
|
||||
|
||||
Updating /var/lib/irods/.irods/irods_environment.json...
|
||||
|
||||
+--------------------------+
|
||||
| Setting up default vault |
|
||||
+--------------------------+
|
||||
|
||||
iRODS Vault directory [/var/lib/irods/Vault]:
|
||||
|
||||
+-------------------------+
|
||||
| Setting up the database |
|
||||
+-------------------------+
|
||||
|
||||
Listing database tables...
|
||||
Creating database tables...
|
||||
|
||||
+-------------------+
|
||||
| Starting iRODS... |
|
||||
+-------------------+
|
||||
|
||||
Validating [/var/lib/irods/.irods/irods_environment.json]... Success
|
||||
Validating [/var/lib/irods/VERSION.json]... Success
|
||||
Validating [/etc/irods/server_config.json]... Success
|
||||
Validating [/etc/irods/host_access_control_config.json]... Success
|
||||
Validating [/etc/irods/hosts_config.json]... Success
|
||||
Ensuring catalog schema is up-to-date...
|
||||
Updating to schema version 2...
|
||||
Updating to schema version 3...
|
||||
Updating to schema version 4...
|
||||
Updating to schema version 5...
|
||||
Catalog schema is up-to-date.
|
||||
Starting iRODS server...
|
||||
Success
|
||||
|
||||
+---------------------+
|
||||
| Attempting test put |
|
||||
+---------------------+
|
||||
|
||||
Putting the test file into iRODS...
|
||||
Getting the test file from iRODS...
|
||||
Removing the test file from iRODS...
|
||||
Success.
|
||||
|
||||
+--------------------------------+
|
||||
| iRODS is installed and running |
|
||||
+--------------------------------+
|
||||
|
||||
installation of irods-resource-server:
|
||||
|
||||
- disable selinux
|
||||
- enable/configure firewall
|
||||
- set/enable ntpd
|
||||
|
||||
install irods-repository:
|
||||
|
||||
# rpm --import https://packages.irods.org/irods-signing-key.asc
|
||||
# wget -qO - https://packages.irods.org/renci-irods.yum.repo | sudo tee /etc/yum.repos.d/renci-irods.yum.repo
|
||||
# yum install epel-release
|
||||
# yum install irods-server
|
||||
# python /var/lib/irods/scripts/setup_irods.py
|
||||
|
||||
set this server to a consumer (resource-server) provider= icat-server
|
||||
|
||||
encrypt storage:
|
||||
|
||||
create keyfile:
|
||||
|
||||
# echo "some difficult string" >> /etc/keyfile
|
||||
# chmod 600 /etc/keyfile
|
||||
|
||||
# cryptsetup luksFormat -y -v /dev/sdb --key-file /etc/keyfile
|
||||
# cryptsetup luksFormat -y -v /dev/sdc --key-file /etc/keyfile
|
||||
|
||||
open encrypted storage:
|
||||
|
||||
# cryptsetup luksOpen /dev/sdb irods01 --key-file /etc/keyfile
|
||||
# cryptsetup luksOpen /dev/sdc irods02 --key-file /etc/keyfile
|
||||
|
||||
format storage:
|
||||
|
||||
# mkfs.xfs /dev/mapper/irods01
|
||||
# mkfs.xfs /dev/mapper/irods02
|
||||
|
||||
mount storage:
|
||||
|
||||
# mount /dev/mapper/irods01 /mnt/01/
|
||||
# mount /dev/mapper/irods02 /mnt/02/
|
||||
|
||||
create resources:
|
||||
|
||||
as user irods on whatever irods-server:
|
||||
|
||||
iadmin mkresc ReplA replication
|
||||
iadmin mkresc ReplB replication
|
||||
iadmin mkresc ReplC replication
|
||||
|
||||
iadmin mkresc Vol01 rdms-prod-resc0.data.rug.nl:/mnt/01/Vault
|
||||
iadmin mkresc Vol02 rdms-prod-resc0.data.rug.nl:/mnt/02/Vault
|
||||
|
||||
iadmin mkresc Vol11 rdms-prod-resc1.data.rug.nl:/mnt/11/Vault
|
||||
iadmin mkresc Vol12 rdms-prod-resc1.data.rug.nl:/mnt/12/Vault
|
||||
|
||||
iadmin mkresc Vol21 rdms-prod-resc2.data.rug.nl:/mnt/21/Vault
|
||||
iadmin mkresc Vol22 rdms-prod-resc2.data.rug.nl:/mnt/22/Vault
|
||||
|
||||
iadmin addchildtoresc ReplA Vol02
|
||||
iadmin addchildtoresc ReplA Vol11
|
||||
|
||||
iadmin addchildtoresc ReplB Vol01
|
||||
iadmin addchildtoresc ReplB Vol22
|
||||
|
||||
iadmin addchildtoresc ReplC Vol12
|
||||
iadmin addchildtoresc ReplC Vol21
|
||||
|
||||
iadmin mkresc pta passthru
|
||||
iadmin mkresc ptb passthru
|
||||
iadmin mkresc ptc passthru
|
||||
|
||||
iadmin addchildtoresc pta ReplA
|
||||
iadmin addchildtoresc ptb ReplB
|
||||
iadmin addchildtoresc ptc ReplC
|
||||
|
||||
iadmin mkresc Randy random
|
||||
|
||||
iadmin addchildtoresc Randy pta
|
||||
|
||||
iadmin mkresc pt_top passthru
|
||||
iadmin addchildtoresc pt_top Randy
|
||||
|
||||
p216149@pg-interactive:~ ilsresc
|
||||
|
||||
pt_top:passthru
|
||||
└── Randy:random
|
||||
├── pta:passthru
|
||||
│ └── ReplA:replication
|
||||
│ ├── Vol02:unixfilesystem
|
||||
│ └── Vol11:unixfilesystem
|
||||
├── ptb:passthru
|
||||
│ └── ReplB:replication
|
||||
│ ├── Vol01:unixfilesystem
|
||||
│ └── Vol22:unixfilesystem
|
||||
└── ptc:passthru
|
||||
└── ReplC:replication
|
||||
├── Vol12:unixfilesystem
|
||||
└── Vol21:unixfilesystem
|
||||
|
||||
proof:
|
||||
|
||||
p216149@pg-interactive:~ ils -l
|
||||
/rug/home/g.j.c.strikwerda@rug.nl:
|
||||
g.j.c.strikw 0 pt_top;Randy;ptb;ReplB;Vol01 515106669 2019-06-13.16:48 & tivo.tar.gz
|
||||
g.j.c.strikw 1 pt_top;Randy;ptb;ReplB;Vol22 515106669 2019-06-13.16:48 & tivo.tar.gz
|
||||
|
||||
file: tivo.tar.gz is stored on Vol01 and on Vol22 (replicated by ReplB resource)
|
||||
|
||||
p216149@pg-interactive:~ iput ./package.tar.gz
|
||||
p216149@pg-interactive:~ ils -l
|
||||
/rug/home/g.j.c.strikwerda@rug.nl:
|
||||
g.j.c.strikw 0 pt_top;Randy;pta;ReplA;Vol02 36609 2019-07-03.11:24 & package.tar.gz
|
||||
g.j.c.strikw 1 pt_top;Randy;pta;ReplA;Vol11 36609 2019-07-03.11:24 & package.tar.gz
|
||||
|
||||
file: package.tar.gz is stored on Vol02 and on Vol11 (replicated by ReplA resource)
|
||||
|
||||
client-config looks like this:
|
||||
|
||||
p216149@pg-interactive:.irods cat irods_environment.json
|
||||
{
|
||||
"irods_client_server_negotiation": "request_server_negotiation",
|
||||
"irods_client_server_policy": "CS_NEG_REQUIRE",
|
||||
"irods_connection_pool_refresh_time_in_seconds": 300,
|
||||
"irods_default_hash_scheme": "SHA256",
|
||||
"irods_default_number_of_transfer_threads": 4,
|
||||
"irods_default_resource": "pt_top",
|
||||
"irods_encryption_algorithm": "AES-256-CBC",
|
||||
"irods_encryption_key_size": 32,
|
||||
"irods_encryption_num_hash_rounds": 16,
|
||||
"irods_encryption_salt_size": 8,
|
||||
"irods_host": "rdms-prod-icat.data.rug.nl",
|
||||
"irods_match_hash_policy": "compatible",
|
||||
"irods_maximum_size_for_single_buffer_in_megabytes": 32,
|
||||
"irods_port": 1247,
|
||||
"irods_transfer_buffer_size_for_parallel_transfer_in_megabytes": 4,
|
||||
"irods_user_name": "g.j.c.strikwerda@rug.nl",
|
||||
"irods_zone_name": "rug",
|
||||
"schema_name": "irods_environment",
|
||||
"schema_version": "v3"
|
||||
}
|
||||
|
||||
Future work:
|
||||
|
||||
- build more irods environments/playgrounds to learn/test/play/fun
|
||||
- set up auditing (ampq with ELK stack backend)
|
||||
- set a performance baseline
|
||||
- find out user needs (budget, storage, performance)
|
||||
- create replication-check-scripts (check/pinpoint/report missing replica's)
|
||||
- do some disaster drills/scenario's
|
||||
- create 2 resource servers in irods on datahandeling nodes (Lustre backend, IB network, direct connected to peregrine)
|
||||
- performance testing (what will be the current bottleneck?)
|
||||
- adding more icat-servers (behind F5 loadbalancer) connected to a separate database(cluster) (icat-scaleing)
|
||||
- create landingzone on peregrine (for irods to pick up files automated)
|
||||
- compute-to-data, data-to-compute testing
|
||||
- irods-hpc-testing: integration metadata BeeGFS, integration metadata Lustre, let iRODS read changelogs@metadata
|
||||
- storage-tiering: tape-archive
|
||||
- test out this new iput-on-steriods for HPC performance testing/differences
|
||||
- test with S3 object store as storage-backends (big-data-not-on-filesytem, but big-data-object-storage)
|
||||
Loading…
Reference in New Issue