92 lines
5.2 KiB
Plaintext
92 lines
5.2 KiB
Plaintext
IB tweaks Mellanox:
|
||
|
||
default ko2iblnd.conf:
|
||
|
||
[root@pg-mds01 ~]# cat /etc/modprobe.d/ko2iblnd.conf
|
||
|
||
alias ko2iblnd-opa ko2iblnd
|
||
options ko2iblnd-opa peer_credits=128 peer_credits_hiw=64 credits=1024 concurrent_sends=256 ntx=2048 map_on_demand=32 fmr_pool_size=2048 fmr_flush_trigger=512 fmr_cache=1 conns_per_peer=4
|
||
|
||
install ko2iblnd /usr/sbin/ko2iblnd-probe
|
||
|
||
ko2iblnd-opa is the 'wrong' interface, we use Mellanox cards.
|
||
|
||
To check for example the peer_credits setting in this example:
|
||
|
||
[root@pg-mds01 ~]# cat /proc/sys/lnet/peers
|
||
nid refs state last max rtr min tx min queue
|
||
0@lo 1 NA -1 0 0 0 0 0 0
|
||
172.23.52.179@o2ib 1 NA -1 8 8 8 8 -8 0
|
||
172.23.52.35@o2ib 1 NA -1 8 8 8 8 -4 0
|
||
172.23.52.124@o2ib 1 NA -1 8 8 8 8 -11 0
|
||
172.23.52.69@o2ib 1 NA -1 8 8 8 8 -11 0
|
||
172.23.52.158@o2ib 1 NA -1 8 8 8 8 -11 0
|
||
172.23.52.14@o2ib 1 NA -1 8 8 8 8 -11 0
|
||
172.23.52.103@o2ib 1 NA -1 8 8 8 8 -11 0
|
||
172.23.52.192@o2ib 1 NA -1 8 8 8 8 -10 0
|
||
172.23.52.48@o2ib 1 NA -1 8 8 8 8 -11 0
|
||
172.23.52.137@o2ib 1 NA -1 8 8 8 8 -9 0
|
||
172.23.54.2@o2ib 1 NA -1 8 8 8 8 -8 0
|
||
172.23.52.82@o2ib 1 NA -1 8 8 8 8 -11 0
|
||
172.23.55.212@o2ib 1 NA -1 8 8 8 8 8 0
|
||
|
||
You can see the peer_credits have max value of 8..
|
||
|
||
|
||
should be like this:
|
||
|
||
[root@dh2-mds01 ~]# cat /etc/modprobe.d/ko2iblnd.conf
|
||
|
||
options ko2iblnd peer_credits=128 peer_credits_hiw=64 credits=1024 concurrent_sends=256 ntx=2048 map_on_demand=32 fmr_pool_size=2048 fmr_flush_trigger=512 fmr_cache=1 conns_per_peer=4
|
||
|
||
After reboot/reload of lnd module you can check the peer_credits:
|
||
|
||
[root@dh2-mds01 ~]# cat /proc/sys/lnet/peers
|
||
nid refs state last max rtr min tx min queue
|
||
0@lo 1 NA -1 0 0 0 0 0 0
|
||
172.23.53.156@o2ib2 1 NA -1 128 128 128 128 -19 0
|
||
172.23.57.43@tcp12 1 NA -1 8 8 8 8 6 0
|
||
172.23.53.4@o2ib2 2 NA -1 128 128 128 127 -49 672
|
||
172.23.53.9@o2ib2 1 NA -1 128 128 128 128 126 0
|
||
172.23.53.1@o2ib2 1 NA -1 128 128 128 128 102 0
|
||
172.23.53.158@o2ib2 1 NA -1 128 128 128 128 -1601 0
|
||
172.23.57.45@tcp12 1 NA -1 8 8 8 8 6 0
|
||
172.23.53.6@o2ib2 1 NA -1 128 128 128 128 121 0
|
||
172.23.53.155@o2ib2 1 NA -1 128 128 128 128 -92 0
|
||
172.23.57.42@tcp12 1 NA -1 8 8 8 8 6 0
|
||
172.23.57.47@tcp12 1 NA -1 8 8 8 8 6 0
|
||
172.23.53.8@o2ib2 1 NA -1 128 128 128 128 126 0
|
||
172.23.57.52@tcp12 1 NA -1 8 8 8 8 6 0
|
||
172.23.53.157@o2ib2 1 NA -1 128 128 128 128 32 0
|
||
172.23.57.44@tcp12 1 NA -1 8 8 8 8 6 0
|
||
172.23.53.204@o2ib2 1 NA -1 128 128 128 128 127 0
|
||
172.23.53.5@o2ib2 1 NA -1 128 128 128 128 100 0
|
||
172.23.57.49@tcp12 1 NA -1 8 8 8 8 6 0
|
||
172.23.53.10@o2ib2 1 NA -1 128 128 128 128 126 0
|
||
172.23.57.41@tcp12 1 NA -1 8 8 8 8 6 0
|
||
172.23.53.2@o2ib2 1 NA -1 128 128 128 128 103 0
|
||
172.23.57.46@tcp12 1 NA -1 8 8 8 8 6 0
|
||
|
||
On ib interfaces you can now see the peer_credits are on 128, on tcp (not changed) the peer credits are still on default(8).
|
||
|
||
Futher explanation of the settings:
|
||
|
||
peer_credits=128 - the number of concurrent sends to a single peer
|
||
peer_credits_hiw=64 - Hold in Wait – when to eagerly return credits
|
||
credits=1024 - the number of concurrent sends (to all peers)
|
||
concurrent_sends=256 - send work-queue sizing
|
||
ntx=2048 - the number of message descriptors that are pre-allocated when the ko2iblnd module is loaded in the kernel
|
||
map_on_demand=32 - the number of noncontiguous memory regions that will be mapped into a virtual contiguous region
|
||
fmr_pool_size=2048 - the size of the Fast Memory registration (FMR) pool (must be >= ntx/4)
|
||
fmr_flush_trigger=512 - the dirty FMR pool flush trigger
|
||
fmr_cache=1 - enable FMR caching
|
||
conns_per_peer=4 - create multiple queue pairs per peer to allow higher throughput from a single client. This is of most benefit to OPA interfaces, when coupled with the krcvqs parameter of the OPA hfi1 kernel driver. The hfi1 driver option krcvqs must also be set. It is recommended to set krcvqs=4. In some cases, setting krcvqs=8 will yield improved IO performance, but this can impact other workloads, especially on clients. If queue-pair memory usage becomes excessive, reduce the ko2iblnd conns_per_peer value to 2 and krcvqs=2.
|
||
|
||
The default values used by Lustre if no parameters are given is:
|
||
|
||
peer_credits=8
|
||
peer_credits_hiw=8
|
||
concurrent_sends=8
|
||
credits=64
|
||
|