From fa327ba3ff83e57226c61e460b9f93b88e907877 Mon Sep 17 00:00:00 2001 From: Jack Leland Date: Wed, 6 Mar 2024 14:07:55 +0000 Subject: [PATCH 01/21] Add comments with additional ideas --- docs/source/deployment.rst | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/docs/source/deployment.rst b/docs/source/deployment.rst index baac7c0a..55f0e4e7 100644 --- a/docs/source/deployment.rst +++ b/docs/source/deployment.rst @@ -291,4 +291,8 @@ everything on this page, this was true at the time of writing (2024-03-06). - Uses ``nlds-cache-02-o`` tenancy, ``nlds-cache-01-o`` also available * - API Server - `https://nlds-master.130.246.130.221.nip.io/ `_ (firewalled) - - `https://nlds.jasmin.ac.uk/ `_ (public, ssl secured) \ No newline at end of file + - `https://nlds.jasmin.ac.uk/ `_ (public, ssl secured) + + +.. Possible additional sections: +.. - \ No newline at end of file From 0126d86650689c0ece957820f13c792204ed198a Mon Sep 17 00:00:00 2001 From: Jack Leland Date: Wed, 6 Mar 2024 17:13:24 +0000 Subject: [PATCH 02/21] Make minor corrections to deployment page --- docs/source/deployment.rst | 43 +++++++++++++++++++++----------------- 1 file changed, 24 insertions(+), 19 deletions(-) diff --git a/docs/source/deployment.rst b/docs/source/deployment.rst index 55f0e4e7..8505b0a0 100644 --- a/docs/source/deployment.rst +++ b/docs/source/deployment.rst @@ -48,18 +48,19 @@ The string after each of these corresponds to the image's location on CEDA's Harbor registry (and therefore what tag/registry address to use to ``docker pull`` each of them). As may be obvious, the FastAPI server runs on the ``Generic Server`` image and contains an installation of asgi, building upon the -``asgi`` [base-image], to actually run the server. The rest run on the ``Generic -Consumer`` image, which has an installation of the NLDS repo, along -with its dependencies, to allow it to run a given consumer. The only dependency -which isn't included is xrootd as it is a very large and long installation -process and unnecessary to the running of the non-tape consumers. Therefore the -``Tape Consumer`` image was created, which appropriately builds upon the -``Geneic Consumer`` image with an additional installation of ``xrootd`` with -which to run tape commands. The two tape consumers, ``Archive-Put`` and -``Archive-Get``, run on containers using this image. +``asgi`` `base-image `_, +to actually run the server. The rest run on the ``Generic Consumer`` image, +which has an installation of the NLDS repo, along with its dependencies, to +allow it to run a given consumer. The only dependency which isn't included is +xrootd as it is a very large and long installation process and unnecessary to +the running of the non-tape consumers. Therefore the ``Tape Consumer`` image was +created, which appropriately builds upon the ``Geneic Consumer`` image with an +additional installation of ``xrootd`` with which to run tape commands. The two +tape consumers, ``Archive-Put`` and ``Archive-Get``, run on containers using +this image. The two consumer containers run as the user NLDS, which is an official JASMIN -user at uid=7054096 and is baked into the container (i.e. unconfigurable). +user at ``uid=7054096`` and is baked into the container (i.e. unconfigurable). Relatedly, every container runs with config associating the NLDS user with supplemental groups, the list of which constitutes every group-workspace on JASMIN. The list was generated with the command:: @@ -67,7 +68,7 @@ JASMIN. The list was generated with the command:: ldapsearch -LLL -x -H ldap://homer.esc.rl.ac.uk -b "ou=ceda,ou=Groups,o=hpc,dc=rl,dc=ac,dc=uk" This will need to be periodically rerun and the output reformatted to update the -list of ``supplementalGroups`` in [this config file]. +list of ``supplementalGroups`` in `this config file `_. Each of the containers will also have specific config and specific deployment setup to help the container perform its particular task its particular task. @@ -80,8 +81,9 @@ tasks, which some, or all, of the containers make use of to function. The most commonly used is the ``nslcd`` pod which provides the containers with up-to-date uid and gid information from the LDAP servers. This directly uses the -[``nslcd``] image developed for the notebook server, and runs as a side-car in -every deployed pod to periodically poll the LDAP servers to provide names and +`nslcd `_ +image developed for the notebook server, and runs as a side-car in every +deployed pod to periodically poll the LDAP servers to provide names and permissions information to the main container in the pod (the consumer) so that file permissions can be handled properly. In other words, it ensures the ``passwd`` file on the consumer container is up to date, and therefore that the @@ -181,7 +183,7 @@ The, relatively new, solution that exists on the CEDA cluster is the use of `fluentd`, and more precisely `fluentbit `_, to aggregate logs from the NLDS logging microservice and send them to a single external location running `fluentd` – currently the stats-collection virtual -machine run on JASMIN. Each log sent to the `fluentd`` service is tagged with a +machine run on JASMIN. Each log sent to the `fluentd` service is tagged with a string representing the particular microservice log file it was collected from, e.g. the logs from the indexer microservice on the staging deployment are tagged as:: @@ -189,9 +191,9 @@ as:: nlds_staging_index_q_log This is practically achieved through the use of a sidecar – a further container -running in teh same pod as the logging container – running the fluentbit image -as defined by the `fluentbit helm chart `_. -The full `fluentbit`` config, including the full list of tags, can be found `in +running in the same pod as the logging container – running the ``fluentbit`` +image as defined by the `fluentbit helm chart `_. +The full ``fluentbit`` config, including the full list of tags, can be found `in the logging config yamls `_. When received by the fluentd server, each tagged log is collated into a larger log file for help with debugging at some later date. The log files on the @@ -200,7 +202,7 @@ not exceed the pod's allocated memory limit. .. note:: The `fluentbit` service is still in its infancy and subject to change at - short notice as the system & helm chart get more widely adopted. For example + short notice as the system & helm chart get more widely adopted. For example, the length of time log files are kept on the stats machine has not been finalised yet. @@ -246,7 +248,7 @@ these were arrived at by using the command:: Ctrl + ` within the kubectl shell on the appropriate rancher cluster (accessible via the -shell button in the top right, or shortcut |sc|). ``{NLDS_NAMESPACE}``will need +shell button in the top right, or shortcut |sc|). ``{NLDS_NAMESPACE}`` will need to be replaced with the appropriate namespace for the cluster you are on, i.e.:: kubectl top pod -n nlds # on wigbiorg @@ -295,4 +297,7 @@ everything on this page, this was true at the time of writing (2024-03-06). .. Possible additional sections: +.. - Helm charts? +.. - API Server config? (this is related to Helm charts) +.. - NLDS chown .. - \ No newline at end of file From 374288b6019f55e058854b9dc93cbc5a3d52db20 Mon Sep 17 00:00:00 2001 From: Jack Leland Date: Thu, 7 Mar 2024 13:48:08 +0000 Subject: [PATCH 03/21] Add prototype logo and icons --- docs/source/_static/icon-black.png | Bin 0 -> 2626 bytes docs/source/_static/icon.png | Bin 0 -> 3697 bytes docs/source/_static/logo.png | Bin 0 -> 14988 bytes docs/source/_static/nlds-logo.png | Bin 0 -> 23185 bytes docs/source/_static/nlds.pdf | Bin 0 -> 3067 bytes docs/source/_static/nlds.png | Bin 0 -> 7446 bytes docs/source/conf.py | 7 ++++--- 7 files changed, 4 insertions(+), 3 deletions(-) create mode 100644 docs/source/_static/icon-black.png create mode 100644 docs/source/_static/icon.png create mode 100644 docs/source/_static/logo.png create mode 100644 docs/source/_static/nlds-logo.png create mode 100644 docs/source/_static/nlds.pdf create mode 100644 docs/source/_static/nlds.png diff --git a/docs/source/_static/icon-black.png b/docs/source/_static/icon-black.png new file mode 100644 index 0000000000000000000000000000000000000000..1b4b68e881506df2e3a5ab88d11c994a78abf889 GIT binary patch literal 2626 zcmWlbdpy(oAIHat&8@i}x5_HzbhLF`7BST35@A9vF*}NWR3nP!dt8cAq*ggw7>dIh zrOXZvW|@@S&WX9plqNHm&YElE_wA3*_w#tXAK%yO`F_8?pYQkaz2)oerU5sCLm&_h z4|kj&xR-$Iq3Sm9dS`rNH@Ly#+=CM!5KaD%rSOK#JOdUD6Y)WbC&^KXNu*0>AxTL| z2mg05HX)J}clIFpQgq>hgApj~^T4_ICl}2#LaW09^lr4qhxmUv;ctDkYJ2l*-1XrK zxKWMPH7#v1v^XIo^l|Cyb0P0TN?k+szPi944nk~HZ)ZB^nin|d7W`y$JeSaJUJ;&M zinBy2>f=nX)tt_Sk(83+-;|HyiIe#m?IdI*D`x_Nrb&Q&SXPt!j(Z zYG0xQ%n(;Kmg0pRo_0jAeDOEou)n>{&5V2&Wmo(}$8Mxb#;L>Z&jJN0t`_&Y0e^$< zQ<<`;PeH;!#MK8Z?B{aBJ?0~krvh3v#*0`VOb%-_v=LD5UgAk$r?RH`b35Zg{PcOZ zmR)8ZJ(aAx(7B9gzd?qw!bNQ=?eMHJ%wzMfDwfl$mH5@)eumyIi@FD?3hX*4taRmU z#OudV%fjzLbfizkX&Dy6LpkfvVM}C9!ho7R%l;fApq`sz$br&oPti1G@IbJqeNuntV=tG{#OC%^Ntm z+LmM&?Lr3fXFg2zgCym=f=EwzfAifKEdTZdZ$)FSZ+Z{tauhCQKqT7K`aMj+F;i@q ziocipBUI-+e2#{NCJ~%go+j#wlt|-)RtI1^bwxIg_4X_zu zEb>xrnJ9?b222=Bn++QFxb(OH>Z@fW`J-wj>?bBBu-tYhPY+pZ?=rE&UqkKSD)+~9>#{fq2wDzC^tq5?~`_5%I)u&EoFMIk`_tfXI!l?h1mqkD2S5DS3 z9n1#uwiakUwMD3-3J2uayfePblO(4`WT7vj8_9@c}DpVV6AP@jq$o; zgRC{nh|+r0Ryh+A)^}KV0h=C+pU#IRruYgrUi{uTdvioq#{e@#Pfwyct_02e42F^O z6uZxLQjU;G z`L#vX^Nlln_OdQ3U0zMJT*}1D6{_36P)6I_I}lXeq?U^ZIYxt>x>%d0Vq3I0?0{^C zWd+}!MS;du;^!6Gm;U`FH>P+X3CkQs4H*y2FCoCmyE2STWJbqsiJ-}?+12d*MHax{ zt~l%$=8)=fo1MIH{og^=3dl4`FpCBA@<4W+P8GVeCjVx*5%Qc z9q&Z?OX=s~1Fs7Yy_$PwK11^ui58Ey5Z4=QKb6aO;h z*Q7%k^J9w!Uo}d+A(&u|lQy@^$JZd?F#!$W@lx6G^8FWnR|d*XtmmP#0En1ksK4CF zr$D3Q%dlZ1M1{%$6I$a_zci6gFQ>w!ri`3)6%$!U)FN%QCrQ^|h7bZ^npI1VVjH#i znL;RQJxpj69n7}&v1n(I7tX6JTppN{$ZNX^{rL^j9x7#~-vUU&IvNhoFlUa_$*N+8 zPyLZD@7r(d*cECd?o^|G!&q=ajljd(?$#u5>#%BD4M;OI|xb_)%zQip%`@ zZlFo!KRi&FYSvLR8|V@QYuy?Ra}e!s%0&|)rL5LB zTf#>j+eKF9{SxwSRq2}0abrxi4bVJOAh#?=m*~hOi-N`>f;I_DJgA(SpMv;vyC|W& zSk6$=oC)`YWv&$w1klyKO??kKN-HIOaXK7#u?;X}MyFvEZ(gd|yN|^6Vv3LPlpBYr z570!{Uwbz7kI_+1gIsB#piQDEY7+Wfl_Z+3ycr{0-vA2s3-1B0$E@ZC3 z_x4L_4v@IDorgB&T&Ys{EpGgcq@ z*(14z)F#_&X0rexirq2xzfK96UPBsSxAw0VZ^VL}j5GEt7&{VIbpO-%WO?SA(Q(I( zCWY0M1x_~&O}zdToLUtA27pOzip+nYMqPIQKv|YPWw(;JfBe%vGKnTOy^UX3Y=%iW zKS;fWXkypfM6jSL<=g`cnkVQrExJw{O>ym8TWGqMXFK+BorZ$-lRpzF-Sm308vB{0qGr&lpq)o zq(_1tNDvT2I)b4{lOshCzMS(t&;4_M-1}qg^~}s(vuDj-^R9Q2?QJdi`K0&&0N}T> zG`qr@t5~Cshl}-e{o`sLYvK*Jbd3UlFSr* zN|gZt)MGyQd)zVN)jB?kzm^RCvSV|@X}9Jit>Chd#t?@ezFohwo;yAL@s904`Yb&U&p7Ev zx>Bspm z*U&-T7E`ib6+#7@S!$Vv9p23SA1IV4R@m@V0f_F3(TO|W8-qUB^<yw9tlUJFwR!A?K%ue{Sd}L_LkO z7lUX1#TjoA2oOY5n>8;$*A>}$nUzvtXAE26MCLAkQW!))!DD>t?37x5NstPb$O*I? zy%FgsjIRuP?b~C7mb3U|sSE}KmDDi#nZVhnTny8?GGC_2EX-N4NUdtUHs|?IWzIqs z*w5LXlQwG*y1kC@mo9eX8;wl~kXJe#Gp81s*HZJ0?yx1k+{NDUJY%22zWX@*0W-q> zQbl{ahpt7x<03fcuLX8oy(Fxq-+r(~mlBW|PI_K3APxZz+JXyw2gIrOcnWT;ORS4B z9HPFNv4VeUx65&mh&?W}7R*??U5ib94)x~)1D=}ok`JtAv)&~XPE58E;=@y)gDUvk zl|2Z8g`#mt-Q_{W3~_vd_ApG>83k-`3mdVAKf(#lZadMa&U1+6TwNCwkagi)28isX z+XIStadYdl)-3Gi@m$sKy}dn>q)|S!wQkVG1Lb6+wi0@NI*~K(N^G|y5cZgkDLgre zxK(cPF+-jXVMQJj$;~5R#nc>%hKD{!spXZ>`<7Ul(ScyuamKoSr|}+#J`B&7c$gCh zpZ>)7fmW8b;8bWw!lP^F5H~Y@NP5~#R^ktlk>}4+teAp}sn}bi8H0%BVFd?P=(}%g z1vk#jW=SY2zs4HfX6;mu*ego0N9kX#`}`0{<*{gODRNs9q1#soX0sAMlls$6bGBe1 z9p{a|RqTAhqSdvOm9T}FA$Z&^8zydMHM_4S+J;#va}GMV7zBtROxjQdrxC~wv4=L96Y_Z_Q|Pzk)v-;+#IWHn_)_j1a>x`&Hu}*u z)AFVH)P+-dC8=jUQP(|$Yj2n3=gaJUw+;cvRU|mh$wVP??%-a8r`{!YX+H5MvX0zZ%;*a-UpeAQM?ha>$0pZ2&apG^E_}mzTKK&TV!S6og6nZ0wXqB`vLC znbgFxsABrI#9DZy)n&o|>O? z4f_bXUjMedo$y<>^C{^-im@a8qNnX-u2A6d7@-+PxgKr2%<&+eDj?b9qQ^Z$u=KYG zOJgLT+9KxtwSP@=r=t0A#gKg{cV@BZg+7~sdR}+Z&!w`OH)oH^z)`p5H}_yv>p_8o zhYvz>JL{lr0yf`(x3CD6<+7S6|F6mS5Us}J!yrPe*AFYd=F2Sbb&+rvUdyr zoGZ^d93X4*1fbTJ_3$;h-7Hq*!wc^h_}zqk|{_o zsSch~Sb~kZHpSy_xbWIWmM*|{xXLv6Svxi8qOC3Ksbqs@s;5-{X6w-=*X@;=vjE&9 z3jO1U#-|OOlJkf2M^At~X*ZHgFE$(6j?IHiPv%Siwhw)@H+r>tB3d@iqAUuz82H!s z=1T@N2Z!BhhQf8Jpun*)WyvQjNdA*62l#c*=Xe}v{&{`XJ4;s?@}I25Oy3Jsa?l*lHWc&Tyt zZnU$(#USaijlYGFdj8kA-lsGYtKz9yS{w(ix+?Vf46Ir!P@!0*>P>L>{@x`6 z7-iDsXXx9hb*$f<@Gys^M}@6hsMH$|aBaNoa)?k+Y5W(|Z7Wet@%qobnp4j^1;jqj zriQ6AvRc#s=aoNT6$4UfXh+AcbHmMlgk*=O>cDHd zM;0o6{d3I-ex}mn#iktW4zjh~mtPrBc0up=C9~4#@*Dh1^vxSfFsQ01{Ff0dmrpxO zW%Du+8z#5Xh8?XDux4hJ7m!JNhs~KuA367Q^7(o0MxLr~G}fglt5tFS4||Fi=)0ZR zemc+l31s96Nn)C7#tP(+GbH`BKr78Vfu)<8`qphW%!fADy`QBln=x7MhDy2rJDJ9- z8LkM|*zuD;{Gu$&3_GaA^z*FxfjhbWR6Coq7Mn`ZIDC#S(=3!ZN)aa)<#N7t6 zyuB*?2qGFoM>?tYeE*kS9rH;Z=prRd7fkS?S#++GAR;j5>oQ$7Ge6+)8u&!un2n3C z430JYBRjoOAqZo7qM&`v9y!casFL@t!^LrIGp8&n=rS0pwa`ZOMo$IFeE@9$JT6kb ze@d!_@E8&{=Lnn|+NKf38rm;(Sen)M)zw`&p`eZXtFB$DLUcMKFcQ162!2 z%J{)6n{d=%Xpt3wF;$1_M`=r>vrJ&MkX|vqQCT0qO`ZIjfiAs5C(A>Lj`s0<#bXWR z)c+El>1`M#5F>q7>s18l^tLM~*ueBMcqQ%1(Jy}>W^5Y@r6prW^PRCn-!#V=$Dh51opa1T@C;FFuFB z6|7rVyyVO)RuQFv7{yZEfxiWj`XLN?D=<_i<1+-X8Q}pi?6_sdD1O%6~8Z&m1#_tFA#TCzy9r->d|4P+tQdK;Ux+ zhx5^FPeT|EXWTyPX-y225B#poK8b%Cc6zQfj1Fv%O~1^}#=TGuSnf-PuWH^P335r~V@qvC1~SbGqJ; z=D~90WKfBH)70Y#cCX+gt_TQko&w@mUFk#!C6!f~hH^1O+c2gH*qvGKmHkTvc@&CI zt!I5PN>LH&f4wv0Ok6j`9|3hk5lxEr2+8Dk+c1Aq@Jj%AcUbNzK7tFveDb1#mQ8v` tal%oPp&?GkhwT1ejnV(nH+?t)l{}eh){Rb~v)V<#>Y}Y#jS2ehe*meO_7?yE literal 0 HcmV?d00001 diff --git a/docs/source/_static/logo.png b/docs/source/_static/logo.png new file mode 100644 index 0000000000000000000000000000000000000000..4c77be376bc5a75e3b0ce673f724efbf441d2c3c GIT binary patch literal 14988 zcmXY2c|4Tg_n*bsm$4@MzHgaq*%|vjmMqEIlr8&6$V|x^g~Cuo_ARnx%|wz6Wgjsa zkrEA`#E9to_`QCA%xhk+`#jIR=bU@*IrqNL`(!xUTd*+;FoQrKHY-ac3IqbD1HTML zdZ6WjXyqN?k14{^BN_x^A3FVkN5X3ZfyT2jX6`Z0;pmt+zo;vqxVSi#pz9&g0e%rz zRKlaKmQeKsKp-)Y71G${#^aylxGrJW?vektZPGyEDO`}LJV)*a>{H?r%29O+-+6gm zGUa~o!uP(wjA5a^&(g(wjUC~18~rqqn;g=23*eUV+oB=tIy&FB3=kpv#O+5 z+8Os?Ns0!Uqtm3JB6YOF{iY@A-d6HKySgKvP9VjUy9k6 zST8YO`c73fn~?NI!c2ZxUsnn@kz?%T@?@HNBGe_=#ozUYESmY)g4rHsa(E6Qxnx1& zAaT22xGpjJ4nep}h^L+yZK6j#5iVlcBt~)%9;+qpe<1jfkF(#!OGyAv)!SyVst^W$rwqt93hMM-<&Al2;EczeGI=<88eazzi^ZA5}WkVU zEu_umt`sGd+o#W(I8A<@DU5{hrb8?>zb~*H$hSl-otK#OW)}KFRei&birsy~?ih>i zJR5PwkN(H|$mw*R!Dhk`8gt_Xq!n4ghm=BhO6?2~%4;37W`2o9wDc}>>00vE8LbhC z>#$eW{7=qH0YhD<5IhKROG&6LHP;GAt|VgdVat!NHv!wklwZW%SvoFjz2kDXM>EP7 z&4ORJk1KJCga^&5QAE-ES_m%%Op$^ki$uVv-S;-^Y`|4c`CKN#Me=bT$@smka{9gB z!5T{`nRhHd6+l(&J-JN-(&QC8#MFJ!A)KjA&BD4_maslZ;sp75ws2+W%xz0NK8ddj z%FB@3$I8&G+2&J>?@n*60P7Jg|tk@U5TeOftz}IT0$Fw;>v`QAzNya@mPdHoA zLIt6laQ$?IU>Rs{?@@N^9UiGz{( zFM$u3W6$mRqb=Z?G+@r^N?QrQYOw_dDCeVlU|XsK)|SBVab{fC+lh=zK8JZgQt%NY zhF%_AdkF<5GI#JssiRcsJz=S8>obf9FVF#Zi^Gm=h}E_Hs*qf}Jk(=TEYO=syY`Juhuvgg&j=Sg2n{S6~@o*Lnz0Nvj)qGqPB zlagzWv2Qv+_$36>S81<}UNx_dp z|8gG0tjS5BRBWIZ@e9GNcLM9$VG=M^W-*qz+>7Y!|=sg0jS)Lzb4Z-VZs# zQp#1q#zhN<1RucYxqp+2KX!NvR0RP|)`T#lpm6>S%Zu7edpja*k4l z?hM=#d-DAU1e_>nm%^RvFTnJ7me_;&XgiNNhhNYnxL$jcm(H`ONH}~-`>-a=a=ms?o5H$0;61)bK_I5(t+mGJwU^f}-g*{qI3YL|%J zu`3>z9eO?w6R4l1!~49Qf!A3hbb~*S5iZl)!zN;ATK;sFnl+!!YBL_tkqxaKU;L*W z_s?SsKK?xun4~+nUZ$b9$eV&3LM(sB$yeX7PfIRXuFLrv`kuUttaL|%ytbsr*skc)~0MbNI2lMH>MO%&bfe9b6Zfg0(93Nb8$L^nDSu~06RDA66WxB)}V z1B?_+?v>fKKT140IXpw+)RPjDc(8mZ&eA;X5?iRCZWi(z;W<`v>G+uBci{>;giEu~ zLl(oAab2OjVPlkJOpjVqsU8BB(~Yc_m|t4&9fd$k$L9t=lG*~U=6{8qnecbb0K>Z zwBwSMF%KH?3zrFHACoc^O2DCCsI+1BDKAbh_{g&_j z2}Ijnz|%H4!Ul<*ke(=yOIG7Q3H#*s%XR`w$HBgFzBjj&xW6{%DQKAJmX zO147T#eoCJ$3IEN{^#2CaRW{*9N~E-9w|-M+nK_2nR})OhPB!*3v|Bs70xh#;s)Rc z*8zpb`v_;zZs}h=m?v+mwQ&+mQ#1~$ZdOCR;ubysc&N7-mj8@t*jl9CirRvk$-^(g z-3A^8ANKW@aqA*;0oR*Svey7a4kuffdGAs5IaB1ezj~WP`4`74XKJ;7f1_?g%4ArR zAO1K?_xA!Oy%tc3kD7Ec*A%SQbjZu+ct-%J($%y?= zPB}f0?YAyAJ8GbdXBbUxdFJL+w9u3SJE^>h1_dl7C4N4WGkg1hM}gfH)0$~ImXjlF zRDmzEq*X+P8%t&Vr%&IQw|Te9(%=V`;|=>{7;#I*cZj4TRV=)8T(76f{|pw%c=~Nt zrs?7>PF*Z=udR=k{jS5PXaPRs(%iE=%>Hp$1$6nout~*PHld<#u-)s%iD%r^zlB&w z+R~kmsfc?Y`>zV`dJodZaH346~C(QrqDdm}hQJcRu>h2bjz*6;ButUFk+)I_y zMZqlVH}vI8$NU-Oa9+5J{%H?tUtK}p+3j*v0qkjD?S9d`ep8>)LlWzYZGe;Ukd&};~M?Hb4$z#|vsK6;obBra>6Yv_!lz|t0sWC7Mm_%4{ zsN9`*H;~SA%IVLdx>9+Toq_Ad7F>EaVhe;~WWX=@dpyx=(+Wp;@n&DjqfC;(*58Lo z;7WKZ$YWGF-m#44_|mp+CceLG2|X-@7kw@I1i26FEra`&Q8WJ8BCg zT{+;q(W=+t!zVjY5Bo3de2C||W1QvT=X7Bs=C6zho8x1>J#HDr&h?%zPkBlWepWjS zU8;IHi+eI{5X7gPt_n}}$@pCa*EM3Y6K56W+laLMBx@UM(H;1F<3B>}jJ?8@mXy@) z>JP+fkj&=v)mtqDvnRkdk^?B2XeSErDp7(kLgN<7~ z{~idF{}#U5*J?AHy5JEEOp6;G#%?naCEx%YCH;$iB$AY^GPxg4KYq8*>-(AKgY2Io z>jSAn%^kiSnHEO1qK^amT<*1Zb{mWWWwT$c({SIgqbbS%N@*!DY!!CpyMe_^9l8Tk z0a@!RV7=TXfP)^}sB3_`=)3ScZ;vZ`=bz+;pm(`7ZIDFMme;g59%>pl&^ygtrJgxT+=oqU9w77X zGT_>9ic9^g8o7D5TQyn))!5zZUfY;g>C~I7TN)Jg+mi&J7SM2tI2PR4Qj$OOyfEDs zD6wD%p_W|p?cvOmA5@>s-%>%m=_O2MN|pT8kZ+)5%T;HjKDtw7PffWeo+BfUqP@jiM%-uJ{=V=vQpjgq%N=vvGcEvyr6rVQ>#MLh5Q@Lp%Af-sMh z#F_2(bdC0R{R4B{ zjmItQ>2J<~59)h;xOCGkIS_&jkLg*no>UE@L5eX`B_AjC7!iwb-n1eWh_l~W=qoTrdH}%Jpm_l9DT88PBE8XYpBt_1P6rGT{>ocx;pAL^`esn=>-hZ3HSSb zZ|>UYhu74TMUprYh={V#<$F(_nFzTLlgq;Q?*8l)!F6VTjtNT2BY>Rz-RBmHF1-4MO*zGjo+yEhuJ0aG!b6H zT^#P(m;&aA!OHH7p7Bn)%IR#5k?+Pb6xdO2R0ENNWs;7#k+_LSv4|ckL@onHO*o zIUk3m7H-~aDHyQakQVd^8fx8AW)Gcw&*B+oymh|$J|FZ6#N04n${qRU9uIMR!3v8m#brDGh>rNm*v7KpdSuFb&( zC&$<1An95I$?Xp-PV#pKe8W0}W{_6)`JbHtpyO}dgO_sE6mTjVZ~EVvr2FA6Vs9(?kw2VqaHfI`;io1)UX5rez&ES^zikfC z*Tj52p~VU3E;}rSTK;jW@S9XXxOZv@ub#O(8mGr^(U;I%)sPfa&ti&eo zP25do#$(B^W7quA&n!o?>G^VmEBu2$r`28*i@9?l`O`zIdV8|XpN4qRipx0uN5=3v zhp}SI*`N^I-&r9Z$LB_kk8iOD`63V{su;%H^iDvO+vYPjpI*jY3;uwznyt9DR^;*! z-^AF+qiX?UL$sOE-}EIwa+N6U{?)ma1wE^IX4awoB(TE!+DWn6+1va%bM%>2)Gr+{C+!~8mRf*a**TD(${sc*kA}-#k9Cy-KpM+x0Bd{Xp8Gy zrhf~)54zubq#NYjU?KyAb-xDo-*z2hncrTMz?mxY^d-M9XjWhp#S~5R0|*|q=zYDF zZn5~33>(HVkHXd*ekIwf&PkgSvHNH z5JS&Wd?{ixY>zs5myW-FUj7bT>dwYG@7lWhKL;5y0M{n@)*+r;{7=Tj!oPDAJh;#J z`{xqoJ{$h`pXU1rUsB3jvf$<>%BHLOE>ph*cAP0MsVA51=eGpVooW$s5j+9{r3Gw9 zp9KYLjT%|?N_?tO#mtZC9)G7QbVAkI!f@O;Hk84hErp5VH2C2^#OQn>b{>EWF%F^V zUG=Nqx8A3yw74=21zY5gC*?5_m9SV?cKqKA zYVSDPN%5>CLodcoNxZv*&lYQRJY@1s7)#%ua3wX8BA09Eb+bJ3y~WXZc=d z{ClQF_GgoRJIhWBUYPB1J%?m;G_8RcDohC-Ty8rb{@<%*{_m-a6rtM>N1+561$HQ+ zO`p-*u!)a>p&7OhPVGLzQEz!`@ut06O6u%g-k>hW(q@@#ddrOXi61kEu|D8rjW&DF zfT}fWIQP1f&>3Bu!k~Ji-wAm$F`_L^y#xHZmXq!gdMo+KiqCATAvK_oo}+#0En|WTN@)Zr@DhG})Oq(dvB0Uz}Bj)C5R!OjrLqbHpinJEIneB7Bm$va}J;Y=0N z43UUnZOQ8uU8kmlS4aUf-v9!pPjwkjE)mndk#bSH&UQ17<^@`rfM7AC7iPUA) z>8o;P-f=728Y1#Z5qS)sjuzU&a$s(eG|`@;O5$KFI4*gKJYY-WK4BefA)BW{3QHDx z@Tr?9t6XhyOpeM&7Z#JG;ypWt{TSLs!0yF?$CcAKa>_zjQJ|&YR&fHwbmoleQw+)XQKw#2uAFAwh%aqdOT5@|u zDb%LVPn#ZAnzcgxlIqQy8m=&!nbRG*ii-kv1k&*%fs4e5D1JdLw z0u?_jqmt=269}KFC%kcG1r9HwtoTrisXKArANC32c!nP)R-6ASNjl8+%YE9s>FdHP zRBrz?x!M5Sl}$Z^?3RhK<>>d$o>YS&G6(&wDN5altWAwgvY|{Ur($$Sy-{UnczA{$ z>tkMBBqx5sg2_g8UY(aTOnyEpkCN^>%1s%K>QG`9Dsz6^oX483t&Yio1lJo`CR+rJ zJTJ~E-#=kvKSgCfDBE=;3U<{^td?oCImO@IQ2z!N2vvVs&73la!7u(yDkMB25a%EU z9^d2#;%Lchk7|OKlIV9}gCb|3__UVpoZq>PetWxsBGq*zGYatNqo6Gi3)3;}^}iSHXREc{ zgfc29daQ2#PIYL^xfN1c2Xk5;CHLLDRzM7O9h8Xfc$|NcZ^iS+{h#w&KR!(uZO7{$ z9<#CmS%U5;v|u$f83?R|w=?lu_{vSws0X2DH#Tl~FH%F{aFmU>{*R`?#^n2l*DE@*i{D!-r^}2(O0$J4C_nnen@YWy zH=Lf2IVh)J$m@9Bt{CX98m0o}WmK-DSA7^;GR89-Vtns>8*h{iDsoiDZgW09(-WSp ze^%EB{9CZVz!7%?ktSwVbGJ{RVg9z7??B&eMqRGC%k^f;>2F=c=)3M~0Iw*XzOwg4 z5O@W2(1X{wnNztc$R=`;rK0upHj4!&(-U6IGIh^9{A8>=35&Rj*5%w!ueGo_(5mLR zs{pra=s3U=TOGMS{3*tV6~^ZQ=^yd)pY<5&-K(4rC%5~Uk<;hbnp$u6j_lodnWh3|3u{AP2 zK+Ymy{6w#C;SJynYmNE}OdnJT0>(UZ?dzFfGm~6*1nl6pJ@RZk7H(h5cH*Z0!A%%07q z8oO?z1sd(#1b$S0*bf0xEptC)SKgm%b0uz>Q0})ImKg(sxaA0oz4*p(h;D+EiX|&Xq}N>p?6Mm>*Xzw z_ty6L0h@Vu3K((p4Bzy`ue#Y6ZG&H+F&^m(8AtEhfl47Fwf`0M6#ZDFE7HQGzw=ap z7h{?sla~E_Z{nE3|BjysjdLW5+;~S+IoGCqfo(pF7z#z`W<@;PaW57P*5}sdbWi(K zLSJVzGe42^rW97JI zQPPvS?T{uj_puEfMDJVTD_`_jyGWb+P2RhT*o!#CwPWtB7!$>?fVhF0dD@xg{}MaI zg4No-PHyYW$iX`dJg&7p_En`E0uH=>qhyA9;(+L-*QVFLH)k+8VS14AQdn(~NQE|o zR2Z(u(L9tNE5@NU(;Y#bpib*om|v+$(h)_y_eCkKp33R*#88pAc`o(iq|{HM4;umr z5I?_9Lm-Z#{)~dhH+bDI54a!W7q|(v0vyk&REL=xWPXZPiq?C3JMmDqgsV|YD^#aB zE}Qk(M!I|SbD+0y^S6z57rgaWapB!ZS7OzdI2yfHS704XZpmnjYR|IvB)L%JJMN}o zmjg)fN`6fUt7K!G^KK!^6G$iG7u-@72o}&1Ohk`3p?MTcv0CZL-$^~x^Fw})2F}u- zr!LE)JLOiK|2X>AUG^!IAod+wK9_93cjYr`H6^c7v)#072_6EA-R1(S0>46-b z9Ttjp$oy!lERmR2OPIZf?-%G_~+MaAv<0>Of5J%Wgl9dI~w2RYS z_Ae1_-r!>u26m+tYBL>F=T}Ei-o2VCBApb_-tvuF9u>Tk2m&U)Csj z0)%3s2-Kk1s*&2J7r|9tcIk&rzpiArQ$l3a@4pq{)P*7VKza+?3Hb?MmB;`Snc}XHm?bSFFjKPEXt#z^V>UIH?Kq zrZYIRqUh-#hoil^62@DHrgp{^xTywsHRkVgukK541}mG&@!4?8;8YluO!#NpLE>@S zuShh;F;81rqQ!R|nO@*{c4F~q`uld@)UzUDs6ORVOhC``9`D!L%y0VT>E*Me8CPjq z({=;Jsc+doUVI@nlCmcE%-l3i(F=fLbxh{ho{~J6Yv;KBGg>W4q=7uq6$6iOHhF?I zQ$8!qNZ9#LGB5K^5vCM1&&Ku!NSyD5a+~(*9+-=z*=;Y4@)|J8T3|-U2Ti4O9BID6d z_kT}}pV;ZuJp9Lp?o7uE+ZDA{HJO&gMI#)7G0G*?_A;Wyq6hRl?ReowN-Rzq&n6pZ ziPB|{2WH z<&W<1d%}tEV(r4NYicQ&;ndn-#L#4utQA%xTF|`3*6J>Rlh^696LBE7{xON~r%xbwU$lr^c ze79_-Ef}?^Gz_h}xC2#&Z?chNZ(?0ly2K)}ant)=OU{gc3LovuXjvNE0DBLjX)lr= zvqZ9u#LnHi)@%+y{q*-Nb3Qq!C){t*ve=#soMy@z-76YiVi^bbStazo=!c&7Ebe(! z)lo>_tlq|rT_yK_eg#9Bs;3-U4qoea^V&@)^4R8E{z6Pzy(w}yz(i9*lNQAW2UGi% zUq3e6Sr4Gbmh+*h*`@T;-bXhcPR5QKwAQJ}97J$jv@6cH*Jl^{L7jiYK4nJ?y-Gno z4W_k9B|Td@u0xjs@LN&Z$GDC_hS$iOm)H_jn|^AJDd2hkO8PF6JSurgBWcxz^uaxK zg`pMA4oP_ht%RRwOg{c+l@51w(Sja&-wM&2+w$^m=AFu~@7+q5f_*4glns0;bCnrQ zHGB|l%OwkZXtz;2Z(?W>UikIn-0Q^$(m_WKt2M;Qm^0i)FWUF$tS=}!qjovgLys16 z{sZ8LKjZt4Fp3B-A%2D}=oYdlaG`#bAbnoznP>eb)Hf(@TeqmHx`3I^NPTH9K$$*2 zc%eY8`&UScNAr`=#uzQXiFdPvFzSi%xBtEX7j>*K&Ku{TMw2UZBhuM}PYQR&4D}yj zQEtF-dc-T%w6xvqIN5f@rbtE6t-$j z`c8(ol@XtGvCn&sj{BPt+3RRm8x7%gYEApE+Zg6I*C^!$m!(|W@_Gk89tom5btAYo znkNmxT?id2gc?WFLg^(>6GO!ZD+x|1%^gmJV1k8J+}g4YVUo~2O33|uR;DUZS-#M9 z>A2hN^2aC~NUd#z331zuUY)|>|4Zi3jsnQKpwtJ}W7RV&$~6%Gb9iCLz)yXpeeFZX zZ(HgQJWr?gN_L#(uUI+1U(0GHPW<#rk52x*zf{e&ZVW&p)AToQ=#SmxYJ*PM$xZ9b&xT zqSB#KQtA(8$;TPy4(e@xaQe6*P}stSmnB$dTvndGhk$xp+(xJo_ThXF^~AS9HMqeD z+k>xc)&O87x62QMT$7T?V^Qp%iLo^oCx(<1FN<0THgsW>T=EO86^spOjM*Zq~2UAGQzF%WUf zU{K8<_j6|~ZL{q{H(uF>bd=yd`1tm@6zS=#si`6)zuBH|l%6fWnYvB`i;N8}42(;i&7fs?>JaxyLQg;yHBQXCQ@v2uQ1yJ!?FRI1|8m z7*0SuYDt6-T!El5x9kO|8TJrsLTGf83G zLmC5;e>gp*ZKVLjl)`@n=O?7A)GF#1Q_U;w`YK1D)LU19#9MUmNjOTcFy^HhV?IfW zVHAF`DN9Bl6{K&}1N*W~Xpk>;=vl1sXQ+JuJxyX3F+VPx8$~I?M=!q~aM(BZ!aT9s2m7>`H|F#^o`n z7Jeb8_0Ge)0PLnD_$qcw2_;uknKU@VlG3{jy_CNnU)8`ye-ehSkuFQDvP18iQ7-$A z1D?{c6ry5xRO#{4DAwQn+au}~0uH#0b$V^7k>3*C5N%^P#N=S&)UrdHd!+Vk$&6^} zdE2}mj_kXIpRPk+ZBv88=KRp6xMw)i$?K8oErnRlQOhZpD^s_39AjrPAKcW00>|2r z@lf6IzS0uB@>R4fr#VanNUhFbP<+E+^B8BC~b_D|JEk?>WG5Xb=JvS!=#c<9lZsrd$e zAp&t^{$k`PjF3EiGqQ`fOMWi`6*KUzUfU>DnGu92;o!gTMr$md#c~q-Q&sUue4aZ|GA#!HHm&a)m~-k$xI;M+^68>G&tHnL@>j5DFO) ze`8#}@y~89C(Bayc_wsV2eJ|TYQ<|qc3MIyoR^+qlW;!pp6Lt6M>n3Vy^VdodC3Kv zoFM#1_+R1PhwIhKj0X|X2P00371nOiG}(mYAtFYmT~5<}L*O}Qj|)}&UQzD{vX8^Y zQW(J2V`>|*kRExrws}b4lJ`anl)w91xh-pf=!G00t><(76VA9VkY%sL=Mq`yu=pf_ zgXr}$c@+R|eN*c0muoK;1}MA~Fy*W-WXJzkzkG^MZP4ia%lw2u=vObHqBHIf)uOKX z4ffrU#)|j9Qn4cGg#TIFjCeslbMwNEE@gyz;+egr1@;OoGR|BLjQfX~J2To~#2>-X z;}Jt|8zGmFlcK_ZmUNUNEa(tHE8NtdBvOq-5U#I#3GX~bd8QsN$W?#N zi|1S(UGlo8Lv@U9N_-^wsi z1%?^@d7H~-h!5!l$!$m3_s>&^sX<%P;w8Ots z1c0mi!+9Rx5Fgq8sV^>&RX^zgfM1o_&Aefi$CJ+Rf$j$%Q-XHouPRQXo6_zrNKGqz0GnUBq2n(w{G(jv5Gt zugBYs&T3E%G`F+fD!{Y2G`rk2X<0TRbCI+(|Fqh?$}4*&TS9pm3xu~KngL_3Dnpl! ztcyv@_annb@6wB}g|`CC6vVf;a_sOrBdG}~+tU1BcvT>;hr-F#1QBTd-g=(6ofiGi zuZ^z9tx0Qr;zA@J>Z;ZNyrg0GGxq`B0(o1yQC0Hm)K62V-bv0iH+3^-)?^D-Ut$J^h(`VOv1ZrmD!q`hHx z;zf7LE3jXQy1|(V<*9{%ebEfi0U#iu{xjHV4HUZ__=d;_YL+ZvE0RjS*mY#zYWiEw z6IaC0#Yeh;;j5ZvsjSH!~9+NN<(fWG9?2fc@(4HDB;QD>Y#ECEE5lZwhC;BMrL z{`5@j3ZKyZOjEp&N97j~Oc4&Eo+v%|bGad}gp*)rW!POwAku@cLM;FmRb4mNLD&{N ziMKwh&f-Pd-W4Oi!!MNq2%IR$(8@D&iXO4Z2yjIK`d41Dd!~H^n7_Xq8I<*b1=yKhF94L6ozYJB0pzQ*2yXt|rZ9jft8~Yrf!QhM7|8fFAOIgO_Do0|d{NP;Q zvQ+3Tsjs^bKbA@P);lv(Y%qULii&R&$wLZ(zzu*0Ff^}#v<&%;Te@@yJS~oE+Idg# zgih+M=q<55o>xYoqJ;}7{gS^~oNwLd+-KGqAup#3%Wwb*u@(+L4P3V5NN(a^0{K1w z=H@ob!9nPKS7vQiTXh7qOQegji$s=vlbx@E@M0w4KB5k!{02hnQU$R;$^APCD=~Jb z$IWQmF)%Sp;HqC!Ng!-GD7o|fY_tuy6Z88Ji<|o@BKb#Qj5U!J8DeklinX0VIXc2#lr}S`^OS-X-h}~6%+?O>PqPE*7SbX5RRff+Cs*uw1l5B zm~|YWqS3VZ00t|Yx>I$OjY!Qgp{GyIhQl+5oLlcy)}8mZ)z33q!bq|v^Ky=UI`8}P z1uuh~9}Zwa2Bk! zx)fJmoBmgs>A!dRTkyThqPmv30Opk&%6kJ8fdah9c zP%X-R0$1*{bjxUaD0?jJEWq;T=1tecFUV0KzU0$Rs%cC#!SgTc*nm#*dkvXF&H3pS zC5Cq;TJC0=9=V8JpisBFkHCZzZ0h?Ao{<3@TVB6UrJVv?K7!W9l=W91Ajn$|CxH?L zNwZ{)Q1t)W;cn|1c{l%N&=*z&H7Fq`bsy~jDb`ZWBKzq;3SAy}UUUAW253o7N z1icI1te8f>;}N&)F)VHvY>f0rdw~ZS=736wN{WiuJ;$lO)W>x&&Wzq=#zv06{2w9y z636<`tu^XVt@ayWzQ9Eb#e@$y2Y@S!0y*PWab?{{ze#@ewy+8#VyMy%%b*Z*u0qel z7NwU|1P`{Si1`iqxeRN@F|*n^pjJhX%no;`8&baZOu}` zqZpKkZ<$`SYaM#GG&AfHfz}kzE@&aw@qr$`D6(}Y{o;w6NBx0SK_;b7^X@?M71#+e zz5)@xp&|>v@R(Nhf4cWFwAp znZ~tY%X*HYbK5MlfG3afVPGNfmH=RA)oreQO053ou3E;bC2gw3abGAR_gQzC+Y$IRDqO&euz zPx`jz09@ z*U5<~W57=1qL+@Jko=?)6yUspKrxz+M2ul}rjSKu7*~V{zwRH^CRzl~HNhhV{Zb z^PC^^2u9%^R0DXr?-^eXUkhKyCFE4)JWdfihEpUbMXl*gNdq-i#z;+^i8GOCl z-&i;iAymkTog6bQn5(4S6sWSxU3y8P=eOw^JjK3A9AFRKIs$ibVmQ$Aa);CMoG?GHBBAWeC*k_E{9Y zPSlEUh|*i9BAx+7fq<5BKy{3-R15pc$#Buy2Stl}ywjmTePJYJa;=Nu26EWQ-wF4? z#LC<{8*I+a{I0$~XIR4AW-NrjLWm~ZFcH5Y1b{Bn66;(W7BG_@^C-!_8uvzs5{nX) zRZ@w2D*6q--Ja%-d6cb0=;K|J=d1AmdQm1KUSjD5*uW*Px#0@HvK_|V#l23M4LKv- zZo+3P67rVES7wKOVpMc%pgs*`1l4ot^TTJTNuZxb191H!S$74ehvI>%13u$(*z zJ%uzieZ%*#l{mAo;WF&q_{@cm^~z-((-M8FB7+s-dGRl%Yt6A`!W?H+L)>GUTMFk! z`#E~@vNHqR`>@`Xtr;ko68IzdNVOWCm7Sd(@}N5te2hXGqrpcu_rH0g(4Mkpw5)!F zsBObe6GVYW!INe3fQ50I;k2>whLkl>{L0;3L(HV%QjJ4uEu`wW0>8C@^>J$<>$06D zl`33F&t&mwo^>CoX7&cbt+0)dox&C9FNj|PsdzWNqsJ%1uC2$1iZ+5+veDQqpQe|6 zqC89=sJm@huNcYMz}kRpaDG`itV@l1Fxpf00t{Y?K=0sSF)XlN-k7b0KL=J)aAtbNGmK}rbaZ%X4(pczg*JEyBh zpkt=H%oQyQ^MSV9Gq1%!dGXvs(H0SY{JVO%@${UJJzm{l#0|y%|R(Xo{)t9KYE zshTu2c#(dc-kJVs$3%QivBoTD`+|XGxk9@}6;BmhLC+86oq{D4LbVb|xiuYGqaPch zv&CQ2rX!^pW?cmj++CHqRfrb^LhW$sX@IHw^ffn(6PdU8QP47%Otw(x*xWgKbNc71 z__Ug=Q5orrvRJU>B{-yHzgDBF41kq$R}bwA6FSmz0h=&-r14(1c)<1rK?(dK^bp>{n|#McDgS7+ChDL-m|DgK zw(Eobz%4q2Zcn-QMZvu_P>7LAYLLT{Y;u8KC6bFy5&Pj$-%I(*?-sDc9N9C$jb{pf zC`$w{>go?fQzQr`Qz1=d`ywoDoY9w11s@Tvv~z&Gj+BcM3=uk4xK89^K`y~Whq_W7 zmsR(7cjo9Zv@h45e3e5eZ_TY@=&mmH6d*jOk4r13_phQ5bpF*w&Yt((N9)<-((gP$ z7lvLEWKWLH?sd8!Vb2Z;mgrP0E&VEKr6ckfFz^7-ah^QE63u0e!6w*f3&EomP=23) zjL>q9vT{u8wjdx}P)*Lg@#jwNHFKrHR*7s*2;pa$Zdii^+Q!sZMjokNe+G%-B&yk+s-X2wc>+4=8#!vCr!tX?U2 z=2niLzd@-aidO@@3KoL5NTYsrBV01x7Ymo)C@JM{hwQQP$=9ew3J1JFJNo+M&!ozG zbK?!5dXSrL9*Q;28i*DN6rva5g1HUk!%3#W4_FM1lU&@Nue&<6lfutqNuy61qHU(} zVF~Qr_A5v>2-9CT03y8KcF=9cKR#;=aW&5|QbyI)C=+;h& zOET^>D=p~e15$BKT-HWTcc$|FB42_QecRsMK2WAg52v03HaipS4-(yazxw`g5qJe- zdI$@BMEVN!YUANtw9PVpjB86n&cWTjRx`j|d4^JU3cNr=FLUj|vXrn1*)OsUrJMwB zu`4dJX3Ce!;tOGI$+5=0M>Al?X|*`BGs#tc^7q4aN`kJ*V^`1FwN6Z-Ta~ zZBs82gC1;SnQZtPu}+*s#`XX~^2#|{3X;#k^k9`h(cJ6)oMg~QDkvbx+l|pSvidm* z<}D7HU?T;l`JMVnTlQeO&kUTqfmOcO-YGyXM(u>;SL8A{k3JXe+GK7hHy-FwL`H3T z%2Llk+ADZnFy1Vg@&*>CDn%hHePD@1%%yQ=3a?+?BxA5mSFh(k0cWyxijgr5M$ua0S5EBfB=AGicwVd2BtI zLhpN;W{!3VRY?GHjOLI>7O{9~c<1*iH}|+d7w8NxMUNWWfbHrwa_#8=^{VTkyh@8@ z!_dhtgfk|;J98tks8MQHxgr(FM{Px$yxE6vW-C~CyK7$V5PO7?0k+xwaWP?8Z0gI> z8JaIRP%+!*fy^xBgQ0r*w&GY)NUmO6*&KQqDwL5A?pRC(3nqaSUma(C2}s`rZ_BfV zZL9v=v#@;HDU+0yAxv%R1c$`~EeqL0@X0Hcwhu!cdl1gwex%ATKJfZ$UsTfnw)^uC z?9BClLns@RZxr~@YIuk!G~s&Z6or%0MJb^~QmFsv*7F{5dP`DK0~90xKJFEnz#Y5# zoD&130$>5>0%CA`QR+PBJmsA1oh;}xLvMbDYuicy+aQUm0Jh=V_cgcabDlT5|a;EL5IgrOYA~mBAWu-I<9AMW>|v?GijZ-q=Q- z-r{F;kQ&a3UA{pLg>16bcLv-0(qfJwcirQw{D8m}(=Mh4jonULlCi}8fjF|qm=||% zihb)XCz8(B5N;DFSY?50d4qq?bp|B|sb|rIsOJD0zW+_htkJe^j`-27Yc%kk zB3Kh_ExX4N?!mleZnczbJHE|TRMi!^kARAgs)^tg7JRrb9o(_L~;cqo40D8Cw|cj zSY2q!C6qUrwm?c!TYaU(?$YeFGHP#y9`WW&jvlQ-muuU+pAh5=8#ba~kbxFKNyM~T z{s5E+bg^1Xm%0cTjtc+LeIJdd@C2?dRNo|)H(R$rP*W?EK8nStjZAMJIkX$$ybN@t zBSK<^Q(^ICC(#}?HM?{s8#Q&Ees@$=X6O%k+eK`VBU3(C7HJ%3>HCN3(Dqz8IOgGzerT=Y`o~O-m{w~GM#vS0 z*3P|}FqLDS1r}4Pf}OS|_4qqJQjx)^luBB;dUCu%51oDvO zASJ(JUB!&48r*cBfxrekV&gk6bkbGri7m+rAl! zFbL+Uio~gB2~%@A#2oJzMG)XXNlOZw!V`^fdAh0_x4+f0I7Cu)Kh1PQHsR-(e40SD zdJ&%k;@#P+u8XGB1_7QCnUzUtwUUhaC83OkAEd3YP||&DUNzs#kil`v9H1f?sG$c@ z2mGsKx*C4b9yfk8FEhWxy!FPw0@x#dyEe5lfpe3pRf;fxV14lWKeH)c9bm{X5|)tJG;9YZY{fn@~i_BJIXlQ+PX-M8YaB7?RUj^Vk?>yKZ6 zH)volb2Ubf-Z??!`QM|AyTrf26uu3Tzb7-y}YmK?od zT{ljB4i?Ao!X~nX-z{{`lww2KGl16D1?d6tHECf9LtSm;y=GcLht{!gZ^8h1`OVev z+AnvRvs$v&gCn>xwKfDLV^}cYc+~}aFyF(1_xjqv5C^-qCg;K>h$S|8deC)$N?hH? z9jl*{2Nvs?DV;&f9$iY!@9>&yB9WGk$-D?Ft;*pYNhEO1T5D!j z@}{BHKeus+u=la`6)n|;%Le%3)Gsvqr+}~&S*2KYs!CSD^JIU-cGlonK%YRzpK+Ev zwmoHCbxem@hppbV(CRoLKuTE zSUWU{y^nvT*crbLt7LKDT4Iq=?0MUQLKzG92ahe`n?$VLvYPFW3ULdaKxS^zRr%QJ z**94n61D7xKjV?BB|y3QXG-6r_n{-I*!cavTQcp)~m8l&D$EP4xRFPL}K;@(SJQwq3_v-{dP=g1o_>V|!zv%8^=@#=@9iM?w! zxRcjyWq`N+tNO1-6YU>d^wacUWZ>wNLOT$foTCV&H@eK}Qdi^pJ}Z+Q*nEt#n?sZW zY;o!re_Yzo?AW8Q6ZeU%Y}@0Qo9sBM70@fW%*qUMEl+@HC}P@&E`|(r2tBqa2Rvv; z_(Vqj`-bx<8gN}2TfX3;ma~PQOpCxZIa@%uQmsR^N3Mo;*(nsRb=AErGWx%{cuRQK zqdbEi$zgfIopI4v)JwN`#aua*@sl2?`dO?&X9P?2v|ZbnC(q^4vuX=r6wCcQLtl&t zGElxsjPZC$M!3Ei-QT&akXaeJCe`^57d@76Y5V-u(q>W?FS78Z9dg?+v;bjyGCU&m zNcZ&G&;>FELhJ4OdPgKDa zZoy9PmyQVsonyQKzGv$(q$3pZsqeF*=EjJboa6G^*b+oI=u(wld;O6M25Wv@Y?GMN z&0Akwfmc@7oiqwnyH!a7r~96bybjeO{WaGgH;;MJd!lT$_6QvexmWa> z&qJs&6m1|_CtE8jPjD;l##u{)&H!<4(BgxnY&*DxZ|G>3A_I-v{ru^XfR~Q(J zfT5C2Imi7l!;OTGW!?vTAfUo^_j0w!VDE7!bz7~M^T=chKZAqt&?)0A>KB_a*ed9R z*DGziwjBRbATtJp9@1LYoz#c;7m&$b^$#FPvodKsRsu&&1AB3NAIh+_-LHF7#0o^^ z)DW#)*ll%~pD#?2vA~UU<$+zAnWe7G<$qrraw1#8Gr*=m=WekploAta;CCprQK*js zSgf~@yt9I&4*)il>BCnJeZdTDz;2D&we76jCo$~!lrk%?jOY*i1umaM@PcnQpe_Iy z@tT|lHSA^f9u6O}@VgywvY(=5SzFFv_1MSrgehg`TIy&{Eh<>T1|Tfh7b=OxvF`z+ zX(NoIUT+zvo@s~ zwILTd$KD!*!dWb?vjq)!ez~AMIX0@Q_5s*GAj8h}R<{0`?j}F&mLBSph=IGzHIhwK z5fcVA@GSAK7l2$Nirz4XdISOD#`FjFn0wwT10Kkp_XG+_69r35f0~TDB%!%+TUgu^ zRuuu1cWsCUu9zWpIY(9X!*@+yH=LdVrLDUb3R&dItYk~QC|ZSM4uhr9ZWMUrfRImC z*&`Gp`1liG(B}=f`kA3hjFW>Cs_M+lLK6qYqGl_@gCIreptvGee-fUBriMj)oKCIG z0B4A|FsNt!PAK6exmKu3V8&vvE8TI{0FnSVu-H7B7nau#Wk3QX0{8;30ha;<1GocN z11xZcg8l1vzQlp`dcf*{SeZ1SkRy*rp$io`S*^S?Q};4WEpd+Utcy>MfN0CMrg1+I zZw1Q)8AuOZsxrKbc1r^G=lkc$qaxd0QaZR$Ho>#_NfynF{O57MV3>y_Y<1k1#3FHm zFf;ckSH)=kbt~07?yTCyYryuD$@-h8~dL7CW#ZaT?rz0ycrz%Dpp(-c0JaOY> z;g^2EJ!r`>|5)D<)wkKbgADWxJ+dgv+^IWURaZlkcsb~mluCkDV}M3s(F`#e;{Uuk zIHus~$#qe)hm8`;{Y5*fqdeiFRRKYH^${SxPZxd{**#Nw_PX3$WL^+|TVnqSiahZ! zfh0|?`9dxlBAA%zD$k10EwnD%T%9tbr!Mp>*^+|Xezc@Ju5l@jm^I`QE1CID3HGg) zL{s06jVtuDaA%rBL=u2NV^D|6RcT;7Hsn8X}g~1YaB(qF_N-1Y!~W0rm9H5BKML2aY8}=3pLB zKC^SxlPW(;>p_Ox=9kM|`rj0H%7@>6+X;dM6C8CV>UVfK%2WJW_T&lEpuCqmemD@{ z{vQ|mV}Avywj>%uw2Jorik86|3ggE(w&bR7TKwdv7g8K07q8l195XAs^4fB6^2o^X zaJP++fWe^H6R4?ic5Liapn{G$16d%4nm|8{PDh_p+1O~2SFu1 zGFHEYb+XaR`{z8&?Nxqzc15cSW)F(FEHX9J-nz6{HhSjYkKZ_aW~P+A$)Ux0C?Vff z;CqtyLBumte@dRf^eMnmYfB#*pM>RMy5dJqspygaxa8wcI>Izl^Iu%qo0z-`4AAFV z-<#Fra34XWl^gqOF?f3q6*c+cTJ58%Y~6qy43e~ zxvhXt=ZsszHwLXKbVa}YF2e?LS0CVxvd6LR;HtdCCJRl3wvH09&q_bq>dKDcOS2(b;bgnR33jX zYyj80?)_-=C~6AXrNfzg%j5MM)f%`pQ~%?;hs!jK8p|ekmwJnAG2iDXu#WK!0}5>- z$kPWkg>mxkif(%9qie(7<9-}zHMQ-i4cr8NAj^R5oI3xAf?=Olo{?Lmukfwgxaj(d z($AN4aOyU)CW_-bHwsg0Kb*Os{BJU$yIT88dbd?Ka3VU#N1)r zw_Yt27q)WjZA8b9r;X7X-F$p?kATzE5PhoHgZ))wL}z&JMyI;=`0SeUJyHW3@TB#) z)#ppozJf~)xaSYOYv5nTqqWEDAELQs9ze@fsJc8%I>ZpRQ&dR!C$Q=d@9=}F8yU1W|IyVB`CdJTfrQTmzqoTDxNSVOPTydWM#Q_i@kpC0MW22|Ox5XlixVHW-db}(A^0_8^OAyPS^$%km!P~%7 zVNTW{fxXPVwNUjv@s893H$pq!_I06ilkR{Xu|JvdOJrN1K)#B&3z8QrOsUudsQ?EP z!mW@K(qj4Sq(U-bgHY5DnRNMa$fLfd+#(J{ODP8lfIoI^0w|OS`^#H%`t|yu1h&)5 zs1j-ToWa;@QRk-1186)uls#$f=G^aRZ#!(y0V0?VawPDWZJIQ9>YfyFFeNY3-owXw zfS#8oN1yjOiEcfnB0A#ni06Cz%2U}~AUD4*gi|YUXJx|N+4awL=_hU!5cY}dXx~)f zm%q@xV)GEXC}nIqTr#urX?cq9nb8wl+F{awlWt)qCml-=_WpUtSkd>cEvyZ;18>o? zS_js>mX9ljA$+iQm!I4<9=}x>m1q6_6zz_t)Us6Bh9=#L>Gzk)taNV_2b==cxUJp# zk_Qs%^h4~`I$V_1FXaxYsL!`c*7FK}g;frIK>asLxo^p^tA7!%Jq{ewnm^_rF5$ogGg!O*&uJa%YPv;2?CJM!ds$mfHC!L?Bu_}auJf*V zJz_ZUPx`Y+g({k)$kY{jE5#BSsI0xBMlYNhjk00tJ^Qc!oY&0C>T#+xuze#{VywtO z6WkK4CGoQ?UD8RK?m@{XBhqT)iaxBEsc3)SaqL=L61qkd@O!?Ny^pk(K*e0>u*(Mo zPXdBXeLvUU3{3(DD#ic^Ji(Flz5>YHhg$53nmXoWM4x*UBfg5Kh)@cNyJ%qvPe*)d z(punvzOBPR0XgQfYHKipf&38q_qTFTVq4uIWo_5~GM)J{<4volTB-eIg00VR+~Pit zv_OC3kjc*%AMGP=CLLMX#TbM907DM11&=vWrgc9)n{VWeTFM>S0D%C%5IcUP=Wfuo z96{^+YBW`CXgHDhNh5qqk5~zxd_?MW4;D?1PMjzJgigI&g3v@`zF+EBZitrE%~pv4 zdC&XV%df0)d1CWA>7_zEPtqU1iM$fhQoJ)4FK&NX%Eii5H^3j}mrN?|+(8eGx4)Ot z;z@uSm^uh{Tj&~DKC^q=aI`hHGM7z=Q|K7-El;o`SnaBEVZfgyA!P;D23g>Q#XD~G z26GpXIA4HZaKrS*y%?$aV_ov?Ch)}#`cpngeMJ(#hL|VkOlHsXq1At7+G6__6wjGG z?D>l_{==shATqq^v0_st$e4c9nQkmbeUg1uvjUPCkyyD2&U}G)HT(3U85W2!3K_^sXP)aTgy4ZA|^5a3WV7T_qjpsg<8clBm@$rWG~Tlwo8`i00$`=ep`3jAZUfB1J5HsFOPe?fH$AHDq@-s-mD}LTpN%u1*gg zqG@BzjSc*+DCx;oJT1E#dx7zm6yDI8tOf3jZH+A(_B5sB$_ij}d*t`-KOlLMK?=XO z&-_HUh1^E0V`vK7&+SDGl;88c0d;G#6++^XZ-K(*XKnl+@kCZFe#^6s7Ul3*p;-y;hhBmq`Fg7N2D^J91_D*V(l+zTpxNwjyq zwJQdvD%2U61lyx3R|Z}J8xWe6voNebn01|Xo_DM@f)Nm`gpC?A8(`_WTDR!#@f!H4 z8Kr}EM+F-b2#S?QCtOM)?fxCQ{dcRwL7Xn&_vT;Y7)Zq|`+;bcMQBFOjup5QkbTRw7*t~D@5FkKhVuD>w0W967$>mEwV*hRefi6~YU zb0pK!?i_Iwez#n+e(}rv{}Nxz(Oo$T7)&?Rk?mK?o$4YUeOp=V7l;gO@4Z3b?*ec+ z-i=GGbRPN%c#OgOr{3WV;9ywpMY*Yi+~Tv(@_Q8tq}Sv=7Kg|AFTT&h>dO&mjmx9& z=C9Iq>6Ua1J&DwSmA;i+el$;deghd86Pks2wJEiy%osC^zUga%wM=*_r$2f#7~18z zt0RfUmmz!%8*>KR^&+Qfqy|vJ;lA&9u{VrU&at)r$jte=Is4a^p*qf8hMi(c<1>m~ zANmJ1ln0mp`B`b6-d;K+L4l`o2fY%HELT%8|93D|@n70(TM$g^f#$E8>=1 zoXbk=Y^3apz0)#ye#VoPE=|b=1=i_SaVC7knJNE(A-1~154)SATSjmMyCz7pxW&I# z!Y7z#m~KJ(MIT=fEQe(HeXeB_yf2}Pu}k0F8(tEkaRx)8U{-iF^MTn>BB(Lm!cVgL zv1t+$GB-ZFmvi_AlBkH-K7>>G1?b2r6@!8IWKFr(HKl1*RtkRd^S6NkiAz^Cb8iiD z0qHB5!cZUL!pV>JTQE++vA-$RlmXYR3D0!aGd!$GNhJ12{<0)=IoA?mNoQ2zPAyF3 z)QZRLj<4rX#-O+p2h!!o$;Qupp1NYN{_rH2CU-Qsn)<=okkuHH#BQ8K5{x_{Z&kOw z^C8`r7|Kc$=}WrCVs+JP-q2Dkzou1gM)2h;)K_t#&gO1xfJ zNjKNpbFj)%1lrzY^xEt#`WMn|>iv}=HOoa)T1g~pm9WCZH^i@EL+YWY`tGMqD1cRL zCWWmoh$KhX;p*&r9)tQ7%|QSeTX?{8V>|p(z?o@z;Xw`U(TMMM^uqMS$CZiHOF0Ye z!Qx$^TBUGr(@mlAV_B7YChHdi6c~S#8p2-AX(hjTY`1t{12x51I>t<3%cACkoR*h_ zEN|xOQB{==4e(rWN64o0QZXui?Z`~rKW-Fdig5aj*)Pa1HjxC9p5Bg=Qdx^8hxMKN zu1F`vu6na>OUF!j#`ngA#Lcl)N+-%CLo^0$5ApfZIrnxplG7SE#Q>!iH#3;D9CVW5 z2olKk?v0O)LUI@#WCTnTmn}%^F5R9j&KvuBD>Up?C?Y)TLZMHW%&+PrXPmx^%nBrX z5L^i>I@EyO)$Nj?-26fNmZnc2X=rYrAL0p0k>%xjrYBQsk1^z&9J@4QHp_Mig)ps0 zq+GoyW|g!YEtum)r3by(nMt@w4jw)wVKZ_Ej_GfnSd-^RsDNdhsRXz zwyJ=NBC6nY?n!$UIXqU-woiTwbPVJmQBJI4$M z5ZyZ6Use_C1y+fyuj^P`p3s85R& zD(Pn@+#n~nI=v&I#MQSe2x0Jhdi%q7=dM$kRuDc&COi{I)!2``ioax`i|BnBl~tX( z={kBvCSI>?Z2l4`E3eCrRkA0MB&2j^gXk&qhYkZ%lv`bDH>GdH78u%C`IvPL980)x z@cF4{dZW!Bt4sZ`eIt0`DEAd z(siz+2?Lw(kouCk54PJ%a5I0Ln<)S_8a3`GCR=lpN!b zDf~p+p`*t8arSmx)qXrk0NjXL+h^ixK1`5lpAerzhe8b;dqFKZFcfD8@`lVxv0x0s z5LnbY1?1NABJrBI+VL>SDEl1fo&u;eCx8O{^;|a5$Doqzs!aF6m%6cgis9K|Q66(u zV-14shhv7I4n5t>tXaS0tIE8^?#YEeAubGyKF1Vnzt}quktL83P(z{m;0FLA<>iK_ z8{!YeJY7Pw&ZX7LxvloRbq9!|2sRavARPk5jknQa%o?{H#e>MD&WVX6&!+yip+`GC zUsAWk>fxwHfY+EG??aol$piz!27uDYvIbmaY?B&LSi}4ql`k$q96L)#zLv38#o+7! z;%Wgzcf#hr-mAS#vjYodm3s_A(NYig5Tlh3n8i#Jpt)8zB>DGR9_x;9A zDJ`XV9qLAF!ga@?*YA6!UU-{0$?UplLhQJG#yW0W#!Bc?Hvt1}KqRjO_M6zU@`T>A zdEayr?_$|q&68^;1$8ZdBsEm<{j+d&fQ2qjStMS9Nk3YB`Vz(t1zFG|(QOjYa;f@< zBqgx@Oe8~k(Vr=UC(Z7QB$d|llTl{0=K?EvFARw7mYw`aMYp`OD1$zu$3WNQmWOdFpbEZL88fZ?}gs2 zadznL@1Lq=2ldIM6QCN+#A{>~g~7A{mI1z=x29}=iDHHqs+79c?%JA6A9%%#( z)CSc0;%_{MW_~Yy+vJilz1_t7?f+8SM;&erCB5t!RD^fAv>*fRa6a%d)@ZJVORARj?1V!&?GZ^B zHFeLd$1V45a$}hYYzy~4JWgp0aj7uBV?hd4kwi+im^EBx6Jk@(%Jp&*fyChX>ul9f znx~P&N#6rO!Fz>~Enll3_|OC^03j!_h61eLUFBzkNgc1x{q$#TVH<&2i)ID4B;SN5 z!w(^Suaas|Q=e8fUlqlE-@pFV=X@3I_Y0jK-DlkXSv0}ICuM=x;~(_UL^ZOTPa#d9 z#n;)%v(L-Y>iA#WGIZ>7PIXXs6lDoea?366_SlNNBHu-m=^2jitlimaOUO*j&}&=L zLf_@tx*UpF%=HzUs*H=gfKLDdH>XS~McWHmr6h@rFX1 zNi^IgnAS&xbMFs;eA$J%DF=BM5|g6&|7|W$D2oB*qu1aRX4|zbsiLO%!8KlV%bU++ zPei>i$fwvsYzkmzXdu4XR7VCP-0q5bXS#u6;lBgG%*_?BYy$7C7a$R9)}^{mxza?s z-PnIV83NZt|6L_i1;F_H_YhR)>B|?!+#fR|E-w&}z0M`b0~Ek4z_mWz6H-wI@kbAh ze9kTY=jlqHNjZQ-cLg+UgPWng>1tF7}+?d_F#%9Fc}?%OSm;+B>bKD=W4Z$vN}*JgrR}<~AhHr% z56e@7Nf;C|kagzb!@78J`*G){|SDjB3DLuoX=HA*r(l zfHMJ9D@$KZDBKmOWdc^JI4kxf(xSQpRAps}aDo*;Vzz}JAGwH<1ODMMfJC{P%btl) zR8UoYxX~X+xspC8|JEb^6WUXg-b>M>uzd<6jHi3J5pYkR>q}(`x)J$G~ zlwHJ45SG;H%$<_{r}-Sg9yOIbdZkPdXv;WS_NvW(%*>KZ#sVk(BHfUFmEPnPK^{w~ zRjE2&#aZPX;=>}7o!vcfR|?fas^xz4A7GfzY&JGx_iXMjWEVy^6EbX8@vP->(85}b zlNU~1J%hbsQ_vn_eI%=}nz#X@yrUdX-?mBU4=B4VWk2zA%so`S%Ho@*q;)gEIrr;F z$_E65BE0kT2=i%MqFL{k*VmVHgq03}2U!7k5JJ}MG%76$g3bpiycTQ%01JD3PA5Zh zb*OHDi755gUm};`;fH!krEehDzfPvawF zo*7aK)0q$0B%s*vvp2e#@aY6n!?AE_w~#wZHEInaBcMs38qhY0qkg6wjH>qZNOtEg zoksrgU{?euu&o61YX3?oLTRy9f@m)`wEz(3bf9UFp}(>^VP8E!w?ZEWz{{(s1H}0q z0QPJv#W+snSF@&OkDd{xPq~Fmr^*Xvc{#S$5R{padr(&E5c!%VLR6VXs$baGMVuHD&H3(KX@a?%sL_ zG1H~X2l@j5<{pXkfLF4K^a9X*c_0!%nAL$h;{yOD8Rt-aFb5uWCT?9|<+#&VQurA_ zX8;qwkIOsS!oiS&DG8PuNWJea9lj|B3S~_s6}AyLpMsKLYq!+<#RzFJ<|YSx=c}3=a^#1+!@&zB_Zs z5@*Ne!(o>T=pVSt1L~vd5?=%WHpNR82GFF$O#FU z|3eZxvnM3Y1Z(TctjHM?JNLI89C7d^nKk=aO6G$^^={0rD5%xyrM zts{#rZ*>OwSpo8Rjb4q82Z5v6?Y?sKb;%-CO#o^ z^k+kK_KjyMEY>_*cSA*La&mNSs@-}?j76%?(deBR(5Q|4}wW1<=eOP z>;Ry)yDQogLFcAq5{JmsS&74L6h5)3k3cJYK^GOqm^=%cV2$RrmQzg{y=~9E#}{yG zi|?<|%YefTQW2gqk#5wSsh{ME$@3;w!s2lt%j4YBLYQBDngLG`RP&&6^Guw(Z;j@Y~Ju1R3*UNw`s3 z!6Zniv|XCu2m@hjtOPIj50%0p#E2B1Du5o;2wU)h< znmABSSseNjge42;~|6M;zFwaK-T{TXjIB(1JjT$OSQM(Zxd$HJ{F z9?~8U1>V4o=YUpR+xIiwJs)rbIHwCQNWI}Q${mW1mTflS$ABj;4BCKS@9Y$I6d&#N zIR=(3^9Qy~hC_y1#!NWD6oYUinD?E7aakSuy?UWITWoiRam>wS*Ol@-eK}0znas4p z3$EYzeeXe-?sap)7hpR=p(b^IpzP`0R8{}UxI#4SRYh# zbb%sAxt6ZOJ69}I)j;V9dm+=HA~FY!y2qy&g}{G>XbDp+^7)beq7}Zd?SItQMNOpa z*B=$P)KUJ!z1tfaAcyyezX#Y#bl1DS1asbC2f}~`U>~|}&Kp(`kvg|cX!Hnw6;Zk} zKcuBpwZ?pQFiqAXI_At96dvO=? z>9bejYks}bt--)CteB_i6)!BbH$I)wYa@};(0BQ^7=(f#d-@`U{XaT<73F96zPnS} zAY?yyAIzB5_z~&m+LX;8-WC?F%HVY3ax!95Qc3fBGE*7^nURldg=$R3kc~;C_M<}fsiq<;rfE%#(n>x!vL~{2xcN(cyrNqDaO)40tiS;~YY07=5^eQF4J((zN^tiezZ}itthL>g&XeE0-_dBPkh}e1<>- z*qFZy&~uL;#iFnO>);Jd>g?<-mC?nB;fdV?yWzGVvJh?m#}s;3btQ6m-C?CzhWiZa zKKv_0wAibuDYDYC_~g9Gq7g$6t(yevOC+^f_gz$yc0yP2F%0OhNh)Hf35$kv(rEDE zuM~L_^wtacwH%mynrT~s42#eD!lM-x`+Le9Wueg`4az7`_|{m=HlTH&TyEinc#%Ai zx8}=lMQ${dS)Qjw55Y~fxYRRGM_t9;T2=->HZVf+YFK}c42;G-lCw_FF7eWfelGpGImWk-{F!eVjGIMV~=cJQ?fQ z=MUNp{Bv*0gkOJ0H6YaG$Jq7!#_a4FLAE8%@e@dp{61yCcPm%|Ri8w6c(5>UxH9-a%;$$qF&g9mSKs4sZQUS%t*%@1T>#U2< zyOpfmtM23DqA%@?fL~0I_BdNIJtsmUecQhtxr?K&Z4^`-!Zfcp6*&C+aF*-xfIrdy zKiRH20>9t#G;1ClfIBVa?Gm8KH6}qxV?C7f}+}+ES*czl!^|JQ*S|rj8VUg&{ z7$ND_Rcymj?jRU7)Q=&rGAw(0%Iq^(VvtC^h-ANDlnWO<)T!p(i{F1+VjS!4zJ595 zJ~}xq(bHoKg3(+`e{|-vb<`AuZtFBUQ1?55hH)epuz!FJl2Pz2v(_aLI4swQ1-Rn2 z2(yS+f5P%P^AJ6bO$9LnV@&E|VUxy>>~g#gGBfLYd&WFZ&|i|QU*9?VO$Now5|dQ8 zCI4)w*J@dk-bYnWE>uwRep~m-k}^)RJMT^$VC_jN8WwYr8Tu#R=Fd)Y zGRe`PNm{DEr)e_WX$axqk91NQy7VbwxV>Is$l~_8nJ><4_q(BDoVarYroxjE3o@?6^ulvbGImb^hPPv9RyMg zS{jF=k24}qybsY?zp-oOgf02{7eCtv6QPuJ4|f`p{1V19T~hH@#RK<qts21PB1-%tMc723A|+Ze)dx}kUb7#J#+&1o zpnF6w8;yG4f1PV$(rc8|imlR5G|xD6@7Z zK;$y6g=?fO<}Tq^xQe$Py^vU$DOco+auIbp90=)oQT*!VvLlb(r1$Pl(|7qVH}#v2 zswgQ2oeW6oG|94QPx(zKN)x|(B%D3q^uOFZXYR_P>ZR;)xj3qpY47R++gr1{<-QOZ z2~R%!+;7`yR{N3YAq4D~NzYd{6$PdZI)&CaKaa)>nl7K%Rb<$HuVzycWu%c#+c%|P zNTgKob01X$bv^G+P##d4$e4iD|Xi zstW-^Lut;wO2>J+UmmO%vX+@PWn^cu*%@y@zFi|DGVhpt&oK{iTW38p&r5eaE~W>W z+qF|vS_~r>WZXgDII$^D>Hgaw=d&e6^F@NBkEm1pfKHFo`2a*!Qjsv)esEUec0AiWHNCI&>>~{v4cGhW*rV*wvY^P{D z(Zww@rj8}|ksKPw-`Dr^JkRI(zMuE=d7kg*^GU)H zF^=v_m!scHU24Md#Y}&hxc@lv$?nAmFU@WoE=;3W zy6ZhG4L11Y%~5Wc&yC-@AH@U#0!#q`8COG&UDn@gQlJs|v87I<#@5i~C&C;Gj?IqQ zA{KOidnyB;O{Z7og*o^r3gs;#G6+hogzqseeK;D&3py>gtDarFF7ePBpVBzmCsBmH z(3%-*ISM3B>%oJ);uH`6yH^w3U3xPo4LUQDG{X-`Q(u!`1OVgky!)PM^m!x%COw5i z4ISNzcynE;8yTGu407H~X^cD&l1~4eLj1)CeQ1k8P(zvrt(~{nTX^VGdrzd~cls|s z8$aggw$N|S1D5iTjU6KpaMTi@8*TEA=JOsrh3dh<-P zx}1Mvx(i7hRWfQVT-Rj+~WLuUA#+#bV$WHg+;XZ(`X?`Vad)_w z7{9&^!m@){N)ME&pTPzuZ~ftY{S5Y#_!Cz0;ZSOjEQMC(JM7T+^4KgnqhtSrenj8j zly(h(aXgS-R89zJI069O%nwWVN)JkeUwX%A78wYZjk@1K2RT_n{UT&{rH*UjY$Gq; zA{)5ZSs!le6SOK5-{9m%$)_(uWPW1rCdci69mR@P)uIyIyV`pO!i(3@=&3F=B*PSZ znbxW_Z8?7BHuF;jCZ4?lu5gT5e&eO6?AnuY*-XU@9y-SZPW#P^lWd%cyQ zdgp9Ih^7iqJhKb)d%F7mm>?bl0nH6j!65>blWaDx8MZ-eyPg6?vQq#?2Pv# zcJZM@T3bAY36KU$YwkoHc}x@MFPT!G_eccaYD9>y<#>wJBew9FSb@EdSQQX~;IlGdLCZ=yWxz7-}4$uz24gOU16Jm1; z=KxTVnJYi~rO!VUwFZXyBdp7z7q4{OpO)PoHX$g|Nnv~t-vE*>w0k&fCV605<)rx^ zcQ7Lsh;Rb=L~QeVW`2(zuXJ&1iVe5+^0C8GmmsRZoK<77v%GLDB1Mw`2hTQr^Z zJK*qdeQR9e6a6^~He%b%E$xu#_yjO?chg#0x60YGN=|md=%{v@y{*j2>N-#atu=f! zi^c~J|5KdJRi5$SYX#oG6-`W2^BUIn&ZX0*GNr(R$Wr$B>-LPOBUj$GRA}hdF1e$S zAs{o?mH=#^pQ!GOkzxEr^r6E1FhVzh#3ztGwN$t&->M&sHK;x;zV3u|`u1&|mY~m@ z6Q*tGV9<-#lXIEmyLU38a%9hw6A-SK8y2s+)mKr#;TsIPT2f4!H>aruM*s})BPdZ$ znOO(}Kbr&PP#YH46$*&%YWMBBh1U6vZoz7(<~UNhw%5P<+nd6W?m+!OEP4j59b)I& zahtx2Re52hWWsn+-VT@NPh^E>ruT7;OGwX@fv9NI)2#(u8-EGkJ<9M;eA@5J2jyt+ zhX4z(Me$44awA$D)mv8%)L;9f``?c+{7>$H$TJ&%`5uM87~=mp2E6CFkGEdVte;cg z$Vjmb0bfXOa2H)+FbRZf8J{Q?Ik|RLsU{Q6Ir+Y4278AJvwLFXV(1-LkEJj)Wkaup z^;`wL;jQYlu)-6I748O1wt6n3*g0@Hpo^Ff=dA`}h2Kv}tQc{|rE|69Wj+AwCw~Qw zwFZ0bE}Lq+u=1Nleda^Y!19rq#+Ihf0yQQ25>2Do7smIbldLOaz)r#h z{&)@tMePJ7W7C45%HeCGk1Ow(%>3LdU$Yhu-->Y5PT^?Ktjyj|Lm@_<$^yuu&agl^fz zv%>H28*SnI;Ca@<(Fe0CVu@Io&M;EZR^r$Er>89ydeGX60vn^N3C+Iu^bVdDVIkkH zNglNHUWSdRW9rQadCaGD`sXjm5GxZx`&~WSh0NzmU&G!Dv$?I#!6k@$D(^u0rej(f zmiOrC@VJ))!ua;QFV2bG+s#8ViorTLRYk7pXN7tHIECyv=Lt6SnD*LI4mL9zXELw& zTUir#&rtB}x-Lg(Eqk8L)qWUb73mi>HDNpgZ3R9&_L7UWEDU34_Yvf^K` zuW2ZI7le8LXBN}WB6>_#({hN&y`Wn=Kp|(q!3ip!X74Q3e(+28$>(#ODo5FXZ?iv9 zoTgDz+L^Q4W*m+&YA*kk*e4QJKp-`y)q(ggCjtMSfYjK(Jeg?p{e!P6JP1S9wCO?_ zj|?i!C_tJz{Oj2o*X_&_tuS1=5_;QxqqSA1u|oZ`cB{?`%0Wb}b{mO5?a@&?w}^`x z3ZeAZ?8^vNb5?80+1aQb>5gYJ{okWoRs@9B8WR6d?P7e&?f!d-Ska)sH{-HX(rY14 zY%0)Y#C2lgZLB*3Okq1zkZ04Hd_}gXkakD4g$q%|%B3}GDJE*jLTQN+eBL&taqo>= z{vl=eZ}k=oZ2F)CT#yMY73^sCo(EhR@+kj?S;M0HKIwIdytU>s^<*ZHGEyhs2t=(+ z5`W1rk(w}vEEuM;xVFGnvZO47<#u3;gF}}ve19yqgQcG4(eWY{*tvMtD~EA&tekEI z2I}inu*)0}8N^!pPOb1lgM>Sr&`d;S(DQ%Y(lc$h4^WhfbZl0wYrUo2_jpEqe%P@{YEz%Vju*9KMcP~^6|tsJxrd@{{GDj`s2Rkqh-fdJP9jmsE`y) zJfv4EH-NVTQClp2&tBc=_s6+SpWake?$L%P=Q;TlJx#J7wc;rc?yqmkKd02Y!POhP z!|Jmk=9NX(wG?zqj)0LevK}PLM6CVJ;?CLcZm3f##+J^jR}fPBz}OKs8$eKx~7rGBxlq8_QUv|b?>j<{2XQY zraaim3N~a2(L1l@WwB~E55}0m=mw3{S=xRC`JTuHNy8uZ)3o=hGK< z&S>B!M0Fr5)pgrO&nUT=V1WqJlb{*13coP9Pq?Gnh_ijYMpz;F9iG*mBdeuT)qF8KEJZgmHp zE~Z+66TH@1KmBR2H*6(8={bvQxfcVe(GWjh6Frq#*RDY!9w0}rJ?EM(evTrHUBRI+ z-X41Rlfa@~rBEdnjI;>sn94T}tI`g?As3s#v2?lWdf%G8Bg2Idj7&ef6Rsfwp*ehusZRkWr>>7-{_^Dr9hCD-O|D z|57aE0e!nI6(<$y4ZCKJ<9=?<8~@Wz@DP|Wvs{1q0`S*T3piM^ek_4~uiz5*oL#Vv z&X^0p9arGkvinmzU6&)kx&E3Tm+MZe5Zh#g3zn!**vVhQZyfz?TecIH+Q;D z{X%WszG}Zx-jQ&dnFE-5ZG{Sn&wIe>Hy$w;Y*J1mP*It8igt=x-@`Q_EZnrMhE<3pX8bSIP)PA4UMQ89a4J`XTNmoyDCd z>x%zjv;&U6)n^Y|3*^x)qqv;=*_QYc82*B!PdhjYG88x%{9w)*A0jdqbZ4*+2Dzoi zpIo}86&Sy-jB8y&2qkwpzd92EYIIw=AJ+`v(b6O=06nEWfKe3Osq6gJr=A?%(d5IU z#@QOIuXoJq;WRNi*_QDCQ5J|5S-zztyWsyZ?kZk4uklT!UMpXUn2N$>nvTMM^vYNh zC|q3nP7}zZ%}X_t(Mq`Hd_f6{6{tc2R$qq#^goZpd7(m@KZ7JR(WzXf-IQ+GcKU?7 z9zOAN01k?ok6n6A;>S}tk7$FXNYy;1A`eA1E_sJ(3gGRO#dT3fOW40BK4E}RAx&xqam^M^tDi@IPt2tEx5Ax-0m1*jMWp2D; z>0V$wTBI-V!{Ma?KvMV6*1P`@+u8vM$9=zQRx6hQMC&0 z{L{Mj4;F|*yILok6utH9T#mq{aVp`1-ensB8D&XFV^L+U1YmJLt|oEA*-YRt)`K z#5e)BN(L9@iOE-=b-qKe82wZu&_sRWDWKJA^uX2`rme;g0rHM4&by?erEzORh4Y%Q z;=n77q|JeW_X8$31IvNi90UySh+7ub<@$HVZC}_fUd!`HoT;<><)C`plqBF_O8dQC1IaLP79t^y%sB)$1)5>YY=f!`cS^>=uXU{0xmb!G2 zBGssR=#t5z&` z+isKp_9^>BN@jm1qYej)^wo&w?*MZ$W&_Y^PXoGxssVo?J&RSKG>kyzlQ~+b!7fjx z7~WLB`FoO+_ED*N+xV|Qtbv9+m@zQSAF>#F3AXlrq;x=d8(y(U# z;j$Vksu?CNy`9h2gJ@d6NDa5ftu2lEnmB!WS$2fG7H#Rv(@wbCW8xIdqd)nL{*&N0 zt&Q82A#e@I}^b;5um#=G0kq)d}3M F{|84jt4sg@ literal 0 HcmV?d00001 diff --git a/docs/source/_static/nlds.pdf b/docs/source/_static/nlds.pdf new file mode 100644 index 0000000000000000000000000000000000000000..4d58c3870867e20934d73aaeb187acd0cfabc275 GIT binary patch literal 3067 zcma);2{_c-8^?`hFve~qT>UcX+M0!tQA3uRkR_z-*I;BBhMCc|XDy7SaV?3aQMQtn zOM|jyze4nnh_YXGg?f^;WEoe&6$+=lQ%U#}DK8p%FR| zm3#BIHXs-P1&}@bAUGU=v>*naruhIm9Fr{s001Q3mqa5{I4y}lBOWGtlD&u!0|N+^ zMj;XcAfW}?Cr~`bt|kD%$A@*AK=AerEWA+g$X+O6JsAvn-3D2Q{SHB_xx^X@@k}a1 zgdxO0ukC+1J+4=-D{jkihSB3ra|hbrByEZ~5=uL2N8<DG9_TsVr)zf1)Mosp#Y4I4uD3Z0oTn5TOySlLh&S00liH}95LSx zGT_v)^qKw8(G8jVF*?d>Hc!x$gmF-wBMq53y2})%vQb+a{}QuCO1|O&`?T_IYPM13 z)5jInxs8?y!uzA zn6Naqebu8?x$xt;J#MVu1AcI(?(CwZE0_Cc&dD*Prs(bJw^}u_OIBQ_q>?``;@@LD zuAKls^Ul$O*R(f8iKM+V?-0;m>nuLS|6pEyxW`9Qv8>TTQ9mfGc&7azK|C={r-bYqB?5y zv9yq6KY1Chu~e6_Q)!UzbQP&nP0{`&(-ziuGz8~nqQIUqkc)Y>{(>nFf6tKVN$QJU z!Fdj!Z_sZP6Ih;9Qx?8{w&c-!1EHI6w-!nJJ8^v~?f$&tdt>ZfWwh1jl@3X_SYZ{M znj%E0BI6FA(TJ+lsT&Pb1rY^f=pOyYxh6k5_jy5)BPR!TUf|hSZBw@5vrzoVGS1pn zoi?e}@@$@m9&$%@43S~%n`==;ejnm`Hd3vCLEoqh%AG|NFfL5n)TE?BQ7;l(P!&`` zWL-dWz25M~nV|-@a%R(SI|Lod8HLY-!%k8P=zcMcFBv1R3S}0j7jR0om*ePWEWRn5 z+$ENGT;3n>h--mEo!*zi#W5{Rqv3kxmNvrl1&A15$iVA|rDd9<6UmNPB0{Y2eCmxD z*<^Wk51V9C2YLuEiXAuca5s!e|8c1d@urZj`T%puSyAqMb|^~|?SAvs7lz?j4F)H| zo3)Q{rZxQuZ5JxmC%1QYMAi&?KkW4V0BkMnjjp*!^Kw~7E(GCwysOQ6vhX9 zxM8HI&RZs;osf}Pv~MtPpTiCQ;k((=jYCe*Wp?u*FsFeMVqMh7Re~{hS5f)m)7^<6 zwFj<$SPL+%JKXfxQ%2Mw4JCP%6x zZ;MI5`G&%I(isO8QH=o?j5om9761b?Zlz`y`Vf+!K}Jk<2r z0VFNNr=l+_KljT=t$-r3O}$d<4zmHlMNs)5>!G`@mZggY7%q=#TD~uAq4uI9zJJN% zN=n0Pp^#Ia)#@^sK`8b(SV0@hr~{Xgv<=zp%sEiks8~Xz5}JF+e=%|Bp5Q*2$i#B=j1%+mSzFD+Is5? z7h^*N8h3d4FQr`TnptyTKCe00+VQ-@JkT)H3M0kad7IsJGpqcq-fs)R@bXXQ^hwW! z$z$N6qiX6#VoDZi)8@bZAM zAlL644tM)+hx7Gl&|3dS*`Fnq)i*U`8ilzA@io&ZZ|o}7$YJQ<98GMMuD zYB9W>{b*)>XL*8d+5CO+9|hRNJ-ZdHDz7gJ8Xfp_uj;k@1WfZ1tQ1!)+_Tf#oY{yd zy&g?%;_H=vT7O3!s&qAV-0||gp^N11qrt_F=3O{XN8wZ(TbT@^TX){Uj9=qv?CL?N z{t0_kA^~;sy z*z>WV!zi?triZ(`*SMf$J%APi=nPTQAQGSCUceY^8srowbT(-}UgBYQ=LbI3cG0II fJorek|KFpf(g+kq69@*yZOI%xCl)tQ!4HB^vxKjUFfV9U>paw6Z zrzn3Y%s~JQ<{WnTbfk;4bxP9qoGn2Vsx-Krllo8h0m1T_-r zQ*%w!5u6y|@BVcBM`ibtY(LBpMwOM$nf}}E3eh^UF0o|Fc4A9<5@Jt%6iLD7BtDvb zci%yb)!|U>Se!c)iz-lnY=Ly`)H&MfiMN%!j8N&YNg^?v33a%3c?dkFHUHDblvYdS zP@AdEgpw1srJQ!Wi}>)NVe#Rr_?YIW#@Z*sUw4xXk+hrCN>pYw-}nQ(`_5n?@oR;; zMYFHZ$mY&h$g(M0it0;c5=zt^i^83|FU47Ss<5r7$6suhz4AD9^V@P5yO!FIy3+|d zDnKdVc|m>)Cf6QK1`k3fWY~6Waa4l3Yfe+YF#-X?SfBmajSlf;FbygxpkGa%N*4wt9qHmjAnK z+4*EdU+vZ>pc$dh%O0w!mFn7 zS3+NPlTvZF#g%lQM)TzpU7Oxssl~g4^ra!?NZR=%Y~y`nS}tT+o(-<5HMsWkKDN$$ z+~c=5l|<`OfkAqoP#Ow$1Z+!AME+lVC z%Eo|Qv-0@YgAtGW==8r2pUHSHr{ddmpb;tOOm?nCB54eW){DfyK~?A6S`y94vBzdG zY!dCg6hGRA4u>~MeG^&N3XMYsUg*(m;o|Yi(23BhPFRspjcXgp`qWq~)B35*MBcAr z$nr~?J{-Q~SIFP<0Dshk7G0LwkZDY7!|O@6r3N@ne@L>Jec^N#O(+rOmUTP{%+)ld zRg16C>4ASD1a}WYR7`1?$7_ZQAdqXe-;1XT9oDR0XayLr2rAAfREe#-=JIK*uJ466 zIk_CtlW9Xh$Lu|c*S)?@?h!iY}&qK3>4h7Z-_ho##Dhs(p%ck06X zq!LNh6>0UDJA0WXm*H@1-ZHfkaAm_L z*e~HaOZtd(dg*C6wkZdv3Vqd7E_oT*CD2w7FiJRlni2+HQ9?m(fEx6+F*X9W7F^eZ=R|9^WM$^|^kHMe12wQnO(uE}WPd zV!R16M%I`xqwaXeU8)VCfM5uhQrdeN>iPZa*2usUGoMM80KYO;CtvN3;iXNPiM7IA zjl1CpPj^pF$sw09WR*~$Dk?Hh)*|QwIKY!n-us)yu6+=L?xf0Ybo8Csx zYM=ALXIuTu3cfkuif;fw?r!5#?8l_{d7yF7STRou-~^0)9Dl(+H-qilkOsN>!%IS* zSWozFL*tOPD%h}Aw%*iSYS4)@wOPtX)^N8-*XQ4+#9KZMc=1Q@PuA)hAxRzIe-_=G z4jv(}ntN2C`!iQLBz<_(TU4~jy6cW2RI=WwfVOl}< zuvkoiDGerDw^6Whmr-gD88d-Vg+{ppDS`LXw z732KOj?SM+qh7#$PdqG=+xoagpkKhiwj)DwQ@`F~25R!-4!hMMd7(}qID0krD7W{8Twx3m3{+=2;;~(i?*jBhvNvg6EG7|y;WuXNLm3iY2s4FtZTQ!uG_W2N zamZYr##1qQCNm*C`8ie>CT_;u!gIi;wv4ZrVeDjaY)M!9zJO9+5)6Y#kIfQ7Hrx zdJRO3>^Lf8n5m7W(`#~Iec53e%-;66N^6If-nZ0pbEzxC<(1r*&AnL;;k74c%ure8 zQV!-n`<#(A%rDK##YSWEnu*Mc+|WL{PzmWPp})L9DFW6`HlYuwD0$sw7-t`Zke|A5 zbzwkgzM;%8hRYO-llm}+ECHP;7W4RL%f<pU42@&NM>y#>Zm~XppS=<~`Co3#mcLBYiA$YCEI>BO#TyDjll$FW9m8pX*#S?-19k zO~Iyn`s>HPq6gN&c0j`!v#@LaDy-WdH$8Tyw_|6{CywB(U~!IiQ?JbYe9odje(4D| zZu%)>uy^~(%eeu6iEqm=T{u=AwZ?w|gjRL?5a6JC`Yt>4!+7h8Ax^#fn|25QIPrSS z;5>Bf8`3P!BBi@hsM+`2DQJWlPzvTAZaPdRC#Lf$hX~%bj0<^9ISl#8|L%~+&$H}q zLdi)t(>Y&V^d;@OAIeq&s7-#Rh)eEGY9xYZun(c1kNafw-gXWy{m_U0zt=zXub%f{ zK;nrvU;5~xZFiK{t>c~z94|5n*Q`*h22++>h+p;1HWijM9o809f7Hi+FPDW_fRhia z{TPSE)p^EGXWyVa?zldeX?2h3zJ{ZETe&3&ET-VsQ~;udr;#iY@Ae{!4SXD{?DKbi zgV@GNjRToqo)f)_ZL2o;^t7Nl{!m*6zr#w;nPN%E+;P7GV)z5nqMg+_%^?N(d1 zPe$jYi(cH-|t=wV+EbMpgY~FGmq1Pl+_G~}C_{Rj&;EEf4 zi=;VGv7$&rk@;i((YX13?2C=6koora`V=2`_Cq0yb?H!>yR6c|%8s5bvU+ zMFz~~tG|9LaO%@v3TtRel&uPBh!{9^#DtMUo#8R#XGQSwKg$vrjMCQr)9NR&`b23*XD;<=*@mr{dvs1wpJhN!QDZ%TV;K>K}%;H9yzn7DyFKZ`jvWvo+${`;s z4F*;XqTU27QLs+OJHvRielzgC^Ovo2A(;ikATIsM_>23LGzA#8y7gSgq(sP#`Kfst zbL9UE3qo)-`%r5Dvg^djYL(d^Rhl*ryHJdIS{LpUH%~sfYoA`VDJvScd9M%re>qnO zF16xul2EFGsm;r0iq4(s=zRB(86wdnWFqB6xj!!piQz6q|0^X4CmI#DarNaSAS22= z!(Kj0adpR18e409c6+3y-2*5fiX;TfSuX&Yb?jM>kYFji)T_#~=Sj$XQlm%2y#VD( ziy_PY+bhh(x~_RWpL6U^v9waXeQo7WumgPoR2HwFylTTRqK;=x452>E6Aed4RTm+El3y-IIPfLamKwuP4#ebkSlnvmc}!Zd>=?n0Q4fDKCisXDPut`VvFk%VSv( z*UYXz%JU7W#^Ze;<&O{lVMRv`6@$EO;ON+lkxH)^4S^ zdjHBH8+-~|$@wvlcX^f~YOyI9dh`F{R+!?q`|AuG`%FcmDHm959li12 zY9H z;v@V=SaBh*pVEPgHywvz9vO3=)Jzjr9#qPXyvXY)45l;|bS)i?Q?EwY}u|j>JdS|r4}0FRRSo?SH){JHcfA#Tg<9USPG+U}W}f02f2Z#ZdW!ooP1J~SmwK$X0gyCsFVF^+`P9YsS<)LLKFE2@<;hFs z@=kCB?*TkMoWR3(L1=yK%Tv$E*Br*8R$ zginlLK|Kwi)~uablQvjDpn;}CvR-p3%xm5PN9LU;e7`DcVI`NMyh@q}KZ-Xgtdx)y zGFJ_$rUdV`hxi>{YztcAy@p;B^nI9{RU0yRKxhd4{FG9;nCF0ZuBE+h8=F|bySzej z+Sb8PwAYvbdmF#>V=sKJ3%|$n8STBWcvDR`;2JO^@lI#Tve%T>C@$AP?|8*p`&Tpd zL&}{jiKdz)s28+8z-!Wn9IJhi!KC&A+iOhI;YGij`+Ty?P-WV6E6^&qXJV@Q(mVcE z*c9Ndivh2+u|{KETjrBe-xiW_%Kk$2Dod6;4?|!80>`pDlJlsyA(*1Ja!yyUjn(IT z7iPtmW}fyP0p(ispM}pQ)h7B%tn6ycET-HUIs<9gx70yO?Hjr3dx5E z3gG>)HIuSye5EKrDP>b5L8>snvHT)Gb-QtD9BXnI`?Akj29*c)Q+^kJ3fjW6=$iD` z7)L)aOU3VM=e&@Q^`0p4Hu=$5-z@=7!aD3HZ_d&V9P^s4RIXJ})X#BqqffMm)fo|1 z>T=G?j>eV|&OQp*roEsIH!lHdgl+YH|9y4e(#oVhHWMEuzp>I?CmsE|&-n~05>#rL z_FW0ewE|EJ$#@s};hu}Dj)G@u5osz32e$|AkA~h zs3#}M(cPY7<%Q(SFofAIre|+2Ac40X6C}1}>V0tQsiHntFU*^Ut0Vxp>($!x5zyl7Y1 z#Xn)}k=yZ~`kY10It16;;;|{)xGJOIoYnE9#@Y=0aMG=D?*zq}ALk5zPgNs; z7Vvo(1cR|k!Sg37iR9XP^`sv@1R)Dkgw^aS?!l+ImvqtU$Nrah4l`Hj^k{Wx#N+ed zT|Vqk*F>kTHAX+!8AvP>%?ZsR6;5;a0vAL5&n#s!!t;q7J7kyG`reJ#4#^k3H+2h{ zc29&WpKkCxqVjN+ccIcuY?ZPdys1AG{~c#>0zQC!Zwe@_#hine=2 zud#x9rQp{Og^Df}0Ea4#tgtIH(3&HVW%H_T_i?b(DYyO%Zy4fikLIlvrWdETC=Rsf zc)Dq*N|*U>uO7tZY3Mnl=isC2?)Nwz$Qp47nlD~B@LEU7^>}NJ-BxV-M!-2#B;V2h znN|cep7?d;rGJUdB6;xge@gy#09GW_s}HXyu`U{V$k#NF?k^Hr4^w|H)wwYXLZQpQ znccuRIUv=Eb#oG1qhDuALzmDSz*LjMk_0 zn%9(%9q3YUX%0Jv(TRdiyrFtHX#VQ12|c&+_fUNgtWL_`pMal)fk-^$U2^I6dl8@4WAs#NM zLZ-GjQSA57#49u|(!4wXw`kJtmoz4g12nD zshgSz)#~7RG>RiW@z%j=SMtSl*!&GE@R-KT%cPCIV9I$D#tZzgY+GhT@#^U&$p?6l z6L<*Zafulibdke9DkEqyS0DgbVu%qdDKT1fPGAv&hx+6!hHK)Y@K1p$1S39Qv5E**&dZS$_l~GFJm_(VFO2D;0aT7oYvD2T=5wFF`KRKe zy22pxw+cxM=28vyP%rnY!}%W9N`e>e_O42K`k~TuHDE@_8qnyVCrw1mqw^>ov}?>r zc*MC3_-am=9Zc^jN3=TDOUb1F(~^DUYaw;oB$fSdocKp5B4d@sNP>xM%c=UFsxKsc zjJc_Gz;&TM`q}ugvboc63$CeWCm7pRQ)F5xvr}@`0}!wkm^)u=Q!vMD|6X>^v$n=)Iw#%m5zh5dkn!X8DCZw$-$J(L{cSnj7kS$ zM%4B^9tx``la2s0Sc|iTsl0Gc=n7K>I0DAj@gNXfP<=Pa9#$EqHMI8U*|+iV z%UQR7j>dBGH2GNB8bDT>YVhggh5>8{R4ft>RGeX12?GREB4Dm`Z~xvoh#B&=0VkV( zb7v_159UZzDrpCX$bkTwIAGRsbo~ZTJ0<^1Tm+K|s~pA~m+?Yy?B1v#&39&Mdvwup zW(+}U5-%3ANu3o=N5@oVLU;?UNwBpE23L+=wfHqvoe(G8*2)vtm1V`HOI6HdRdXrt zxJ4GVg%FW>w$cFtIG#;X!R=I}(F=&peLtyMR8?59dhfGaHK-nNMD~lIxA!+UZmuxj z&8iIdf!bNKhDzw z>vtsAMV8{ow_IN<2vbChPR?kSr4nTUrnlja(BIr^dMOK7TbP@L<{_|aIg+ykIO{_0!}8vXpD z2JlqG23HM3IJYUxp|fyka2`01Rlaedua5=FT)lymyT}npd-W^dHImM^i){Aab*5@6 zBWzzl=bW1o4eb%EI-$BhjE^azANp8{nZU@l2pn4zrj8jhhvP2bE>!HF9o;jk=5PMF zs^+LwdBE&HGmcy5-Zo7DJ%v3jFFFsZL3bR4t4@VbVZdaQg>o3*^H0s+r1w<~ zna=m8Ji0_IVRZkf(#*zx&4+(ZVL_+@Nl1+S4zd$D6HO1a#DHz=gQ83K>tE8I{@t+2 zdWSQu2>!XExCC0#)pgMk;k(auwJJORc~k#(LLI_#NA6iP1)+&o(ESoxwcP0zDEX}$I%U+R|-Ll#%|rwNU)&b~Bax%k)MDc Date: Thu, 7 Mar 2024 15:25:31 +0000 Subject: [PATCH 04/21] Change to custom colour palette --- docs/source/_templates/layout.html | 30 ++++++++++++++++++++++++ docs/source/coverage/coverage-report.rst | 6 +++++ docs/source/coverage/htmlcov/index.rst | 2 ++ 3 files changed, 38 insertions(+) create mode 100644 docs/source/_templates/layout.html create mode 100644 docs/source/coverage/coverage-report.rst create mode 100644 docs/source/coverage/htmlcov/index.rst diff --git a/docs/source/_templates/layout.html b/docs/source/_templates/layout.html new file mode 100644 index 00000000..99ed954d --- /dev/null +++ b/docs/source/_templates/layout.html @@ -0,0 +1,30 @@ +{% extends "!layout.html" %} + {% block footer %} {{ super() }} + + +{% endblock %} \ No newline at end of file diff --git a/docs/source/coverage/coverage-report.rst b/docs/source/coverage/coverage-report.rst new file mode 100644 index 00000000..b8c36006 --- /dev/null +++ b/docs/source/coverage/coverage-report.rst @@ -0,0 +1,6 @@ +Test Coverage Report +==================== + +Link to the most `recent coverage report `_. + +Internal link: :doc:`htmlcov/index` \ No newline at end of file diff --git a/docs/source/coverage/htmlcov/index.rst b/docs/source/coverage/htmlcov/index.rst new file mode 100644 index 00000000..abe3ed1d --- /dev/null +++ b/docs/source/coverage/htmlcov/index.rst @@ -0,0 +1,2 @@ +htmlcov +======= \ No newline at end of file From db0ea9d72d4b32e97e7f42bc20c216ef6721fd2b Mon Sep 17 00:00:00 2001 From: Jack Leland Date: Thu, 7 Mar 2024 15:26:40 +0000 Subject: [PATCH 05/21] Reorganise images into separate directory --- docs/source/_images/esiwace2.png | Bin 0 -> 7041 bytes .../{_static => _images}/icon-black.png | Bin docs/source/{_static => _images}/icon.png | Bin docs/source/{_static => _images}/logo.png | Bin .../source/{_static => _images}/nlds-logo.png | Bin docs/source/{_static => _images}/nlds.pdf | Bin docs/source/{_static => _images}/nlds.png | Bin .../status}/all_off.png | Bin .../status}/failed.png | Bin .../status}/part_failed.png | Bin .../status}/short_table.png | Bin .../status}/success.png | Bin docs/source/conf.py | 6 ++-- docs/source/coverage/coverage-report.rst | 8 +++-- docs/source/index.rst | 31 ++++++++++++++---- docs/source/system-status.rst | 10 +++--- 16 files changed, 39 insertions(+), 16 deletions(-) create mode 100644 docs/source/_images/esiwace2.png rename docs/source/{_static => _images}/icon-black.png (100%) rename docs/source/{_static => _images}/icon.png (100%) rename docs/source/{_static => _images}/logo.png (100%) rename docs/source/{_static => _images}/nlds-logo.png (100%) rename docs/source/{_static => _images}/nlds.pdf (100%) rename docs/source/{_static => _images}/nlds.png (100%) rename docs/source/{status_images => _images/status}/all_off.png (100%) rename docs/source/{status_images => _images/status}/failed.png (100%) rename docs/source/{status_images => _images/status}/part_failed.png (100%) rename docs/source/{status_images => _images/status}/short_table.png (100%) rename docs/source/{status_images => _images/status}/success.png (100%) diff --git a/docs/source/_images/esiwace2.png b/docs/source/_images/esiwace2.png new file mode 100644 index 0000000000000000000000000000000000000000..ba3c7db797b8816e7751f3c4a046648f4ba8abdf GIT binary patch literal 7041 zcmV-{8-C=8P)XlmMQivCB!K!y?7kdRSOnE~|Jxta4PaXZ`Qp{`l?yV3Lr% zE#&RQ`qZcQtU&dlBKNp#>haV0%$N4ET>R(F+UBkM;J*9Ji~v`T`owzl_viSrO|YXp zS+8Gsymy(ROmn7R{OZ$IyrG`Yr@o#yn9|E|ty$oi6KcV*+3)Cxr&FJ)LcQ3Ot`%Dj>pi-F%MS&02#$eL_t(|ob6qUf84f_CdK#i?Uk0KBucSYTGN!1IB_0MlSodq ziQe*~mM?ebHfig-_L|Cxow!Nw`o|vzAPMlb%jMdZeaG)-YqcZ*g7`3lnZc0axDFjy zxR1}z&!0Shj($EiF1j-v+QU5m_1Wg);^OM+>gML=)vH&pUAyx4U!Fg{=zevW8OHh3 z)up9N36;PlW|QFOj?y;?#`%wzFB2{kHm^PV=lNsZH4z>^K7antpWpqiWAx2{|8xm) zd1;Zbxw`tpujgjD)Z>4A_m7Ih6x}-eI^g`LuP!f9Hh*~H))qA;3Fx?^^Ws6sOxRpp zeX6%@$QZxKZgpq_{l~8^XKXI6{yK+%?RIotG@cSNF`J8v^N`7nJ322I!4kvf(zA(X z&NLX0c3)JtI<&&m-(JqxytH~gUt^E%bi#R&Fpvq9vAOhYPOEhH&Yc(2tq!f=UuN0- zbkYjUo zE{hhhyT9KF=f&X#vpL@1-|x)Y3qzjGxiu}f-wEdh;+$b~@sdAxd-qOGPP*XxMM3|H z^-$)vX}LQmC-)z{fPQso17nG0^Xc5}?cTqC|D;`2)3=8*3p=P=$4x!R7nO&$TT}@{ zD~c=`eluXxy}B*atUXt(W$K0g4eL7yH&ZC9*QhKrD7LD%H(!2a*u3;Z>t<4n5t|R% z*g<+f6eOVlNs1%8Skf3qv8a{HYJ8~X7JgL{MX8S)*F;g&se%ptiojpbn^Rh7(2OBOB4(CI@hsX~m> z)<)${Da;g5wCbxSV;{3}piU~QDbq`)+{-hM<4^4V=KQL5AF+8fyQ+1Q{_|m^Fo;54 z!}lgoj#NDUsHCOj5si3!N*gj|Q)P>00cuF-m97j@i>H!MQcNa@qeOIGppg#7R^gg4 zQ7U3g>&a=^QFZ@%g_436k8-zdp;ZUpwl*uDPZ%~=mx38mjFinYj;V>}x& zl1$8zq?F4VHZGbi;V!0E=#@*FwneC)ZSgZ0O}o1IY^J&uxOaMVblSqI#dBXOiw=fi z!Tc8!J&A{)_AasN^s3K=Fm%P~J_O1v9j+f2D_~ZVdE@bwvQ2;|V|G;qvs*pia8WT; zw+gG8j6kpc`a?;h3*pc?nD`b15xQtt{$n(X<*In6vyc~vM}*h@LZeFQn@^nPrJH0r&&9Gh;Y!t zM@gKB($d8M&wibjtyi{_fXyB)r&8loOD0N}K{!cg79cZp_AAXQTGWvwhv?-#b))M0 zJHxF$W!aoF(~Gcq+N$mpi6hoU;xStLN0h>ZD6KO>Hwu z9=pDDp;Ge1BQLN%PAu8O_H<4|V>NGFN(Wd}f^W}XZ7OH$vF*xJ;=Pv;PiCru_kNqALM#FrtR<5*owI_rXgGsC>huz7W}_0qY!fZ05pBRObx zI_uzrBT2)BWjfhNA5hD);ncPPU1oPx!PW3yR z14rmHcd<+-6X`1kFPLynF&t=5i0lI*v9aID$(ipo)$g1I&n_`+UcK6~m3s-BbEUk` z66uM&IG*HMW06beW(A#C@bI2E8glG~n`82XmG9`XA8&+91 zuU%=^$UY%+dA`IwY)1+cOlvhlAa*7WVNYnw5*d@Uf=(PSz?&yn1X<5Ab3Mc2IAb~s zF$ESZo$BOBtKns*r^Rdyxa)>%kg>$Fd9_^-;ba-Jxi;^ke;z+s?Zn6@(5Qx*MCWK4 zovG1UmN;-dVco^%Z!Q{x%TCyOWa-4B&DUBIKAjmsZeb{cq^s3mmBo2VCw}$QwO^Xm z%Hu)8=G+{TprIL9O`#`gduL8)3Pn6k+Foun0B zvMQ2BTm{dzI9E~#E1i`fF7M~n;~6WPSFSy8R*PacF=<~2o5{vJFKAxW14BN6EUQYG z_|Z>#AbK8VLYW zBe{<`6Un_W9ZYW}3s)v3c}WWuRi^iYZYARB2!FCVt%DW)@nNa^jf8LNlwH2#Zfc`zI4?jY>SKM=;R`)Tv|%I zmrJxwOXhByiC0eA)6apGO@tYh?a5l=p-h#{C1shZ?W> zR*6-X&9~l~fzCS%DH=o%#&>7xRfl#kS95H>^>gEz>9W}@PJ7nT`6^&+=Gc7e)20WZ>Cjvkdise7RuR&4TF3g=EE=7SlN90H}6kgJa-Q>HWy|{J-B!nn2dCz@H``vu=&nA^>rB0Y;4kEoZVP&STYDu*K`E< zfcG(xZ!Y@4!7~$op&*a?xXeL=lyIR6cwIgOdP$2MdFxX069EkZ`fuPiOxwf_(Kj=h zv=j7t0r`%Pxy*HvYPwJQh`K2}(b%k$#OOwb zjx(Tvag%oju-%V7vaKwrEx?eYvALyW}&_yI>QmC12trPO?ac2saqcEPhW#8_I1bt8fq#_XBe9n5BU(i zwBJ3^$;3MviV4oUywQGbvVP`dK{l4tY1Pd=$i*ccyC(QFzC z%5Z%BdNadzgzwmQ@J3i5@jxfM703Xgb9)X9D649<9%q7hdd%LT?_l+2NGcpv@Yb|} zc`VS0>lmWs1Dyf6=7`Pm#dJzY%0+r4J#I^aITA{A`Wnu%j^L_5r>r4bMS~kkF+-<| z`VkBQL|CQ?Y)9ujR9NVxokf?xd-Wk?4rX)`B@{=T!)9dPS<5(ko|nx}KDqrr<#FC$ zprW}v^D{*&+Z)XadU#JVqf@dII!#O`j8BeE%p6R9Ph}61T~W(}%qnvKljs!sK&R|r zp{$@&vMHSb9z;y15J#R@qEn7T*zU~ejCkm>&J6`kr*ztm&zX|S(rG%ji!SA&rcyf9 zIL67om2?_8I!_m}Umq7_^Y-oA#Zk`b*(RHtd7NF}Znkt9WQX3@`jGA)NM0mUIvtrO zT1(D0v29=uEnb{w$ytIG)P@o^-hB3O?v|lN!(o9=&5~uH)AIsEXOFCpqj)Iim92!@ zrq5P-9+yg|tSO@soyw>W1D??d(<4BqhmzOphit;K0F~q3uNtdML#Uj8TMr;oY4uLL#sZTg*{KntTh?9Ks|E>`V@<0^+r8Q z<+ILmp4@{8N$3;?15BqVfEo#N&ysWZmlx;)`CXCCFTP+_&+!9BGFvEGB%IqD8_gao zOX+lM!+=@siyquiBAvE?rPDXE#;A1W*G^nQ5g_R=kxoZ7%%ZaGb0%DJIPx?qogqhO zJ*LyqMUP1hi@k(S$CbGforXB{LGsGd>F83-(3wal^R7J=scAd5ET*%84L6=hr|Blj z7Wx-kio;0f;ldiVW*(Q={POqdx$*r4R?jZ9$&6m#*w|`Vluy=9Y|I46HWetH17yUK zg(w4EJ9BgrW$amy+a6C_J0*E2fpbx}b~?HuFmw)4>j|A)nNC*=N04q=UONX!qDHh4 zSUY<;Is-|tT>+GgK?EIXmk_W1RcohZI<)GewcH9=V)M&yi0?T*S;&dzX`4vh-uC)N z)A)PdvUT)8W9ZaD7#mx*$Z~|%qG^_`BpaeO!Lo%;FmxJ-fqFV^$%=rHyk#pGK&Px_ zbXHil#KrL-buj6yELaw(;%YkKUkapa7nrTB?}}`G`OUv_Mae{Hl^8wV-dJxM5<&wt zbbFcK3AQ_9+3F@5vKqfLM<=eW2zV@QM>dtzl|v1n4^!yt$aw?5(*!y>%OQT}FsSf5 zkswiJ8|Yx$=IB&JR|TgZn&Wz?E?;keEYM1_1bSm>Y% zZa#eDLmU;N4|SQIgAI1}$sVXZFgT>z^Z^LU!0|W(T0{yCf0!EjX>;&LAO`<~D zz~@4AFf*!1RLIA?Bu55L%q3AFpDfHS@8FbAXwC-VhH(b2lBkfAM1>4i@^!%ho$L-6 zIwN!%TYgDrq45#Anuks0EGmTihmH-MwBmTVu=D(<5}W_|$Gl)R$cwJV9%6HShKUfM zyEI%ELYM}GQ$e*tEE9i2M@@1rLlU!P?1H2=4sE(giyRy?(hv40u1ealT~eSBvrRfd zyNCM&$H`?48~aGOIXGY@(O2?=(`_SZ$}Goh*?xmcaf7TWE^W(A+j7mUe-J3nS|bl4 zFa$mRgk|%~KYUxzvI}c1MH5-I>nkf=n$Sx{`fx1`qyH{t^NTP4@GnKdY-!K}#z;0- z*1MhSmx`loIZQ(Wk8j_`bpFS;hea+C?d@L_w1Q-_6V8{6Q_9rg$s?>rfA__||J%2J zTrP9jDh!ljWgY13g!AR%lx1sqZ4LcgTdtxK^z0muZEkHKGFQ43zL$*CD&i{1TyFh( zJ%p7FB%51J140P`FcZPRgCqjq4sy}KKz5T7_$TbvbC5%Ta~O0x2#}rY+cteT$wg;P zWLmoPx#j?!H?c+Rq7$Z_#PfX_8<2K@t!O*28$Rv)8lWki%Psy8N`~wkwkmJ9THR7T3 z;BNNtRRYJse<4A3Gzh5a$oU+BKd3pv8}p|PkESyTWS-9R%BM51X_0KsXa@zFXFzC7 z7v5(kdIt9&S>BQA3oxUoVoITKyu$nmf z3Y#jJ+z*qqs_ zja5j_ka@3{(y3(eZ;SlK&>1NX>@{(tKHaysygq$S9n&erq0P`4_WKN-S|9j@kU7M| zbw}+vDrag||3$qc?-gKof_SbS?h^$${`1EKH*em&K7&nkwzZwIIdcHPkZhn)=2fqP zP9k@#bOyW$bn?u;y(Pwcw%n4nQ`LHu&X|uqX6=+D_(L96N~fklCWEPY-fVv*>SD3K zR?X$|!h_^Vg|n1QU=yKf7#)+@mQ6^JgzO+$a5b-ibcWo}9l%D3ZwqFvvUU8}qnWZr zWovuMh88k|;6DzSzgfz*(i&P`UN}l)vuA(#`s1%zHi6Iv>)_eeb;{<}>~79b3L)=U z7RcT`&`GzI6v9c4PP`v!Yp9lAPiLSC^>q3wPAxHT4v0>+zTY@Hx!s3rY}Ux~^1=Nq z75eyZUn3OQM7TA1DKz#r5Sf(C7QsCQEqq!4bmF~xM5jSt(<75agfg9~&{wj3`zktJ ztw<;2F+h8P6F_E`jbt3sOLW5Hn@1-i^yGA54X|)*5%iZeucyH_a zjT+RVzAm=h3;be21=;I97*BK5S2Pwwb8SM0MR0sqSk`y3&1NQB4n%BsI$6V4H z5Ea_7*{BeYst+N36F(&qppzjqqYWTwK;OZ1x{wbr#~&bho$%-HQ#NlF+58~GN4K^y zX)D_|KK$_3EyN`vla z+epuoaB$KTX_}Nx3bnd{*ARdGzD70^CR2Qd094+%!LYgB(x&b3(r`RUHW?~GE(>h7 zj2?7&nHb-?IaxOA*d(&qz18A%!`=3>dAEBD!s~~#&sxdmN++7HDektA&6T@vqPO<{ z1S>PLxqi3H@_L;yULkxo(zG3!uRn~tA0a}gvWfWY$b2(!yzv1LTF)lpbL+TUGv6E> zuYX1eEwKrNZqzj^t2>CN%00000NkvXXu0mjfGV{f1 literal 0 HcmV?d00001 diff --git a/docs/source/_static/icon-black.png b/docs/source/_images/icon-black.png similarity index 100% rename from docs/source/_static/icon-black.png rename to docs/source/_images/icon-black.png diff --git a/docs/source/_static/icon.png b/docs/source/_images/icon.png similarity index 100% rename from docs/source/_static/icon.png rename to docs/source/_images/icon.png diff --git a/docs/source/_static/logo.png b/docs/source/_images/logo.png similarity index 100% rename from docs/source/_static/logo.png rename to docs/source/_images/logo.png diff --git a/docs/source/_static/nlds-logo.png b/docs/source/_images/nlds-logo.png similarity index 100% rename from docs/source/_static/nlds-logo.png rename to docs/source/_images/nlds-logo.png diff --git a/docs/source/_static/nlds.pdf b/docs/source/_images/nlds.pdf similarity index 100% rename from docs/source/_static/nlds.pdf rename to docs/source/_images/nlds.pdf diff --git a/docs/source/_static/nlds.png b/docs/source/_images/nlds.png similarity index 100% rename from docs/source/_static/nlds.png rename to docs/source/_images/nlds.png diff --git a/docs/source/status_images/all_off.png b/docs/source/_images/status/all_off.png similarity index 100% rename from docs/source/status_images/all_off.png rename to docs/source/_images/status/all_off.png diff --git a/docs/source/status_images/failed.png b/docs/source/_images/status/failed.png similarity index 100% rename from docs/source/status_images/failed.png rename to docs/source/_images/status/failed.png diff --git a/docs/source/status_images/part_failed.png b/docs/source/_images/status/part_failed.png similarity index 100% rename from docs/source/status_images/part_failed.png rename to docs/source/_images/status/part_failed.png diff --git a/docs/source/status_images/short_table.png b/docs/source/_images/status/short_table.png similarity index 100% rename from docs/source/status_images/short_table.png rename to docs/source/_images/status/short_table.png diff --git a/docs/source/status_images/success.png b/docs/source/_images/status/success.png similarity index 100% rename from docs/source/status_images/success.png rename to docs/source/_images/status/success.png diff --git a/docs/source/conf.py b/docs/source/conf.py index fa0b8da8..87601145 100644 --- a/docs/source/conf.py +++ b/docs/source/conf.py @@ -19,7 +19,7 @@ # -- Project information ----------------------------------------------------- -project = 'Near-line Data Store' +project = 'Near-line Data Store Server' copyright = '2022-2024, Neil Massey & Jack Leland' author = 'Neil Massey & Jack Leland' @@ -39,7 +39,7 @@ # Add any paths that contain templates here, relative to this directory. templates_path = ['_templates'] -html_favicon = '_static/icon-black.png' +html_favicon = '_images/icon-black.png' # List of patterns, relative to source directory, that match files and # directories to ignore when looking for source files. @@ -59,7 +59,7 @@ # so a file named "default.css" will overwrite the builtin "default.css". html_static_path = ['_static'] -html_logo = "_static/nlds.png" +html_logo = "_images/nlds.png" html_theme_options = { 'logo_only': False, 'display_version': True, diff --git a/docs/source/coverage/coverage-report.rst b/docs/source/coverage/coverage-report.rst index b8c36006..108cc09c 100644 --- a/docs/source/coverage/coverage-report.rst +++ b/docs/source/coverage/coverage-report.rst @@ -1,6 +1,10 @@ -Test Coverage Report +Test coverage report ==================== Link to the most `recent coverage report `_. -Internal link: :doc:`htmlcov/index` \ No newline at end of file +.. toctree:: + :maxdepth: 1 + :caption: Coverage report + + Index \ No newline at end of file diff --git a/docs/source/index.rst b/docs/source/index.rst index 2d02458b..45f3eeb8 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -3,18 +3,22 @@ You can adapt this file completely to your liking, but it should at least contain the root `toctree` directive. -Near-line Data Store documentation -================================== +Near-line Data Store - Server Documentation +=========================================== -This is the documentation for the Near-line Data Store (NLDS), a tool developed -at JASMIN to provide a single interface for disk, object storage and tape. +This is the documentation for the Near-line Data Store (NLDS) server, a tool +developed at JASMIN to provide a single interface for disk, object storage and +tape. We also have separate documentation for the `NLDS client `_. + +These docs are split into a user-guide, if you plan on simply running the NLDS +server; a development guide, for some of the specifics required to know about if +you plan on contributing to the NLDS; and an API Reference. .. toctree:: :maxdepth: 1 - :caption: Contents + :caption: User Guide Getting started - Specification document Using the system status page The server config file Server config examples @@ -25,8 +29,10 @@ at JASMIN to provide a single interface for disk, object storage and tape. :maxdepth: 1 :caption: Development + Specification document Setting up a CTA tape emulator Database Migrations with Alembic + Test coverage report .. toctree:: @@ -43,3 +49,16 @@ Indices and tables * :ref:`genindex` * :ref:`modindex` * :ref:`search` + + +Acknowledgements +================ + +NLDS was developed at the Centre for Environmental Data Analysis and supported +through the ESiWACE2 project. The project ESiWACE2 has received funding from the +European Union's Horizon 2020 research and innovation programme under grant +agreement No 823988. + +.. image:: _images/esiwace2.png + :width: 400 + :alt: ESiWACE2 Project Logo diff --git a/docs/source/system-status.rst b/docs/source/system-status.rst index bc944dd5..147dca5b 100644 --- a/docs/source/system-status.rst +++ b/docs/source/system-status.rst @@ -61,7 +61,7 @@ e.g.: this link does actually work even though it looks very confusing it will set the time limit to 2 and open a table with catalog, monitor, index and logger rows and would look like this: -.. image:: status_images/short_table.png +.. image:: _images/status/short_table.png :width: 400 :alt: short table @@ -153,26 +153,26 @@ representations of what the whole table will look like. When no consumers are running, the info bar is blue, and the status text is red. -.. image:: status_images/all_off.png +.. image:: _images/status/all_off.png :width: 600 :alt: All consumers off | When all consumers inside a microservice are offline the info bar is red as well as the status column text for the offline microservice. The working microservices status text is green. -.. image:: status_images/failed.png +.. image:: _images/status/failed.png :width: 600 :alt: A consumer failed | When some consumers inside a microservice are offline the info bar is red the partially failed microservice's status text is orange. -.. image:: status_images/part_failed.png +.. image:: _images/status/part_failed.png :width: 600 :alt: some consumers failed | When all consumers online the info bar is green, there is nothing in failed consumer column and all status text is green. -.. image:: status_images/success.png +.. image:: _images/status/success.png :width: 600 :alt: All consumers on From fb0441ddf53885431518532593cfd631f5d3989f Mon Sep 17 00:00:00 2001 From: Jack Leland Date: Thu, 7 Mar 2024 17:51:27 +0000 Subject: [PATCH 06/21] Update readme with current installation and runnning practices --- README.md | 138 ++++++++++++++++++++++++++++++++++++++++++------------ 1 file changed, 109 insertions(+), 29 deletions(-) diff --git a/README.md b/README.md index 93553a8b..9a562b10 100644 --- a/README.md +++ b/README.md @@ -6,52 +6,132 @@ CEDA Near-Line Data Store [![PEP8](https://img.shields.io/badge/code%20style-pep8-orange.svg)](https://www.python.org/dev/peps/pep-0008/) [![Coverage](https://github.com/cedadev/nlds/actions/workflows/testing.yml/coverage.svg)](https://github.com/cedadev/nlds/actions/workflows/testing.yml/htmlcov) -This is the HTTP API server code for the CEDA Near-Line Data Store (NLDS). -It requires the use of the NLDS client, either the command line or library: -[NLDS client on GitHub](https://github.com/cedadev/nlds-client). +This is the server code for the CEDA Near-Line Data Store (NLDS), consisting of +an HTTP API and a cluster of rabbit consumers. The +[NLDS client](https://github.com/cedadev/nlds-client) is required to communicate +with the API, either via the command line interface or python client library. -NLDS server is built upon [FastAPI](https://fastapi.tiangolo.com). +The NLDS is a unified storage solution, allowing easy use of disk, s3 object +storage, and tape from a single interface. It utilises object storage as a cache +for the tape backend allowing for low-latency backup -NLDS requires Python 3. It has been tested with Python 3.9, 3.10 and Python 3.11. +The NLDS server is built upon [FastAPI](https://fastapi.tiangolo.com) for the +API, [RabbitMQ](https://www.rabbitmq.com/) for the message broker, +[minio](https://min.io/) for the s3 client, +[SQLAlchemy](https://www.sqlalchemy.org/) for the database client and +[xrootd](https://xrootd.slac.stanford.edu/) for the tape interactions. + +Documentation can be found [here](https://cedadev.github.io/nlds/index.html). Installation ------------ +If installing locally we strongly recommend the use of a virtual environment to +manage the dependencies. + 1. Create a Python virtual environment: - `python3 -m venv ~/nlds-venv` + + ``` + python3 -m venv nlds-venv + ``` 2. Activate the nlds-venv: - `source ~/nlds-venv/bin/activate` - -3. Install the nlds package with editing capability: - `pip install -e ~/Coding/nlds` - -Running - Dec 2021 ------------------- -1. NLDS currently uses `uvicorn` to run. The command line to invoke it is: -```uvicorn nlds.main:nlds --reload``` + ``` + source nlds-venv/bin/activate + ``` - This will create the NLDS REST-API server at the IP-address: `http://127.0.0.1:8000/` +3. You could either install the nlds package with editing capability from a + locally cloned copy of this repo (note the inclusion of the editable flag + `-e`), e.g. -2. To run the processors, you have two options: - 1. In unique terminals start each processor individually, after - activating the virtual env, for example: - ```source ~/nlds-venv/bin/activate; python nlds_processors/index.py``` - This will send the output to the terminal. + ``` + pip install -e ~/Coding/nlds + ``` - 2. Use the script `test_run_processor.sh`. This will run all five processors - in the background, sending the output to five logs in the `~/nlds_log/` - directory. + or install this repo directly from github: -Viewing the API docs --------------------- + ``` + pip install git+https://github.com/cedadev/nlds.git + ``` -FastAPI displays automatically generated documentation for the REST-API. To browse this go to: http://127.0.0.1:8000/docs#/ +4. (Optional) There are 3 more requirements defined: + * `requirements-dev.txt` - Server Config ------------- -To interface with the JASMIN accounts portal, for the OAuth2 authentication, a `.server_config` file has to be created. This contains infrastructure information and so is not included in the GitHub repository. +To interface with the JASMIN accounts portal, for the OAuth2 authentication, a +`.server_config` file has to be created. This contains infrastructure +information and so is not included in the GitHub repository. See the +[relevant documentation](https://cedadev.github.io/nlds/server-config/server-config.html) +and [examples](https://cedadev.github.io/nlds/server-config/examples.html) for +more information. + +A Jinja-2 template for the `.server_config` file can also be found in the +`templates/` directory. + +Running the Server +------------------ -A Jinja-2 template for the `.server_config` file can be found in the `templates/` directory. +1. The NLDS API requires something to serve the API, usually uvicorn in a local + development environment: + + ``` + uvicorn nlds.main:nlds --reload + ``` + + This will create a local NLDS API server at `http://127.0.0.1:8000/`. + FastAPI displays automatically generated documentation for the REST-API, to + browse this go to http://127.0.0.1:8000/docs/ + +3. To run the microservices, you have two options: + * In individual terminals, after activating the virtual env, (e.g. + `source ~/nlds-venv/bin/activate`), start each of the microservice + consumers: + ``` + nlds_q + index_q + catalog_q + transfer_put_q + transfer_get_q + logging_q + archive_put_q + archive_get_q + ``` + This will send the output of each consumer to its own terminal (as well + as whatever is configured in the logger). + + * Alternatively, you can use the scripts in the `test_run/` directory, + notably `start_test_run.py` to start and `stop_test_run.py` to stop. + This will start a [screen](https://www.gnu.org/software/screen/manual/screen.html) + session with all 8 processors (+ api server) in, sending each output to + a log in the `./nlds_log/` directory. + +Tests +----- + +The NLDS uses pytest for its unit test suite. Once `test/requirements.txt` have +been installed, you can run the tests with +``` +pytest +``` +in the root directory. Pytest is also used for integration testing in the +separate [nlds-test repo](https://github.com/cedadev/nlds-test). + + +License +------- + +The NLDS is available on a BSD 2-Clause License, see the [license](./LICENSE.txt) +for more info. + + + +Acknowledgements +================ + +NLDS was developed at the Centre for Environmental Data Analysis and supported +through the ESiWACE2 project. The project ESiWACE2 has received funding from the +European Union's Horizon 2020 research and innovation programme under grant +agreement No 823988. From a938d96c0aca98a887df0453960c2de1b4f029dd Mon Sep 17 00:00:00 2001 From: Jack Leland Date: Fri, 8 Mar 2024 16:09:58 +0000 Subject: [PATCH 07/21] Update requirements description in README --- README.md | 21 +++++++++++++++++---- 1 file changed, 17 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index 9a562b10..0c176bd9 100644 --- a/README.md +++ b/README.md @@ -55,8 +55,19 @@ manage the dependencies. pip install git+https://github.com/cedadev/nlds.git ``` -4. (Optional) There are 3 more requirements defined: - * `requirements-dev.txt` - +4. (Optional) There are several more requirements/dependencies defined: + * `requirements-dev.txt` - contains development-specific (i.e. not + production appropriate) dependencies. Currently this consists of a psycopg2 + binary python package for interacting with PostgeSQL from a local NLDS + instance. + * `requirements-deployment.txt` - contains deployment-specific + dependencies, excluding `XRootD`. Currently this consists of the psycopg2 + package but built from source instead of a precompiled binary. + * `requirements-tape.txt` - contains tape-specific dependencies, notably + `XRootD`. + * `tests/requirements.txt` - contains the dependencies for the test suite. + * `docs/requirements.txt` - contains the dependencies required for + building the documentation with sphinx. Server Config ------------- @@ -85,7 +96,7 @@ Running the Server FastAPI displays automatically generated documentation for the REST-API, to browse this go to http://127.0.0.1:8000/docs/ -3. To run the microservices, you have two options: +2. To run the microservices, you have two options: * In individual terminals, after activating the virtual env, (e.g. `source ~/nlds-venv/bin/activate`), start each of the microservice consumers: @@ -117,7 +128,9 @@ been installed, you can run the tests with pytest ``` in the root directory. Pytest is also used for integration testing in the -separate [nlds-test repo](https://github.com/cedadev/nlds-test). +separate [nlds-test repo](https://github.com/cedadev/nlds-test). + +The `pytest` test-coverage report can (hopefully) be found [here](https://cedadev.github.io/nlds/coverage/htmlcov/). License From 690cec5ec162a6c7cf2cf5b1b90cde7fd1d2f179 Mon Sep 17 00:00:00 2001 From: Jack Leland Date: Fri, 8 Mar 2024 17:33:11 +0000 Subject: [PATCH 08/21] Update deployment docs with further info --- README.md | 2 +- docs/source/deployment.rst | 75 ++++++++++++++++++++++++++++++++++---- 2 files changed, 68 insertions(+), 9 deletions(-) diff --git a/README.md b/README.md index 0c176bd9..0da3444d 100644 --- a/README.md +++ b/README.md @@ -7,7 +7,7 @@ CEDA Near-Line Data Store [![Coverage](https://github.com/cedadev/nlds/actions/workflows/testing.yml/coverage.svg)](https://github.com/cedadev/nlds/actions/workflows/testing.yml/htmlcov) This is the server code for the CEDA Near-Line Data Store (NLDS), consisting of -an HTTP API and a cluster of rabbit consumers. The +an HTTP API and a cluster of rabbit consumer microservices. The [NLDS client](https://github.com/cedadev/nlds-client) is required to communicate with the API, either via the command line interface or python client library. diff --git a/docs/source/deployment.rst b/docs/source/deployment.rst index 8505b0a0..9c2d781d 100644 --- a/docs/source/deployment.rst +++ b/docs/source/deployment.rst @@ -118,6 +118,30 @@ up to date. This will be discussed in more detail in section There are some slightly more complex deployment configurations involved in the rest of the setup, which are described below. +.. _api_server: + +API Server +---------- + +The NLDS API server, as mentioned above, was written using FastAPI. In a local +development environment this is served using ``uvicorn``, but for the production +deployment the `base-image ` +base-image is used, which runs the server instead with ``guincorn``. They are +functionally identical so this is not a problem per se, just something to be +aware of. The NLDS API helm deployment is an extension of the standard `FastAPI helm chart `_. + +On production, this API server sits facing the public internet behind an NGINX +reverse-proxy, handled by the standard `nginx helm chart `_ +in the ``cedaci/helm-charts`` repo. It is served to the domain +`https://nlds.jasmin.ac.uk `_, with the standard NLDS +API endpoints extending from that (such as ``/docs``, ``/system/status``). The +NLDS API also has an additional endpoint (``/probe/healthz``) for the Kubernetes +liveness probe to periodically ping to ensure the API is alive, and that the +appropriate party is notified if it goes down. Please note, this is not a +deployment specific endpoint and will also exist on any local development +instances. + + .. _tape_keys: Tape Keys @@ -137,7 +161,7 @@ environment variables must be created:: The problem arises with the use of Kubernetes, wherein the keytab content string must be kept secret. This is handled in the CEDA gitlab deployment process -through the use of git-crypt (see `here `_ +through the use of git-crypt (see `here `__ for more details) to encrypt and Kubernetes secrets to decrypt at deployment time. Unfortunately permissions can't be set, no changed, on files made by Kubernetes secrets, so to get the keytab in the right place with the right @@ -157,7 +181,7 @@ schema before the database has been migrated, and this is implemented through two mechanisms in the deployment: 1. An init-container on the catalog, which has the config for both the catalog - and montioring dbs, which has alembic installed and calls:: + and montioring DBs, which has alembic installed and calls:: alembic upgrade head @@ -258,6 +282,47 @@ to be replaced with the appropriate namespace for the cluster you are on, i.e.:: and, as before, these will likely need to be adjusted as understanding of the actual resource use of each of the microservices evolves. +.. _chowning: + +Changing ownership of files +--------------------------- + +A unique problem arose in beta testing where the NLDS was not able to change +ownership of the files downloaded during a ``get`` to the user that requested them +from within a container that was not allowed to run as root. As such, a solution +was required which allowed a very specific set of privileges to be escalated +without leaving any security vulnerabilities open. + +The solution found was to include an additional binary in the +``Generic Consumer`` image - ``chown_nlds`` - which has the ``setuid`` +permissions bit set and is therefore able to change directories. To minimise +exposed attack surface, the binary was compiled from a `rust script `_ +which allows only the ``chown``-ing of files owned by the NLDS user (on JASMIN +``uid=7054096``). Additionally, the target must be a file or directory and the +``uid`` being changed to must be greater than 1024 to avoid clashes with system +``uid``s. This binary will only execute on any containers where the appropriate +security context is set, notably:: + + securityContext: + allowPrivilegeEscalation: true + add: + - CHOWN + +which in the NLDS deployment helm chart is only set for the ``Transfer-Get`` +containers/pods. + + +.. _archive_put: + +Archive Put Cronjob +------------------- + +The process by which the archive process is started has been automated for this +deployment, running as a `Kubernetes cronjob `_ +every 12 hours at midnight and midday. The Helm config controlling this can be +seen `here `_. +This cronjob will simply call the ``send_archive_next()`` entry point, which +sends a message directly to the RabbitMQ exchange for routing to the Catalog. .. _staging: @@ -295,9 +360,3 @@ everything on this page, this was true at the time of writing (2024-03-06). - `https://nlds-master.130.246.130.221.nip.io/ `_ (firewalled) - `https://nlds.jasmin.ac.uk/ `_ (public, ssl secured) - -.. Possible additional sections: -.. - Helm charts? -.. - API Server config? (this is related to Helm charts) -.. - NLDS chown -.. - \ No newline at end of file From 31b02fb95e78aefe3d82519eaa07c267e6384157 Mon Sep 17 00:00:00 2001 From: Jack Leland Date: Fri, 8 Mar 2024 17:38:10 +0000 Subject: [PATCH 09/21] Add CEDA logo to index page --- docs/source/{ => _images}/ceda.png | Bin docs/source/index.rst | 6 +++++- 2 files changed, 5 insertions(+), 1 deletion(-) rename docs/source/{ => _images}/ceda.png (100%) diff --git a/docs/source/ceda.png b/docs/source/_images/ceda.png similarity index 100% rename from docs/source/ceda.png rename to docs/source/_images/ceda.png diff --git a/docs/source/index.rst b/docs/source/index.rst index 45f3eeb8..9ca9b1d0 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -60,5 +60,9 @@ European Union's Horizon 2020 research and innovation programme under grant agreement No 823988. .. image:: _images/esiwace2.png - :width: 400 + :width: 300 :alt: ESiWACE2 Project Logo + +.. image:: _images/ceda.png + :width: 300 + :alt: CEDA Logo \ No newline at end of file From b3df24ba599f7e4c235d492c1fbecd01943f6682 Mon Sep 17 00:00:00 2001 From: Jack Leland Date: Mon, 11 Mar 2024 14:40:23 +0000 Subject: [PATCH 10/21] Remove link colouring in layout template --- docs/source/_templates/layout.html | 15 ++++++++++----- 1 file changed, 10 insertions(+), 5 deletions(-) diff --git a/docs/source/_templates/layout.html b/docs/source/_templates/layout.html index 99ed954d..255c0ebc 100644 --- a/docs/source/_templates/layout.html +++ b/docs/source/_templates/layout.html @@ -15,10 +15,6 @@ background: #1E352F; } - a { - color: #828C51 - } - .note { background: #BEEF9E } @@ -27,4 +23,13 @@ } -{% endblock %} \ No newline at end of file +{% endblock %} + + \ No newline at end of file From 4b96ee5a73769b20a94d44ea0945f4a5ad6d5eac Mon Sep 17 00:00:00 2001 From: Jack Leland Date: Mon, 11 Mar 2024 17:55:10 +0000 Subject: [PATCH 11/21] Update server config docs and template --- docs/source/deployment.rst | 28 ++--- docs/source/server-config/server-config.rst | 128 +++++++++++++++----- nlds/templates/server_config.j2 | 110 +++++++++++++++++ 3 files changed, 225 insertions(+), 41 deletions(-) diff --git a/docs/source/deployment.rst b/docs/source/deployment.rst index 9c2d781d..f3d3fa1f 100644 --- a/docs/source/deployment.rst +++ b/docs/source/deployment.rst @@ -47,16 +47,16 @@ three specific roles: The string after each of these corresponds to the image's location on CEDA's Harbor registry (and therefore what tag/registry address to use to ``docker pull`` each of them). As may be obvious, the FastAPI server runs on the -``Generic Server`` image and contains an installation of asgi, building upon the -``asgi`` `base-image `_, +``Generic Server`` image and contains an installation of ``asgi``, building upon +the ``asgi`` `base-image `_, to actually run the server. The rest run on the ``Generic Consumer`` image, which has an installation of the NLDS repo, along with its dependencies, to allow it to run a given consumer. The only dependency which isn't included is -xrootd as it is a very large and long installation process and unnecessary to -the running of the non-tape consumers. Therefore the ``Tape Consumer`` image was -created, which appropriately builds upon the ``Geneic Consumer`` image with an -additional installation of ``xrootd`` with which to run tape commands. The two -tape consumers, ``Archive-Put`` and ``Archive-Get``, run on containers using +``xrootd`` as it is a very large and long installation process and unnecessary +to the running of the non-tape consumers. Therefore the ``Tape Consumer`` image +was created, which appropriately builds upon the ``Geneic Consumer`` image with +an additional installation of ``xrootd`` with which to run tape commands. The +two tape consumers, ``Archive-Put`` and ``Archive-Get``, run on containers using this image. The two consumer containers run as the user NLDS, which is an official JASMIN @@ -204,10 +204,10 @@ the cluster controller) and no means of attaching a more persistent volume to store logs in long-term. The, relatively new, solution that exists on the CEDA cluster is the use of -`fluentd`, and more precisely `fluentbit `_, +``fluentd``, and more precisely `fluentbit `_, to aggregate logs from the NLDS logging microservice and send them to a single -external location running `fluentd` – currently the stats-collection virtual -machine run on JASMIN. Each log sent to the `fluentd` service is tagged with a +external location running ``fluentd`` – currently the stats-collection virtual +machine run on JASMIN. Each log sent to the ``fluentd`` service is tagged with a string representing the particular microservice log file it was collected from, e.g. the logs from the indexer microservice on the staging deployment are tagged as:: @@ -217,15 +217,15 @@ as:: This is practically achieved through the use of a sidecar – a further container running in the same pod as the logging container – running the ``fluentbit`` image as defined by the `fluentbit helm chart `_. -The full ``fluentbit`` config, including the full list of tags, can be found `in -the logging config yamls `_. +The full ``fluentbit`` config, including the full list of tags, can be found +`in the logging config yamls `_. When received by the fluentd server, each tagged log is collated into a larger log file for help with debugging at some later date. The log files on the logging microservice's container are rotated according to size, and so should not exceed the pod's allocated memory limit. .. note:: - The `fluentbit` service is still in its infancy and subject to change at + The ``fluentbit`` service is still in its infancy and subject to change at short notice as the system & helm chart get more widely adopted. For example, the length of time log files are kept on the stats machine has not been finalised yet. @@ -258,7 +258,7 @@ where it is set to 8 and the Transfer-Get where is set to 2. the size of a ``Rabbit`` queue for a given microservice, and while this is `in theory` `possible `_, this was not possible with the current installation of Kubernetes without - additional plugins, namely `Prometheus`. + additional plugins, namely ``Prometheus``. The other aspect of scaling is the resource requested by each of the pods, which have current `default values `_ diff --git a/docs/source/server-config/server-config.rst b/docs/source/server-config/server-config.rst index f04ff033..ec033872 100644 --- a/docs/source/server-config/server-config.rst +++ b/docs/source/server-config/server-config.rst @@ -75,33 +75,32 @@ brokering system. The following is an outline of what is required:: Here the ``user`` and ``password`` fields refer to the username and password for the rabbit server you wish to connect to, which is in turn specified with -``server``. ``vhost`` is similarly the virtual host on the rabbit server that +``server``. ``vhost`` is similarly the virtualhost on the rabbit server that you wish to connect to. The next two dictionaries are context specific. All publishing elements of the NLDS, i.e. parts that will send messages, will require an exchange to publish messages to. ``exchange`` is determines that exchange, with three required subfields: ``name``, ``type``, and ``delayed``. The former two are self -descriptive, they should just be the name of the exchange on the virtualhost and +descriptive, they should just be the name of the exchange on the `virtualhost` and it's corresponding type e.g. one of fanout, direct or topic. ``delay`` is a boolean (``true`` or ``false`` in json-speak) dictating whether to use the delay functionality utilised within the NLDS. Note that this requires the rabbit server have the DelayedRabbitExchange plugin installed. -Exchanges can be declared and created if not present on the virtual host the -first time the NLDS is run, virtualhosts cannot and so will have to be created +Exchanges can be declared and created if not present on the `virtualhost` the +first time the NLDS is run, `virtualhosts` cannot and so will have to be created beforehand manually on the server or through the admin interface. If an exchange is requested but incorrect information given about either its `type` or `delayed` status, then the NLDS will throw an error. ``queues`` is a list of queue dictionaries and must be implemented on consumers, i.e. message processors, to tell ``pika`` where to take messages from. Each -queue dictionary consists of a ``name`` and a list of `bindings`, with each +queue dictionary consists of a ``name`` and a list of ``bindings``, with each ``binding`` being a dictionary containing the name of the ``exchange`` the queue takes messages from, and the routing key that a message must have to be accepted onto the queue. For more information on exchanges, routing keys, and other -RabbitMQ features, please see [Rabbit's excellent documentation] -(https://www.rabbitmq.com/tutorials/tutorial-five-python.html). +RabbitMQ features, please see `Rabbit's excellent documentation `_. Generic optional sections @@ -117,13 +116,14 @@ Logging The logging configuration options look like the following:: "logging": { - "enable": boolean + "enable": boolean, "log_level": str - ("none" | "debug" | "info" | "warning" | "error" | "critical"), "log_format": str - see python logging docs for details, "add_stdout_fl": boolean, "stdout_log_level": str - ("none" | "debug" | "info" | "warning" | "error" | "critical"), "log_files": List[str], - "rollover": str - see python logging docs for details + "max_bytes": int, + "backup_count": int } These all set default options the native python logging system, with @@ -137,17 +137,17 @@ to be different from the default log level. ``log_files`` is a list of strings describing the path or paths to log files being written to. If no log files paths are given then no file logging will be -done. If active, the file logging will be done with a TimedRotatingFileHandler, -i.e. the files will be rotated on a rolling basis, with the rollover time -denoted by the ``rollover`` option, which is a time string similar to that found -in crontab. Please see the [python logging docs] -(https://docs.python.org/3/library/logging.handlers.html#logging.handlers.TimedRotatingFileHandler) -for more info on this. +done. If active, the file logging will be done with a RotatingFileHandler, i.e. +the files will be rotated when they reach a certain size. The threshold size is +determined by ``max_bytes`` and the maximum number of files which are kept after +rotation is controlled by ``backup_count``, both strings. For more information +on this please refer to the `python logging docs `_. As stated, these all set the default log options for all publishers and consumers within the NLDS - these can be overridden on a consumer-specific basis by inserting a ``logging`` sub-dictionary into a consumer-specific optional -section. +section. Each sub-dictionary has identical configuration options to those listed +above. General ^^^^^^^ @@ -181,8 +181,8 @@ The server config section is ``nlds_q``, and the following options are available "nlds_q": { "logging": [standard_logging_dictionary], - "retry_delays": List[int] - "print_tracebacks_fl": boolean, + "retry_delays": List[int], + "print_tracebacks_fl": boolean } Not much specifically happens in the NLDS worker that requires configuration, so @@ -256,6 +256,7 @@ The server config entry for the catalog consumer is as follows:: "logging": {standard_logging_dictionary}, "retry_delays": List[int], "print_tracebacks_fl": boolean, + "max_retries": int, "db_engine": str, "db_options": { "db_name" : str, @@ -263,12 +264,13 @@ The server config entry for the catalog consumer is as follows:: "db_passwd" : str, "echo": boolean }, - "max_retries": int + default_tenancy: str, + default_tape_url: str } where ``logging``, ``retry_delays``, and ``print_tracebacks_fl`` are, as above, standard configurables within the NLDS consumer ecosystem. ``max_retries`` is -similarly available in the cataloguer, with the same meaning as above. +similarly available in the cataloguer, with the same meaning as defined above. Here we also have two keys which control database behaviour via SQLAlchemy: ``db_engine`` and ``db_options``. ``db_engine`` is a string which specifies @@ -281,6 +283,11 @@ your chosen flavour of database), along with the database username and password ``db_password``. Finally in this sub-dictionary ``echo``, an optional boolean flag which controls the auto-logging of the SQLAlchemy engine. +Finally ``default_tenancy`` and ``default_tape_url`` are the default values to +place into the Catalog for a new Location's ``tenancy`` and ``tape_url`` values +if not explicitly defined before reaching the catalog. This will happen if the +user, for example, does not define a tenancy in their client-config. + Transfer-put and Transfer-get ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -295,20 +302,35 @@ The server entry for the transfer-put consumer is as follows:: "filelist_max_length": int, "check_permissions_fl": boolean, "use_pwd_gid_fl": boolean, - "tenancy": "cedadev-o.s3.jc.rl.ac.uk", + "tenancy": str, "require_secure_fl": false } where we have ``logging``, ``retry_delays`` and ``print_tracebacks_fl`` as their standard definitions defined above, and ``max_retries``, ``filelist_max_length`` -, ``check_permissions_fl``, and ``use_pwd_gid_fl`` defined the same as for the -Indexer consumer. +, and ``check_permissions_fl`` defined the same as for the Indexer consumer. New definitions for the transfer processor are the ``tenancy`` and ``require_secure_fl``, which control ``minio`` behaviour. ``tenancy`` is a string which denotes the address of the object store tenancy to upload/download -files to/from, and ``require_secure_fl`` which specifies whether or not you -require signed ssl certificates at the tenancy location. +files to/from (e.g. ``_), and ``require_secure_fl`` +which specifies whether or not you require signed ssl certificates at the +tenancy location. + +The transfer-get consumer is identical except for the addition of config +controlling the change-ownership functionality on downloaded files – see +:ref:`chowning` for details on why this is necessary. The additional config is +as follows:: + + "transfer_get_q": { + ... + "chown_fl": boolean, + "chown_cmd": str + } + +where ``chown_fl`` is a boolean flag to specify whether to attempt to ``chown`` +files back to the requesting user, and ``chown_cmd`` is the name of the +executable to use to ``chown`` said file. Monitor @@ -338,7 +360,7 @@ messages which failed due to an unexpected exception. Logger ^^^^^^ -And finally, the server config entry for the Logger consumer is as follows:: +The server config entry for the Logger consumer is as follows:: "logging_q": { "logging": {standard_logging_dictionary}, @@ -349,4 +371,56 @@ where the options have been previously defined. Note that there is no special configurable behaviour on the Logger consumer as it is simply a relay for redirecting logging messages into log files. It should also be noted that the ``log_files`` option should be set in the logging sub-dictionary for this to -work properly, which may be a mandatory setting in future versions. \ No newline at end of file +work properly, which may be a mandatory setting in future versions. + + +Archive-Put and Archive-Get +^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +And, finally, the server config entry for the archive-put consumer is as +follows:: + + "archive_put_q": { + "logging": {standard_logging_dictionary} + "max_retries": int, + "retry_delays": List[int], + "print_tracebacks_fl": boolean, + "tenancy": str, + "check_permissions_fl": boolean, + "require_secure_fl": boolean, + "tape_url": str, + "tape_pool": str, + "query_checksum_fl": boolean, + "chunk_size": int + } + +which is a combination of standard configuration, object-store configuration and +as-yet-unseen tape configuration. Firstly, we have the standard options +``logging``, ``max_retries``, ``retry_delays``, and ``print_tracebacks_fl``, +which we have defined above. Then we have the object-store configuration options +which we saw previously in the :ref:`transfer_put_get` consumer config, and have +the same definitions. + +The latter four options control tape configuration, ``taoe_url`` and +``tape_pool`` defining the ``xrootd`` url and tape pool at which to attempt to +put files onto - note that these two values are combined together into a single +``tape_path`` in the archiver. ``query_checksum`` is the next option, is a +boolean flag to control whether the ADLER32 checksum calculated during streaming +is used to check file integrity at the end of a write. Finally ``chunk_size`` is +the size, in bytes, to chunk the stream into when writing into or reading from +the CTA cache. This defaults to 5 MiB as this is the lower limit for +``part_size`` when uploading back to object-store during an archive-get, but has +not been properly benchmarked or optimised yet. + +Note that the above has been listed for the archive-put consumer but are +shared by the archive-get consumer. The archive-get does have one additional +config option:: + + "archive_get_q": { + ... + "prepare_requeue": int + } + +where ``prepare_requeue`` is the prepare-requeue delay, i.e. the delay, in +milliseconds, before an archive recall message is requeued following a negative +read-preparedness query has been made. This defaults to 30 seconds. \ No newline at end of file diff --git a/nlds/templates/server_config.j2 b/nlds/templates/server_config.j2 index 90420e4b..622836e6 100644 --- a/nlds/templates/server_config.j2 +++ b/nlds/templates/server_config.j2 @@ -29,5 +29,115 @@ ] } ] + }, + "logging": { + "enable": "{{ logging_enable }}", + "log_level": "{{ logging_log_level }}", + "log_format": "{{ logging_log_format }}", + "add_stdout_fl": "{{ logging_add_stdout_fl }}", + "stdout_log_level": "{{ logging_stdout_log_level }}", + "log_files": "{{ logging_log_files }}", + "max_bytes": "{{ logging_max_bytes }}", + "backup_count": "{{ logging_backup_count }}" + }, + "general": { + "retry_delays": "{{ general_retry }}" + }, + "nlds_q": { + "logging": "{{ nlds_q_logging_dict }}", + "max_retries": "{{ nlds_q_max_retries }}", + "retry_delays": "{{ nlds_q_retry_delays }}", + "print_tracebacks_fl": "{{ nlds_q_print_tracebacks_fl }}" + }, + "index_q": { + "logging": "{{ index_q_logging_dict }}", + "max_retries": "{{ index_q_max_retries }}", + "retry_delays": "{{ index_q_retry_delays }}", + "print_tracebacks_fl": "{{ index_q_print_tracebacks_fl }}", + "filelist_max_length": "{{ index_q_filelist_max_length }}", + "message_threshold": "{{ index_q_message_threshold }}", + "check_permissions_fl": "{{ index_q_check_permissions_fl }}", + "check_filesize_fl": "{{ index_q_check_filesize_fl }}" + }, + "catalog_q": { + "logging": "{{ catalog_q_logging_dict }}", + "max_retries": "{{ catalog_q_max_retries }}", + "retry_delays": "{{ catalog_q_retry_delays }}", + "print_tracebacks_fl": "{{ catalog_q_print_tracebacks_fl }}", + "db_engine": "{{ catalog_q_db_engine }}", + "db_options": { + "db_name" : "{{ catalog_q_db_options_db_name }}", + "db_user" : "{{ catalog_q_db_options_db_user }}", + "db_passwd" : "{{ catalog_q_db_options_db_passwd }}", + "echo": "{{ catalog_q_db_options_echo }}" + }, + default_tenancy: "{{ catalog_q_default_tenancy }}", + default_tape_url: "{{ catalog_q_default_tape_url }}" + }, + "monitor_q": { + "logging": "{{ monitor_q_logging_dict }}", + "max_retries": "{{ monitor_q_max_retries }}", + "retry_delays": "{{ monitor_q_retry_delays }}", + "print_tracebacks_fl": "{{ monitor_q_print_tracebacks_fl }}", + "db_engine": "{{ monitor_q_db_engine }}", + "db_options": { + "db_name" : "{{ monitor_q_db_options_db_name }}", + "db_user" : "{{ monitor_q_db_options_db_user }}", + "db_passwd" : "{{ monitor_q_db_options_db_passwd }}", + "echo": "{{ monitor_q_db_options_echo }}" + }, + }, + "transfer_put_q": { + "logging": "{{ transfer_put_q_logging_dict }}", + "max_retries": "{{ transfer_put_q_max_retries }}", + "retry_delays": "{{ transfer_put_q_retry_delays }}", + "print_tracebacks_fl": "{{ transfer_put_q_print_tracebacks_fl }}", + "filelist_max_length": "{{ transfer_put_q_filelist_max_length }}", + "check_permissions_fl": "{{ transfer_put_q_check_permissions_fl }}", + "tenancy": "{{ transfer_put_q_tenancy }}", + "require_secure_fl": "{{ transfer_put_q_require_secure_fl }}" + }, + "transfer_get_q": { + "logging": "{{ transfer_put_q_logging_dict }}", + "max_retries": "{{ transfer_put_q_max_retries }}", + "retry_delays": "{{ transfer_put_q_retry_delays }}", + "print_tracebacks_fl": "{{ transfer_put_q_print_tracebacks_fl }}", + "filelist_max_length": "{{ transfer_put_q_filelist_max_length }}", + "check_permissions_fl": "{{ transfer_put_q_check_permissions_fl }}", + "tenancy": "{{ transfer_put_q_tenancy }}", + "require_secure_fl": "{{ transfer_put_q_require_secure_fl }}", + "chown_fl": "{{ transfer_put_q_chown_fl }}", + "chown_cmd": "{{ transfer_put_q_chown_cmd }}" + }, + "logging_q": { + "logging": "{{ logging_q_logging_dict }}", + "print_tracebacks_fl": "{{ logging_q_print_tracebacks_fl }}" + }, + "archive_put_q": { + "logging": "{{ archive_put_q_logging_dict }}", + "max_retries": "{{ archive_put_q_max_retries }}", + "retry_delays": "{{ archive_put_q_retry_delays }}", + "print_tracebacks_fl": "{{ archive_put_q_print_tracebacks_fl }}", + "tenancy": "{{ archive_put_q_tenancy }}", + "check_permissions_fl": "{{ archive_put_q_check_permissions_fl }}", + "require_secure_fl": "{{ archive_put_q_require_secure_fl }}", + "tape_url": "{{ archive_put_q_tape_url }}", + "tape_pool": "{{ archive_put_q_tape_pool }}", + "query_checksum_fl": "{{ archive_put_q_query_checksum_fl }}", + "chunk_size": "{{ archive_put_q_chunk_size }}" + }, + "archive_get_q": { + "logging": "{{ archive_get_q_logging_dict }}", + "max_retries": "{{ archive_get_q_max_retries }}", + "retry_delays": "{{ archive_get_q_retry_delays }}", + "print_tracebacks_fl": "{{ archive_get_q_print_tracebacks_fl }}", + "tenancy": "{{ archive_get_q_tenancy }}", + "check_permissions_fl": "{{ archive_get_q_check_permissions_fl }}", + "require_secure_fl": "{{ archive_get_q_require_secure_fl }}", + "tape_url": "{{ archive_get_q_tape_url }}", + "tape_pool": "{{ archive_get_q_tape_pool }}", + "query_checksum_fl": "{{ archive_get_q_query_checksum_fl }}", + "chunk_size": "{{ archive_get_q_chunk_size }}" + "prepare_requeue": "{{ archive_get_q_prepare_requeue }}" } } From 2a0f69c39b337ec4c263d1a2ade251a3747260b4 Mon Sep 17 00:00:00 2001 From: Jack Leland Date: Mon, 11 Mar 2024 17:58:13 +0000 Subject: [PATCH 12/21] Fix json formatting typo in server config template --- nlds/templates/server_config.j2 | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/nlds/templates/server_config.j2 b/nlds/templates/server_config.j2 index 622836e6..b0b793d6 100644 --- a/nlds/templates/server_config.j2 +++ b/nlds/templates/server_config.j2 @@ -85,7 +85,7 @@ "db_user" : "{{ monitor_q_db_options_db_user }}", "db_passwd" : "{{ monitor_q_db_options_db_passwd }}", "echo": "{{ monitor_q_db_options_echo }}" - }, + } }, "transfer_put_q": { "logging": "{{ transfer_put_q_logging_dict }}", From 046cb1041b98266b70ef435c4735ff5f4de29421 Mon Sep 17 00:00:00 2001 From: Jack Leland Date: Tue, 12 Mar 2024 09:20:05 +0000 Subject: [PATCH 13/21] Remove references to use_pwd_gid_fl --- docs/source/server-config/server-config.rst | 11 +---------- 1 file changed, 1 insertion(+), 10 deletions(-) diff --git a/docs/source/server-config/server-config.rst b/docs/source/server-config/server-config.rst index ec033872..3a213362 100644 --- a/docs/source/server-config/server-config.rst +++ b/docs/source/server-config/server-config.rst @@ -207,7 +207,6 @@ Server config section is ``index_q``, and the following options are available:: "max_retries": int, "check_permissions_fl": boolean, "check_filesize_fl": boolean, - "use_pwd_gid_fl": boolean } where ``logging``, ``retry_delays``, and ``print_tracebacks_fl`` are, as above, @@ -237,14 +236,7 @@ list. ``check_permissions_fl`` and ``check_filesize_fl`` are commonly used boolean flags to control whether the indexer checks the permissions and filesize of -files respectively during the indexing step. - -``use_pwd_gid_fl`` is a final boolean flag which controls how permissions -checking goes about getting the gid to check group permissions against. If True, -it will _just_ use the gid found in the ``pwd`` table on whichever machine the -indexer is running on. If false, then this gid is used `as well as` all of those -found using the ``os.groups`` command - which will read all groups found on the -machine the indexer is running on. +files respectively during the indexing step. Cataloguer @@ -301,7 +293,6 @@ The server entry for the transfer-put consumer is as follows:: "print_tracebacks_fl": boolean, "filelist_max_length": int, "check_permissions_fl": boolean, - "use_pwd_gid_fl": boolean, "tenancy": str, "require_secure_fl": false } From 8665aeeba88d834b126c282c47126d563d863ba8 Mon Sep 17 00:00:00 2001 From: Jack Leland Date: Tue, 12 Mar 2024 13:08:13 +0000 Subject: [PATCH 14/21] Add some missing sections to server config docs --- docs/source/server-config/examples.rst | 126 ++++++++++++++++---- docs/source/server-config/server-config.rst | 82 +++++++++++-- nlds/templates/server_config.j2 | 14 ++- 3 files changed, 188 insertions(+), 34 deletions(-) diff --git a/docs/source/server-config/examples.rst b/docs/source/server-config/examples.rst index 62fc8130..3196a774 100644 --- a/docs/source/server-config/examples.rst +++ b/docs/source/server-config/examples.rst @@ -19,17 +19,20 @@ machine - likely a laptop or single vm. This file would be saved at "oauth_token_introspect_url" : "[REDACTED]" } }, + "general": { + "retry_delays": [ + 1, 5, 10, 20, 30, 60, 120, 240, 480 + ] + }, "index_q":{ "logging":{ "enable": true }, "filelist_threshold": 10000, "check_permissions_fl": true, - "use_pwd_gid_fl": true, + "max_filesize": 5000000, "retry_delays": [ - 0, - 1, - 2 + 0, 0, 0 ] }, "nlds_q":{ @@ -43,11 +46,8 @@ machine - likely a laptop or single vm. This file would be saved at }, "tenancy": "example-tenancy.s3.uk", "require_secure_fl": false, - "use_pwd_gid_fl": true, "retry_delays": [ - 0, - 1, - 2 + 0, 1, 2 ] }, "transfer_get_q":{ @@ -56,12 +56,16 @@ machine - likely a laptop or single vm. This file would be saved at }, "tenancy": "example-tenancy.s3.uk", "require_secure_fl": false, - "use_pwd_gid_fl": true + "retry_delays": [ + 10, + 20, + 30 + ] }, "monitor_q":{ "db_engine": "sqlite", "db_options": { - "db_name" : "//Users/jack.leland/nlds/nlds_monitor.db", + "db_name" : "//home/nlds/nlds_monitor.db", "db_user" : "", "db_passwd" : "", "echo": false @@ -86,32 +90,69 @@ machine - likely a laptop or single vm. This file would be saved at "logs/transfer_put_q.txt", "logs/transfer_get_q.txt", "logs/logging_q.txt", - "logs/api_server.txt" - ] + "logs/api_server.txt", + "logs/archive_put_q.txt", + "logs/archive_get_q.txt" + ], + "max_bytes": 33554432, + "backup_count": 0 } }, "catalog_q":{ "db_engine": "sqlite", "db_options": { - "db_name" : "//Users/jack.leland/nlds/nlds_catalog.db", + "db_name" : "//home/nlds/nlds_catalog.db", "db_user" : "", "db_passwd" : "", "echo": false }, "retry_delays": [ 0, - 1, - 2 + 0, + 0 ], "logging":{ "enable": true + }, + "default_tape_url": "root://example-tape.endpoint.uk//eos/ctaeos/cta/nlds", + "default_tenancy": "example-tenancy.s3.uk", + }, + "archive_get_q": { + "tape_url": "root://example-tape.endpoint.uk//eos/ctaeos/cta/nlds", + "tape_pool": "", + "chunk_size": 262144, + "tenancy": "example-tenancy.s3.uk", + "print_tracebacks_fl": false, + "check_permissions_fl": false, + "require_secure_fl": false, + "max_retries": 5, + "retry_delays": [0.0, 0.0, 0.0], + "logging": { + "enable": true + } + }, + "archive_put_q": { + "query_checksum_fl": true, + "tape_url": "root://example-tape.endpoint.uk//eos/ctaeos/cta/nlds", + "tape_pool": "", + "chunk_size": 262144, + "tenancy": "example-tenancy.s3.uk", + "print_tracebacks_fl": false, + "check_permissions_fl": false, + "require_secure_fl": false, + "max_retries": 1, + "retry_delays": [0.0, 0.0, 0.0], + "logging": { + "enable": true } }, "rabbitMQ": { - "user": "full_access", - "password": "passwordletmein123", - "server": "130.246.3.98", - "vhost": "delayed-test", + "user": "[REDACTED]", + "password": "[REDACTED]", + "heartbeat": 5, + "server": "[REDACTED]", + "vhost": "delayed-nlds", + "admin_port": 15672, "exchange": { "name": "test_exchange", "type": "topic", @@ -129,6 +170,10 @@ machine - likely a laptop or single vm. This file would be saved at "exchange": "test_exchange", "routing_key": "nlds-api.*.complete" }, + { + "exchange": "test_exchange", + "routing_key": "nlds-api.*.reroute" + }, { "exchange": "test_exchange", "routing_key": "nlds-api.*.failed" @@ -175,6 +220,18 @@ machine - likely a laptop or single vm. This file would be saved at { "exchange": "test_exchange", "routing_key": "*.catalog-del.start" + }, + { + "exchange": "test_exchange", + "routing_key": "*.catalog-archive-next.start" + }, + { + "exchange": "test_exchange", + "routing_key": "*.catalog-archive-del.start" + }, + { + "exchange": "test_exchange", + "routing_key": "*.catalog-archive-update.start" } ] }, @@ -204,18 +261,41 @@ machine - likely a laptop or single vm. This file would be saved at "routing_key": "*.log.*" } ] + }, + { + "name": "archive_get_q", + "bindings": [ + { + "exchange": "test_exchange", + "routing_key": "*.archive-get.start" + } + ] + }, + { + "name": "archive_put_q", + "bindings": [ + { + "exchange": "test_exchange", + "routing_key": "*.archive-put.start" + } + ] } ] }, "rpc_publisher": { "queue_exclusivity_fl": true + }, + "cronjob_publisher": { + "access_key": "[REDACTED]", + "secret_key": "[REDACTED]", + "tenancy": "example-tenancy.s3.uk" } } -Note that this is purely an example and doesn't necessarily use all features -within the NLDS. For example, several individual consumers have ``retry_delays`` -set but not generic ``retry_delays`` is set in the ``general`` section. Note -also that the jasmin authenication configuration is redacted for security + +Note that this is purely illustrative and doesn't necessarily use all features +within the NLDS - it is provided as a reference for making a new working server +config. Note also that certain sensitive information is redacted for security purposes. Distributed NLDS diff --git a/docs/source/server-config/server-config.rst b/docs/source/server-config/server-config.rst index 3a213362..b29a6453 100644 --- a/docs/source/server-config/server-config.rst +++ b/docs/source/server-config/server-config.rst @@ -9,8 +9,10 @@ server_config in the templates section of the main nlds package demystify the configuration needed for (a) a local development copy of the nlds, and (b) a production system spread across several pods/virtual machines. -*Please note that the NLDS is in active development and all of this is subject -to change with no notice.* +.. note:: + Please note that the NLDS is still being developed and so the following is + subject to change in future versions. + Required sections ----------------- @@ -53,6 +55,7 @@ brokering system. The following is an outline of what is required:: "rabbitMQ": { "user": "{{ rabbit_user }}", "password": "{{ rabbit_password }}", + "heartbeat": "{{ rabbit_heartbeat }}", "server": "{{ rabbit_server }}", "vhost": "{{ rabbit_vhost }}", "exchange": { @@ -75,15 +78,19 @@ brokering system. The following is an outline of what is required:: Here the ``user`` and ``password`` fields refer to the username and password for the rabbit server you wish to connect to, which is in turn specified with -``server``. ``vhost`` is similarly the virtualhost on the rabbit server that -you wish to connect to. +``server``. ``vhost`` is similarly the `virtualhost` on the rabbit server that +you wish to connect to. ``heartbeat`` is a recent addition which determines the +`heartbeats` on the BlockingConnection that the NLDS makes with the rabbit +server. This essentially puts a hard limit on when a connection has to be +responsive by before it is killed by the server, see `the rabbit docs `_ +for more details. The next two dictionaries are context specific. All publishing elements of the NLDS, i.e. parts that will send messages, will require an exchange to publish messages to. ``exchange`` is determines that exchange, with three required subfields: ``name``, ``type``, and ``delayed``. The former two are self -descriptive, they should just be the name of the exchange on the `virtualhost` and -it's corresponding type e.g. one of fanout, direct or topic. ``delay`` is a +descriptive, they should just be the name of the exchange on the `virtualhost` +and it's corresponding type e.g. one of fanout, direct or topic. ``delay`` is a boolean (``true`` or ``false`` in json-speak) dictating whether to use the delay functionality utilised within the NLDS. Note that this requires the rabbit server have the DelayedRabbitExchange plugin installed. @@ -207,6 +214,7 @@ Server config section is ``index_q``, and the following options are available:: "max_retries": int, "check_permissions_fl": boolean, "check_filesize_fl": boolean, + "max_filesize": int } where ``logging``, ``retry_delays``, and ``print_tracebacks_fl`` are, as above, @@ -236,7 +244,11 @@ list. ``check_permissions_fl`` and ``check_filesize_fl`` are commonly used boolean flags to control whether the indexer checks the permissions and filesize of -files respectively during the indexing step. +files respectively during the indexing step. If the filesize is being checked, +``max_filesize`` determines the maximum filesize, in bytes, of an individual +file which can be added to any given holding. This defaults to ``500GB``, but is +typically determined by the size of the cache in front of the tape, which for +the STFC CTA instance is ``500GB`` (hence the default value). Cataloguer @@ -280,6 +292,7 @@ place into the Catalog for a new Location's ``tenancy`` and ``tape_url`` values if not explicitly defined before reaching the catalog. This will happen if the user, for example, does not define a tenancy in their client-config. +.. _transfer_put_get: Transfer-put and Transfer-get ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -364,12 +377,12 @@ redirecting logging messages into log files. It should also be noted that the ``log_files`` option should be set in the logging sub-dictionary for this to work properly, which may be a mandatory setting in future versions. +.. _archive_put_get: Archive-Put and Archive-Get ^^^^^^^^^^^^^^^^^^^^^^^^^^^ -And, finally, the server config entry for the archive-put consumer is as -follows:: +Finally, the server config entry for the archive-put consumer is as follows:: "archive_put_q": { "logging": {standard_logging_dictionary} @@ -414,4 +427,53 @@ config option:: where ``prepare_requeue`` is the prepare-requeue delay, i.e. the delay, in milliseconds, before an archive recall message is requeued following a negative -read-preparedness query has been made. This defaults to 30 seconds. \ No newline at end of file +read-preparedness query has been made. This defaults to 30 seconds. + + +Publisher-specific optional sections +------------------------------------ + +There are two, non-consumer, elements to the NLDS which can optionally be +configured, listed below. + +RPC Publisher +^^^^^^^^^^^^^ + +The Remote Procedure Call (RPC) Publisher, the specific rabbit publisher which +sits inside the API server and makes RPCs to the databases for quick metadata +access from the client, has its own small config section:: + + "rpc_publisher": { + "time_limit": int, + "queue_exclusivity_fl": boolean + } + +where ``time_limit`` is the number of seconds the publisher waits before +declaring the RPC timed out and the receiving consumer non-responsive, and +``queue_exclusivity_fl`` controls whether the queue declared by the publisher is +exclusive to the publisher. These values default to ``30`` seconds and ``True`` +respectively. + + +Cronjob Publisher +^^^^^^^^^^^^^^^^^ + +The Archive-Put process, as described in :ref:`archive_put`, is periodically +initiated by a cronjob which sends a message to the catalog to get the next, +unarchived holding. This requires a small amount of configuration in order to +(a) get access to the object store, (b) change the default ``tenancy`` or +``tape_url``, if necessary. As such the allowed config options look like:: + + "cronjob_publisher": { + "access_key": str, + "secret_key": str, + "tenancy": str, + "tape_url": str + } + +where ``tape_url`` is identical to that specified in :ref:`archive_put_get`, and +``access_key``, ``secret_key`` and ``tenancy`` are specified as in the +`client config `_, +referring to the objectstore tenancy located at ``tenancy`` and ``token`` and +``secret_key`` required for accessing it. In practice only the ``access_key`` +and ``secret_key`` are specified during deployment. \ No newline at end of file diff --git a/nlds/templates/server_config.j2 b/nlds/templates/server_config.j2 index b0b793d6..b7832b88 100644 --- a/nlds/templates/server_config.j2 +++ b/nlds/templates/server_config.j2 @@ -10,6 +10,7 @@ "rabbitMQ": { "user": "{{ rabbit_user }}", "password": "{{ rabbit_password }}", + "heartbeat": "{{ rabbit_heartbeat }}", "server": "{{ rabbit_server }}", "admin_port": "{{ rabbit_port }}", "vhost": "{{ rabbit_vhost }}", @@ -57,7 +58,8 @@ "filelist_max_length": "{{ index_q_filelist_max_length }}", "message_threshold": "{{ index_q_message_threshold }}", "check_permissions_fl": "{{ index_q_check_permissions_fl }}", - "check_filesize_fl": "{{ index_q_check_filesize_fl }}" + "check_filesize_fl": "{{ index_q_check_filesize_fl }}", + "max_filesize": "{{ index_q_max_filesize }}" }, "catalog_q": { "logging": "{{ catalog_q_logging_dict }}", @@ -139,5 +141,15 @@ "query_checksum_fl": "{{ archive_get_q_query_checksum_fl }}", "chunk_size": "{{ archive_get_q_chunk_size }}" "prepare_requeue": "{{ archive_get_q_prepare_requeue }}" + }, + "rpc_publisher": { + "time_limit": "{{ rpc_publisher_time_limit }}", + "queue_exclusivity_fl": "{{ rpc_publisher_queue_exclusivity_fl }}" + }, + "cronjob_publisher": { + "access_key": "{{ cronjob_publisher_access_key }}", + "secret_key": "{{ cronjob_publisher_secret_key }}", + "tenancy": "{{ cronjob_publisher_tenancy }}", + "tape_url": "{{ cronjob_publisher_tape_url }}" } } From 6648b80d035704938c0adf916cee180a458f93c7 Mon Sep 17 00:00:00 2001 From: Jack Leland Date: Tue, 12 Mar 2024 13:14:26 +0000 Subject: [PATCH 15/21] Fix formatting problems in server-config template --- nlds/templates/server_config.j2 | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/nlds/templates/server_config.j2 b/nlds/templates/server_config.j2 index b7832b88..98896f04 100644 --- a/nlds/templates/server_config.j2 +++ b/nlds/templates/server_config.j2 @@ -73,8 +73,8 @@ "db_passwd" : "{{ catalog_q_db_options_db_passwd }}", "echo": "{{ catalog_q_db_options_echo }}" }, - default_tenancy: "{{ catalog_q_default_tenancy }}", - default_tape_url: "{{ catalog_q_default_tape_url }}" + "default_tenancy": "{{ catalog_q_default_tenancy }}", + "default_tape_url": "{{ catalog_q_default_tape_url }}" }, "monitor_q": { "logging": "{{ monitor_q_logging_dict }}", @@ -139,7 +139,7 @@ "tape_url": "{{ archive_get_q_tape_url }}", "tape_pool": "{{ archive_get_q_tape_pool }}", "query_checksum_fl": "{{ archive_get_q_query_checksum_fl }}", - "chunk_size": "{{ archive_get_q_chunk_size }}" + "chunk_size": "{{ archive_get_q_chunk_size }}", "prepare_requeue": "{{ archive_get_q_prepare_requeue }}" }, "rpc_publisher": { From e57038f5dda1853c41277a60d06928379030aa67 Mon Sep 17 00:00:00 2001 From: Jack Leland Date: Tue, 12 Mar 2024 13:54:57 +0000 Subject: [PATCH 16/21] Add example of config for a distributed deployment --- docs/source/server-config/examples.rst | 112 ++++++++++++++++++++++++- 1 file changed, 111 insertions(+), 1 deletion(-) diff --git a/docs/source/server-config/examples.rst b/docs/source/server-config/examples.rst index 3196a774..9af9b62e 100644 --- a/docs/source/server-config/examples.rst +++ b/docs/source/server-config/examples.rst @@ -301,4 +301,114 @@ purposes. Distributed NLDS ---------------- -COMING SOON \ No newline at end of file +When making the config for a distributed NLDS, the above would need to be split +into the appropriate sections for each of the distributed parts being run +separately, namely by the consumer-specific and publisher-specific sections. +Each consumer needs the core, required ``authentication`` and ``rabbitMQ``, +optionally ``logging`` or ``general`` config and then whatever consumer-specific +values necessary to change from default values. + +The following is a breakdown of how it might be achieved: + +API-Server +^^^^^^^^^^ + +This would only contain the required sections as well as, optionally, any config +for the ``rpc_publisher``:: + + { + "authentication": { + "authenticator_backend": "jasmin_authenticator", + "jasmin_authenticator": { + "user_profile_url" : "[REDACTED]", + "user_services_url" : "[REDACTED]", + "oauth_token_introspect_url" : "[REDACTED]" + } + }, + "rabbitMQ": { + "user": "[REDACTED]", + "password": "[REDACTED]", + "heartbeat": 5, + "server": "[REDACTED]", + "vhost": "nlds_staging", + "admin_port": 15672, + "exchange": { + "name": "nlds", + "type": "topic", + "delayed": true + }, + "queues": [] + }, + "rpc_publisher": { + "time_limit": 60 + } + } + +NLDS Worker +^^^^^^^^^^^ + +This, again, contains the required sections, as well as consumer specific config +for the NLDS-Worker. In this case the additional info would be enabling the +logging at ``debug`` level and defining the bindings (routing keys) for the +consumer's queue. + +.. code-block:: json + + { + "authentication": { + "authenticator_backend": "jasmin_authenticator", + "jasmin_authenticator": { + "user_profile_url" : "[REDACTED]", + "user_services_url" : "[REDACTED]", + "oauth_token_introspect_url" : "[REDACTED]" + } + }, + "rabbitMQ": { + "user": "[REDACTED]", + "password": "[REDACTED]", + "heartbeat": 5, + "server": "[REDACTED]", + "vhost": "nlds_staging", + "admin_port": 15672, + "exchange": { + "name": "nlds", + "type": "topic", + "delayed": true + }, + "queues": [ + { + "name": "nlds_q", + "bindings": [ + { + "exchange": "nlds", + "routing_key": "nlds-api.route.*" + }, + { + "exchange": "nlds", + "routing_key": "nlds-api.*.complete" + }, + { + "exchange": "nlds", + "routing_key": "nlds-api.*.failed" + } + ] + } + ] + }, + "logging": { + "log_level": "debug" + }, + "nlds_q": { + "logging": { + "enable": true + } + } + } + +Every other consumer would be populated similarly. + +.. note:: + In the production deployment of NLDS, this is practically achieved through + ``helm`` and the combination of different yaml config files. Please see the + :doc:`../deployment` documentation for more details on the practicalities of + deploying the NLDS. From 0815788af8177fc42281ecf2c1a961ca01a9a38c Mon Sep 17 00:00:00 2001 From: Jack Leland Date: Tue, 12 Mar 2024 15:40:57 +0000 Subject: [PATCH 17/21] Fix unit tests breaking with changed template --- tests/conftest.py | 7 +++--- tests/nlds/rabbit/test_publisher.py | 16 +++++++++----- tests/req-config-template.json | 34 +++++++++++++++++++++++++++++ tests/server-config.json | 1 - 4 files changed, 49 insertions(+), 9 deletions(-) create mode 100644 tests/req-config-template.json delete mode 120000 tests/server-config.json diff --git a/tests/conftest.py b/tests/conftest.py index 4ded5987..08fe250d 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -11,13 +11,14 @@ TEMPLATE_CONFIG_PATH = os.path.join(os.path.dirname(__file__), - 'server-config.json') + 'req-config-template.json') @pytest.fixture def template_config(): config_path = TEMPLATE_CONFIG_PATH - fh = open(config_path) - return json.load(fh) + with open(config_path) as fh: + config_dict = json.load(fh) + return config_dict @pytest.fixture def test_uuid(): diff --git a/tests/nlds/rabbit/test_publisher.py b/tests/nlds/rabbit/test_publisher.py index 46ded5bb..0aa31ff8 100644 --- a/tests/nlds/rabbit/test_publisher.py +++ b/tests/nlds/rabbit/test_publisher.py @@ -7,9 +7,12 @@ from nlds.rabbit import publisher as publ from nlds.rabbit.publisher import RabbitMQPublisher -from nlds.server_config import LOGGING_CONFIG_ENABLE, LOGGING_CONFIG_FORMAT, \ - LOGGING_CONFIG_LEVEL, LOGGING_CONFIG_SECTION, \ - LOGGING_CONFIG_STDOUT, LOGGING_CONFIG_STDOUT_LEVEL +from nlds.server_config import ( + LOGGING_CONFIG_FORMAT, + LOGGING_CONFIG_LEVEL, + LOGGING_CONFIG_STDOUT, + LOGGING_CONFIG_STDOUT_LEVEL, +) def mock_load_config(template_config): return template_config @@ -62,8 +65,11 @@ def test_publish_message(default_publisher): # Attempting to establish a connection with the template config should also # fail with a socket error - with pytest.raises(gaierror): - default_publisher.get_connection() + # NOTE: (2024-03-12) Commented this out as the new daemon thread logic and + # perma-retries keep the connection from dying. No longer a useful test. + # TODO: rewrite but force close it? + # with pytest.raises(gaierror): + # default_publisher.get_connection() # TODO: Make mock connection object and send messages through it? diff --git a/tests/req-config-template.json b/tests/req-config-template.json new file mode 100644 index 00000000..d6219ee8 --- /dev/null +++ b/tests/req-config-template.json @@ -0,0 +1,34 @@ +{ + "authentication" : { + "authenticator_backend" : "jasmin_authenticator", + "jasmin_authenticator" : { + "user_profile_url" : "{{ user_profile_url }}", + "user_services_url" : "{{ user_services_url }}", + "oauth_token_introspect_url" : "{{ token_introspect_url }}" + } + }, + "rabbitMQ": { + "user": "{{ rabbit_user }}", + "password": "{{ rabbit_password }}", + "heartbeat": "{{ rabbit_heartbeat }}", + "server": "{{ rabbit_server }}", + "admin_port": "{{ rabbit_port }}", + "vhost": "{{ rabbit_vhost }}", + "exchange": { + "name": "{{ rabbit_exchange_name }}", + "type": "{{ rabbit_exchange_type }}", + "delayed": "{{ rabbit_exchange_delayed }}" + }, + "queues": [ + { + "name": "{{ rabbit_queue_name }}", + "bindings": [ + { + "exchange": "{{ rabbit_exchange_name }}", + "routing_key": "{{ rabbit_queue_routing_key }}" + } + ] + } + ] + } +} \ No newline at end of file diff --git a/tests/server-config.json b/tests/server-config.json deleted file mode 120000 index f04e096c..00000000 --- a/tests/server-config.json +++ /dev/null @@ -1 +0,0 @@ -../nlds/templates/server_config.j2 \ No newline at end of file From e0337ab2d0878f9d7139004bc1ee41bc0d864ff2 Mon Sep 17 00:00:00 2001 From: Jack Leland Date: Tue, 12 Mar 2024 16:13:43 +0000 Subject: [PATCH 18/21] Fix permission on coverage report --- .github/workflows/ci.yml | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 45c3811e..11a88556 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -109,7 +109,10 @@ jobs: uses: ammaraskar/sphinx-action@master with: docs-folder: "docs/" - pre-build-command: "mkdir -p docs/build/html/coverage/htmlcov && chmod -R 777 docs/build/html && chown 1001 docs/build/html/coverage" + pre-build-command: | + "mkdir -p docs/build/html/coverage/htmlcov && chmod -R 777 docs/build/html && chmod -R 777 docs/build/html/cover/htmlcov && chown 1001 docs/build/html/coverage" + - name: "Look at directory structure" + run: ls docs/build/html/coverage/htmlcov - name: "Download coverage report" uses: actions/download-artifact@v3 with: From 80fafaad1266676911afdcdae887639a43675252 Mon Sep 17 00:00:00 2001 From: Jack Leland Date: Tue, 12 Mar 2024 16:18:14 +0000 Subject: [PATCH 19/21] Use sudo to change permissions of build dir in gh-action --- .github/workflows/ci.yml | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 11a88556..2a2a3219 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -112,7 +112,9 @@ jobs: pre-build-command: | "mkdir -p docs/build/html/coverage/htmlcov && chmod -R 777 docs/build/html && chmod -R 777 docs/build/html/cover/htmlcov && chown 1001 docs/build/html/coverage" - name: "Look at directory structure" - run: ls docs/build/html/coverage/htmlcov + run: | + ls -la docs/build/html/coverage/htmlcov + sudo chmod -R 777 docs/build/html/coverage - name: "Download coverage report" uses: actions/download-artifact@v3 with: From ff112e465af0f0dcefd3aa2da48a0b514167f074 Mon Sep 17 00:00:00 2001 From: Jack Leland Date: Tue, 12 Mar 2024 16:30:54 +0000 Subject: [PATCH 20/21] Chmod more directories --- .github/workflows/ci.yml | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 2a2a3219..687d771b 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -111,10 +111,12 @@ jobs: docs-folder: "docs/" pre-build-command: | "mkdir -p docs/build/html/coverage/htmlcov && chmod -R 777 docs/build/html && chmod -R 777 docs/build/html/cover/htmlcov && chown 1001 docs/build/html/coverage" - - name: "Look at directory structure" + - name: "Chmod directory structure" run: | + sudo chmod -R 777 docs/build/html + # sudo chmod -R 777 docs/build/html/coverage + # sudo chmod -R 777 docs/build/html/coverage/htmlcov ls -la docs/build/html/coverage/htmlcov - sudo chmod -R 777 docs/build/html/coverage - name: "Download coverage report" uses: actions/download-artifact@v3 with: @@ -125,6 +127,9 @@ jobs: with: name: "covbadge" path: "docs/build/html" + - name: "List html dir" + run: | + ls -la docs/html/html/coverage/htmlcov - name: Upload artifacts uses: actions/upload-artifact@v3 with: From 7def6c092a2d4460e275d0aa6a264b3b4ef1dfc1 Mon Sep 17 00:00:00 2001 From: Jack Leland Date: Tue, 12 Mar 2024 16:34:05 +0000 Subject: [PATCH 21/21] Fix typo in gh-actions --- .github/workflows/ci.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 687d771b..f83b0396 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -129,7 +129,7 @@ jobs: path: "docs/build/html" - name: "List html dir" run: | - ls -la docs/html/html/coverage/htmlcov + ls -la docs/build/html/coverage/htmlcov - name: Upload artifacts uses: actions/upload-artifact@v3 with: