Merge pull request #214 from ARGOeu/devel

Version 2.0.0

CHANGELOG.md

@@ -0,0 +1,43 @@
+# Changelog
+
+## [2.0.0] - 2022-02-10
+
+* ARGO-3375 Pass contacts for each topology entity
+* ARGO-3427 Fetch HOSTDN service attribute from GOCDB
+* ARGO-3428 Fetch SE_PATH from BDII
+* ARGO-3448 NEANIAS dulpicates in group topology
+* ARGO-3503 Support multiple VO SE_PATHs for host
+* ARGO-3521 Retry on empty responses
+* ARGO-3522 Introduce optional scope key for topology fetch and BIOMED feed parse
+* ARGO-3524 Improve exception handling
+* ARGO-3528 Verbose retry log messages
+* ARGO-3540 Tests for retry logic
+
+### Added
+
+* ARGO-2620 Update connectors configuration templates
+* ARGO-2622 Populate group_type field for WEB-API weights
+* ARGO-3064 Introduce async topology CSV
+* ARGO-3215 Pass EXTENSIONS bucket in new connectors
+* ARGO-3295 Add GOCDB URLs
+* ARGO-3301 Query BDII and extract SRM port
+* ARGO-3335 Retry on LDAP queries
+* ARGO-3340 Unit testing of parsing of GOCDB service endpoint
+* ARGO-3341 Pass date argument to WEB-API methods
+
+### Fixed
+
+* ARGO-2621 Detect missing path separator for tenant directory avros
+* ARGO-2660 Basic HTTP authentication not correctly set up
+* ARGO-2681 Weights use HTTP PUT to write new daily entries
+* ARGO-2771 Weight connector issues investigated
+* ARGO-3268 Downtimes DELETE with explicit date passed
+
+### Changed
+
+* ARGO-2591 Connectors CentOS7 and Py3 migrate
+* ARGO-2596 Push sync_data to WEB-API
+* ARGO-2619 Remove Centos 6 RPM building
+* ARGO-2656 Downtimes as global tenant resource
+* ARGO-2860 Separate connection, parsing and file handling
+* ARGO-2861 Switch to async IO operations

Jenkinsfile

@@ -0,0 +1,77 @@
+pipeline {
+    agent any
+    options {
+        checkoutToSubdirectory('argo-egi-connectors')
+    }
+    environment {
+        PROJECT_DIR="argo-egi-connectors"
+        GIT_COMMIT=sh(script: "cd ${WORKSPACE}/$PROJECT_DIR && git log -1 --format=\"%H\"",returnStdout: true).trim()
+        GIT_COMMIT_HASH=sh(script: "cd ${WORKSPACE}/$PROJECT_DIR && git log -1 --format=\"%H\" | cut -c1-7",returnStdout: true).trim()
+        GIT_COMMIT_DATE=sh(script: "date -d \"\$(cd ${WORKSPACE}/$PROJECT_DIR && git show -s --format=%ci ${GIT_COMMIT_HASH})\" \"+%Y%m%d%H%M%S\"",returnStdout: true).trim()
+    }
+    stages {
+        stage ('Build'){
+            parallel {
+                stage ('Build Centos 7 RPM') {
+                    agent {
+                        docker {
+                            image 'argo.registry:5000/epel-7-ams'
+                            args '-u jenkins:jenkins'
+                        }
+                    }
+                    steps {
+                        echo 'Building Rpm...'
+                        withCredentials(bindings: [sshUserPrivateKey(credentialsId: 'jenkins-rpm-repo', usernameVariable: 'REPOUSER', \
+                                                                    keyFileVariable: 'REPOKEY')]) {
+                            sh "/home/jenkins/build-rpm.sh -w ${WORKSPACE} -b ${BRANCH_NAME} -d centos7 -p ${PROJECT_DIR} -s ${REPOKEY}"
+                        }
+                        archiveArtifacts artifacts: '**/*.rpm', fingerprint: true
+                    }
+                    post {
+                        always {
+                            cleanWs()
+                        }
+                    }
+                }
+                stage ('Execute tests') {
+                    agent {
+                        docker {
+                            image 'argo.registry:5000/epel-7-ams'
+                            args '-u jenkins:jenkins'
+                        }
+                    }
+                    steps {
+                        sh '''
+                            cd $WORKSPACE/$PROJECT_DIR/
+                            rm -f tests/argo_egi_connectors
+                            ln -s $PWD/modules/ tests/argo_egi_connectors
+                            coverage run -m xmlrunner discover --output-file junit.xml -v tests/
+                            coverage xml
+                        '''
+                        cobertura coberturaReportFile: 'coverage.xml'
+                        junit 'junit.xml'
+                    }
+                }
+            }
+        }
+    }
+    post {
+        always {
+            cleanWs()
+        }
+        success {
+            script{
+                if ( env.BRANCH_NAME == 'master' || env.BRANCH_NAME == 'devel' ) {
+                    slackSend( message: ":rocket: New version for <$BUILD_URL|$PROJECT_DIR>:$BRANCH_NAME Job: $JOB_NAME !")
+                }
+            }
+        }
+        failure {
+            script{
+                if ( env.BRANCH_NAME == 'master' || env.BRANCH_NAME == 'devel' ) {
+                    slackSend( message: ":rain_cloud: Build Failed for <$BUILD_URL|$PROJECT_DIR>:$BRANCH_NAME Job: $JOB_NAME")
+                }
+            }
+        }
+    }
+}

Makefile

@@ -4,14 +4,14 @@ SPECFILE=${PKGNAME}.spec
 PKGVERSION=$(shell grep -s '^Version:' $(SPECFILE) | sed -e 's/Version: *//')
 
 srpm: dist
-	rpmbuild -ts --define='dist .el6' ${PKGNAME}-${PKGVERSION}.tar.gz
+	rpmbuild -ts --define='dist .el7' ${PKGNAME}-${PKGVERSION}.tar.gz
 
 rpm: dist
 	rpmbuild -ta ${PKGNAME}-${PKGVERSION}.tar.gz
 
 dist:
 	rm -rf dist
-	python setup.py sdist
+	python3 setup.py sdist
 	mv dist/${PKGNAME}-${PKGVERSION}.tar.gz .
 	rm -rf dist
 

README.md

@@ -1,9 +1,21 @@
-**argo-egi-connectors** is a bundle of connectors/sync components for various data sources established in EGI infrastructure, most notably GOCDB (topology, downtimes), but there's also support for alternative topology fetching for virtual organizations, weights factors provided within a VAPOR service and POEM profiles information.
+`argo-connectors` is bundle of data fetch components that gather insights in
+various e-infrastructures for the ARGO monitoring purposes.
 
-Data is synced on a daily basis and all together with a prefiltered raw metric results coming from _argo-egi-consumer_ represents an input for _argo-compute-engine_. Data is written in binary avro formatted files which are grouped into a directory structure that represents set of customers and jobs at compute side.
+It primarily deals with hierarchical topology from GOCDB compatible sources,
+but also flat topologies defined in simple CSV or JSON format are supported.
+Topology lists endpoints with asociated services running of them that needs to
+be inspected as well as contact informations that will be used for alerts. In
+case of hierarchical topology, endpoints are organized in sites or
+servicegroups which themselves are organized in NGIs or projects.
 
-Configuration of connectors is centered around two configuration files:
-- `global.conf` - used for defining some common information like connection and authentication parameters, avro schemas and output filenames
-- `customer.conf` - used for listing jobs for EGI customer and defining new customers and theirs jobs residing in EGI infrastructure. Customers and jobs have attributes that control the behaviour of related connector and also create implicit directory structure. 
+Next to topology, `argo-connectors` also gather weights and downtimes data.
+Weights are Computation Power figures fetched from VAPOR service for each SITE
+that are used for A/R calculation of interested NGI. Downtimes data are time
+period scheduled for each endpoint under maintenance that will not be taken
+into account during A/R calculation.
+
+All collected data is transformed in appropriate format and pushed on daily
+basis on the corresponding API method of ARGO-WEB-API service.
 
 More info: http://argoeu.github.io/guides/sync/
+