Challenge #16 - ECMWF User Profile Page

[EsperanzaCuartero]

Challenge 16- ECMWF User Profile Page

Stream 1 - Software development for weather, climate and atmosphere

Goal

Provide users with their own profile web page showing a summary of their relationship and records with ECMWF. Links from this web page will provide an improved user journey to the services and systems where those relationships and records are held and can be updated (if relevant, and only in some cases).

Mentors and skills

• Mentors: @sylvielamythepaut @kiden
• Skills required:
  • This project would suit a person who can analyse how to extract information from a broad range of technologies and platforms in order to present a 'mash-up' of user profile information from various platforms onto a single custom web page.
  • Competent in html, css, javascript (library?), apis, information security. Knowledge of ForgeRock may be useful.

---

Challenge description

Users currently have no overview of their profile with us; in an age of GDPR/PIIP this isn't very transparent of us. When we investigate issues we often need to look at a user's records in several systems to discover if the problem lies in an incorrect record of their relationship with us, e.g. wrong policy, wrong name, missing from a mailing list, lost data; this would provide an overview of a user in one place across all these systems.

The vision is a web page summary of the following user information:

• Registration information (name, email, pwd alert if near to needing changing, etc) - data in AD; link to forgerock
• Data from and links to profile pages on other platforms
• Favourite charts (link to chart dashboard or mars api requests - resurface what's shown in apps recent or recurring data requests
• Their current agreements and licences (from where?)
• Mailing lists (from csv files)
• Policies / roles (from AD)
• Jobs they've applied for (can we get this from JobTrain?)
• sbu units used (how can we get a summary?)
• Events they have attended (accessible in json over API)
• Learning modules they're doing or have completed (link to learning portal)
• Atlassian activity (likes, forum posts, page edits) - confluence API
• DHS usage (link to https://dlm.ecmwf.int/ )

Each of these summaries should then link to the system or platform where the information comes from and where more detail is held and can be updated (if/when these exist).

[jwagemann]

Hi,
join us for the ECMWF Summer of Weather Code Ask Me Anything session and learn all things ESoWC.

When: Wednesday, 24 March 2021 at 4 pm GMT

What: learn everything about ESoWC - how it works, the challenges this year, some tips for your proposal and listen to ESoWC experiences from previous participants

How: register here.

[varunbankar]

Hello mentors,

I am Varun Bankar, an undergraduate student pursuing Computer Science at BITS Pilani, India. I was looking for opportunities to work during summers and a friend of mine referred me to ESoWC. He was a participant in ESoWC 2020 and described it as a wonderful learning experience. My background is mostly in software development and was looking through the challenges for ESoWC 2021 and this challenge interested me. It aligns more with my skill set. I got good experience building web dashboards as part of my Google Summer of Code project last year for CERN.

From the given problem statement, I understand that this project would involve scrapping / extracting user data from various sources and aggregating it in one place which will be displayed on a web page as User Profile. I have few following questions regarding this project.

• Are the users registered already somewhere within some service of ECMWF? I see ForgeRock being mentioned.
• Will I need to scrape the user data or there will be existing endpoints or database access for accessing the data?
• Will the data we need to display about the user require authorization or will it be openly accessible to everyone?
• Does this project expect to use a specific framework (Flask, NodeJS, or some other)?

I am looking forward to discussing this challenge in detail with you. Sorry if I am a little late for applying for ESoWC, I was referred to this by my friend yesterday. I am very much interested in applying for this project and working on it during my summers.

Thank you.

You can reach me through my GitHub or Email at [email protected]

[kiden]

Thanks for your interest! What you describe is correct: the project involves data scraping, mashing it back up, and displaying it nicely in a dashboard.
In response to your bullets

• Yes, we have a user database. We haven't yet implemented ForgeRock so we would likely rely on data from Active Directory (AD)
• It's a mix. Some systems have APIs, but I anticipate that we might have to create some automated data dumps (export files) from some systems, and we could work with those for ESoWC. Eventually we could replace those with API endpoints or similar in the future (and that could be something we do internally, outside the scope of ESoWC so you aren't waiting for us to set them up). I imagine we will need to spend a bit of time investigating all the data sources.
• Some of it will require authorization which we will sort out; using export files will be a way of getting up & running quickly. The key thing will be to make sure that the data being retrieved to populate the interface only exposes the logged-in user's information (i.e. we wouldn't want to write an interface which pulled back all users' data to the browser and then filtered it to the current user)
• Re the framework, I'll ask my colleague @sylvielamythepaut to provide a reply on that

[varunbankar]

@kiden Thank you for your reply.

So, this project should be an independent service, which will:

• collect user data from the accessible sources (eg. AD database)
• scrape it from certain web services
• extract it through internal systems with custom scripts

and then create a nice user profile with some part of the data only accessible to the user after logging-in (User Dashboard?) and some part visible to the public as a User Page.

---

Questions Regarding Data Collecting

Data from and links to profile pages on other platforms

• Can I get an idea about what kind of data and from what other platforms?

Favourite charts (link to chart dashboard or mars api requests - resurface what's shown in apps recent or recurring data requests

• Can I get some reference to this?

Their current agreements and licences (from where?)

• Will the user themselves add this or we have a database of it?

Jobs they've applied for (can we get this from JobTrain?)

• I couldn't find an API provided by JobTrain, and we might need permission from them if we are to scrape data from their site. Other than that users can input this themselves through their dashboard.

sbu units used (how can we get a summary?)

• Can I also get more information on this?

Learning modules they're doing or have completed (link to learning portal)

• Is there a certain learning portal every user uses?

---

• Also, will this service be hosted on some specific cloud platform?

I have good experience with Python as well as JavaScript / TypeScript, and we can certainly choose a good framework in these languages for the backend.

Thank you for your time.

PS: Is this thread correct place to ask these questions or is there some IRC channel I should join?

[kiden]

Hi again
This channel is good for these questions so we can share answers with anyone else looking at this challenge.

and some part visible to the public as a User Page
I don't imagine there being a public view; the page only becomes accessible to logged in users,

other platforms
Good question - the list provided gives an idea but I'm forever thinking of others, so I think this would be something we would work out thoroughly (and decide which ones we could feasibly include) during the project.

user's chart dashboard
this is currently only available to privileged users but, if successful in your bid, we would give you access and then investigate how to integrate this (maybe a separate tab on the user profile page)

licenses they have
we have some of this information in a db - we would need to investigate what is usable/exportable during the project

jobs they've applied for
we can provide export files - it's a new system for ECMWF (not yet live) so again a bit of investigation is needed

sbu units
A full dashboard is in development right now, so we could see what, if anything, we can extract/reuse to create a jumping off point in the user profile/dashboard

learning modules
If you click through from https://www.ecmwf.int/en/learning/education-material/elearning-online-resources you will find the learning management system we use

As you can see, the project will involved some analysis of available data. That said, I would like to approach it from a user/design first approach - what would be useful to the user? and then: how can we get that data for them?

[kiden]

re hosting, you will be given hosting on our European Weather Cloud. I'm afraid I don't know much about that but if there's anything specific you need to know then I can find out or loop in one of my colleagues.

[varunbankar]

@kiden Yes I agree that the approach should be from user's perspective. Can you help me get an idea of what the general profile a user is like, and what all data points will matter the most to them?

Also, I did go through the ESoWC website, FAQ section and also through the YouTube channel. Even saw my friend's presentation! I wanted to ask that are there any tasks or something that I need to complete or pre-proposal work to be eligible for applying to the programme? Also, what are the things that expected in the proposal such that it is clear and precise?

[kiden]

@varunbankar I would say the most interesting thing to users is their charts and data, as well as understanding what access they have or should have (i.e. licenses, policy groups they belong to).

I don't think you have to do any pre-proposal work - we select based on the proposal alone. @jwagemann might be able to say more...

[varunbankar]

@kiden taking all of the points from our discussion under consideration I'll will start working on a draft proposal. Can you and @sylvielamythepaut review my draft proposal so that we can address certain points / vision that you both might have for this project. This way I can refine my approach which suit more towards ECMWF style and I can clearly state my objectives, timelines and deliverables for the project.

@sylvielamythepaut it would be of great help if you could shed some light on the framework that we should use for this project.

Thank you :)

[kiden]

@varunbankar happy to answer questions here but we cannot contribute to individual proposals as it wouldn't be fair to others thinking of applying - I hope you understand. If it's a successful proposal we'll be happy to refine it further if needed before work starts :)

[sylvielamythepaut]

@varunbankar,
Sorry I missed your message.
I will come back to you tomorrow or Monday at the latest with some information on the framework we are envisaging.
Have a nice evening,

[varunbankar]

@kiden, I understand. If I have any further queries, I'll be sure to ask them here.

@sylvielamythepaut I will be waiting for your reply, as the decision of framework will vastly affect the types of libraries used in the project.

Thank you both for taking out your time to clarify my doubts!

[varunbankar]

@kiden I have my college end semester examination from May 1st to May 15th during which I might not be available as much, and ESoWC coding phase starts for May 4th. Could you clarify if this will be okay?

@sylvielamythepaut Is there any update regarding the framework to be used?

Thanks!

[sylvielamythepaut]

Hi @varunbankar,

The aim of this challenge is to be able to harvest the user information from the various web applications ECMWF is offering, and presents them all in a single dashboard.

The ECMWF web framework is a micro-services oriented framework deployed on a Kubernetes Cluster. All backend components are written in Python and make use of Django as a web framework. All our front-end applications are written in REACT and are created from a template to ensure a consistent look and feel, and yarn is used as a package manager.

As this project is looking for innovative ideas, we are open to any suggestion.

Hoping that this will help you with your proposal.

Sylvie

[varunbankar]

Thank you, this helps a lot!

[kiden]

@varunbankar with regards to your availability during exams we can work around things like that - you can include this information in an outline work timetable in your proposal.

[varunbankar]

@kiden Is there a common authentication system for users to use all of the ECMWF services, or different services may have different ways for users to authenticate?

If yes, then does AD contains the data through which we can identify users across different services?