Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add more extensive logging to the ETL process #86

Open
mauamy opened this issue Jan 21, 2021 · 0 comments
Open

Add more extensive logging to the ETL process #86

mauamy opened this issue Jan 21, 2021 · 0 comments
Assignees
Labels
backend Backend related stuff medium priority

Comments

@mauamy
Copy link
Collaborator

mauamy commented Jan 21, 2021

Reason (Why?)
We want to be able see and track what happens while the ETL process and afterwards (especially when something went wrong).

Solution (What?)
Add more extensive logging to all ETL steps (xml-import, elasticsearch-upload). The logging should be visible while execution (i.e. directly as shell output) and afterwards in a dedicated logging file. The log should include timestamps for all the steps and a summary at the end (e.g. listing how many entities where parsed (x artworks, y artists, ...) and the total time taken.

Acceptance criteria
The ETL process shows more extensive and detailed output while running and a log file is created containing the same output.

@mauamy mauamy added backend Backend related stuff medium priority labels Jan 21, 2021
@MadelineWall MadelineWall self-assigned this Jan 28, 2021
MadelineWall added a commit that referenced this issue Jan 30, 2021
Logging process to a log-file. File can find in project folder
MadelineWall added a commit that referenced this issue Jan 30, 2021
Logging etl process to a file
MadelineWall added a commit that referenced this issue Jan 30, 2021
Add timestamp when writing to logfile
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backend Backend related stuff medium priority
Projects
None yet
Development

No branches or pull requests

2 participants