Skip to content

Commit

Permalink
Merge pull request #1625 from elimu-ai/add-letter-sounds
Browse files Browse the repository at this point in the history
feat: Add letter-sound learning events
  • Loading branch information
nya-elimu authored Oct 30, 2023
2 parents a8be360 + 110bfbf commit 6e81204
Show file tree
Hide file tree
Showing 53 changed files with 768 additions and 691 deletions.
46 changes: 23 additions & 23 deletions LOCALIZE.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,30 +23,30 @@ Follow these steps to add a new audio file to the webapp's database:

1. Press the "Add audio" button.

<img width="208" alt="Screen Shot 2020-12-09 at 4 14 45 PM" src="https://user-images.githubusercontent.com/15718174/101648131-310b2a80-3a42-11eb-95f1-0eed0f183bd2.png">
<img width="208" src="https://user-images.githubusercontent.com/15718174/101648131-310b2a80-3a42-11eb-95f1-0eed0f183bd2.png">

1. If the audio recording is a word, the title should match the transcription:

<img width="273" alt="Screen Shot 2020-12-09 at 4 20 17 PM" src="https://user-images.githubusercontent.com/15718174/101648783-d8885d00-3a42-11eb-840a-8c97c1d592c0.png">
<img width="273" src="https://user-images.githubusercontent.com/15718174/101648783-d8885d00-3a42-11eb-840a-8c97c1d592c0.png">

1. However, if the audio recording is longer than a single word (e.g. a storybook paragraph), the title should be something other than the transcription:

<img width="465" alt="Screen Shot 2020-12-09 at 4 22 29 PM" src="https://user-images.githubusercontent.com/15718174/101649083-2a30e780-3a43-11eb-943c-b6d4229f5023.png">
<img width="465" src="https://user-images.githubusercontent.com/15718174/101649083-2a30e780-3a43-11eb-943c-b6d4229f5023.png">

1. If you copied the audio file from somewhere, make sure to include the license used on the website where you downloaded the file, as well as a link to the website. If, however, you recorded the audio file yourself, _you_ can select which license to use. We recommend using [Creative Commons CC BY](https://creativecommons.org/licenses/by/4.0/):

<img width="308" alt="Screen Shot 2020-12-09 at 4 26 35 PM" src="https://user-images.githubusercontent.com/15718174/101649572-b6430f00-3a43-11eb-8324-e4afe720de8d.png">
<img width="308" src="https://user-images.githubusercontent.com/15718174/101649572-b6430f00-3a43-11eb-8324-e4afe720de8d.png">

1. Remove silence before/after the audio, if any. This is to make sure that a child does not experience any delays while interacting with the software. We recommend using [Audacity](https://www.audacityteam.org) for editing audio recordings:

<img width="1053" alt="Screen Shot 2020-12-17 at 7 10 51 PM" src="https://user-images.githubusercontent.com/15718174/102526312-27b63980-40a4-11eb-89ce-244351b449e7.png">
<img width="1053" src="https://user-images.githubusercontent.com/15718174/102526312-27b63980-40a4-11eb-89ce-244351b449e7.png">

<img width="1053" alt="Screen Shot 2020-12-17 at 7 11 14 PM" src="https://user-images.githubusercontent.com/15718174/102526322-2ab12a00-40a4-11eb-9f2a-ea85230da06a.png">
<img width="1053" src="https://user-images.githubusercontent.com/15718174/102526322-2ab12a00-40a4-11eb-9f2a-ea85230da06a.png">


1. Then select the audio file, and press the "Add" button. If you want to provide any additional details about your contribution, you can do so in the "Comment" field:

<img width="378" alt="Screen Shot 2020-12-09 at 4 32 43 PM" src="https://user-images.githubusercontent.com/15718174/101650579-c0b1d880-3a44-11eb-808a-4ffe292eef4b.png">
<img width="378" src="https://user-images.githubusercontent.com/15718174/101650579-c0b1d880-3a44-11eb-808a-4ffe292eef4b.png">

#### Adding Audio Recording via Another Page

Expand All @@ -56,25 +56,25 @@ Note that there are two other ways you can add audio recordings: 1) via the word

1. At https://xho.elimu.ai/content/word/list you can find a list of words. While editing a word, you will see a warning saying "This word has no corresponding audio." if an audio recording with a matching transcription does not already exist:

<img width="641" alt="Screen Shot 2020-12-09 at 4 44 43 PM" src="https://user-images.githubusercontent.com/15718174/101652036-69ad0300-3a46-11eb-8d77-2efe899a6c46.png">
<img width="641" src="https://user-images.githubusercontent.com/15718174/101652036-69ad0300-3a46-11eb-8d77-2efe899a6c46.png">

1. Then, if you press the "Add audio" link, you will be redirected to the page for uploading an audio file, and the title and the transcription of the audio will be auto-filled:

<img width="210" alt="Screen Shot 2020-12-09 at 4 48 07 PM" src="https://user-images.githubusercontent.com/15718174/101652281-bc86ba80-3a46-11eb-95e1-728a277c0dc2.png">
<img width="210" src="https://user-images.githubusercontent.com/15718174/101652281-bc86ba80-3a46-11eb-95e1-728a277c0dc2.png">

##### Via the storybook paragraph edit page

1. While editing a storybook paragrah, you'll find an "Add audio" link at the bottom:

<img width="259" alt="Screen Shot 2020-12-09 at 4 54 18 PM" src="https://user-images.githubusercontent.com/15718174/101653094-a88f8880-3a47-11eb-9c7f-5612e3176c4b.png">
<img width="259" src="https://user-images.githubusercontent.com/15718174/101653094-a88f8880-3a47-11eb-9c7f-5612e3176c4b.png">

1. If you press the "Add audio" link, you will be redirected to the page for uploading an audio file, and the title and the transcription of the audio will be auto-filled to match the content of the paragraph:

<img width="558" alt="Screen Shot 2020-12-09 at 4 56 18 PM" src="https://user-images.githubusercontent.com/15718174/101653310-dc6aae00-3a47-11eb-9296-91856d152ce9.png">
<img width="558" src="https://user-images.githubusercontent.com/15718174/101653310-dc6aae00-3a47-11eb-9296-91856d152ce9.png">

1. If you add an audio recording this way, remember to go back to the paragraph edit page after uploading the audio file, and select the corresponding audio in the drop-down:

<img width="691" alt="Screen Shot 2020-12-09 at 4 58 48 PM" src="https://user-images.githubusercontent.com/15718174/101653932-96621a00-3a48-11eb-82e3-97598fdac82a.png">
<img width="691" src="https://user-images.githubusercontent.com/15718174/101653932-96621a00-3a48-11eb-82e3-97598fdac82a.png">

### Adding Words

Expand All @@ -84,27 +84,27 @@ Follow these steps to add a new word to the webapp's database:

1. Press the "Add word" button:

<img width="209" alt="Screen Shot 2020-12-09 at 7 41 22 PM" src="https://user-images.githubusercontent.com/15718174/101672844-0a0f2180-3a5f-11eb-9401-6dcb9ad53df0.png">
<img width="209" src="https://user-images.githubusercontent.com/15718174/101672844-0a0f2180-3a5f-11eb-9401-6dcb9ad53df0.png">

1. Type the word's text:

<img width="217" alt="Screen Shot 2020-12-09 at 7 44 29 PM" src="https://user-images.githubusercontent.com/15718174/101673126-6b36f500-3a5f-11eb-814d-19762e2d3138.png">
<img width="217" src="https://user-images.githubusercontent.com/15718174/101673126-6b36f500-3a5f-11eb-814d-19762e2d3138.png">

1. Select the word's letter-sound correspondences:

<img width="580" alt="Screen Shot 2020-12-09 at 7 52 55 PM" src="https://user-images.githubusercontent.com/15718174/101673981-90783300-3a60-11eb-8875-cab1caafb498.png">
<img width="580" src="https://user-images.githubusercontent.com/15718174/101673981-90783300-3a60-11eb-8875-cab1caafb498.png">

If the letter-sound correspondence you want to use does not exist in the drop-down, press the "Add letter-sound correspondence" link. Then select the sound corresponding to the letter(s):

<img width="613" alt="Screen Shot 2020-12-09 at 7 50 47 PM" src="https://user-images.githubusercontent.com/15718174/101674020-9f5ee580-3a60-11eb-898f-76ff99eaa518.png">
<img width="613" src="https://user-images.githubusercontent.com/15718174/101674020-9f5ee580-3a60-11eb-898f-76ff99eaa518.png">

1. Select the _Grapheme-phoneme correspondence_, according to how well the letters match the sounds. If you are unsure about this step, leave it unselected.
1. Select the _Spelling consistency_, according to how well the letters match the sounds. If you are unsure about this step, leave it unselected.

<img width="505" alt="Screen Shot 2020-12-09 at 7 54 34 PM" src="https://user-images.githubusercontent.com/15718174/101674210-dfbe6380-3a60-11eb-8d78-fc3afccf43cf.png">
<img width="505" src="https://user-images.githubusercontent.com/15718174/101674210-dfbe6380-3a60-11eb-8d78-fc3afccf43cf.png">

1. Select the word type (adjective, adverb, noun, etc) if it exists in the drop-down. Then press the "Add" button:

<img width="695" alt="Screen Shot 2020-12-09 at 7 56 42 PM" src="https://user-images.githubusercontent.com/15718174/101674457-31ff8480-3a61-11eb-815e-a28e085517f1.png">
<img width="695" src="https://user-images.githubusercontent.com/15718174/101674457-31ff8480-3a61-11eb-815e-a28e085517f1.png">

#### Adding Word via Another Page

Expand All @@ -114,24 +114,24 @@ One strategy is to add words used in a particular storybook:

1. In the sidebar you'll see a word frequency list:

<img width="337" alt="Screen Shot 2020-12-09 at 8 12 28 PM" src="https://user-images.githubusercontent.com/15718174/101676027-447abd80-3a63-11eb-8437-f391caeaabe6.png">
<img width="337" src="https://user-images.githubusercontent.com/15718174/101676027-447abd80-3a63-11eb-8437-f391caeaabe6.png">

1. Then, press the "Add word" link if you want to create it.

1. You will then be redirected for the page for adding a word, with the word's text auto-filled:

<img width="392" alt="Screen Shot 2020-12-09 at 8 13 56 PM" src="https://user-images.githubusercontent.com/15718174/101676218-8dcb0d00-3a63-11eb-881d-f204ccecec29.png">
<img width="392" src="https://user-images.githubusercontent.com/15718174/101676218-8dcb0d00-3a63-11eb-881d-f204ccecec29.png">

1. Then, proceed as described above in ["Adding Words"](#adding-words).

Another strategy for adding words is to go to the "Words Pending" page, where you can find a list of the most used words across _all storybooks_:

1. Go to https://xho.elimu.ai/content/word/pending

<img width="1025" alt="Screen Shot 2020-12-09 at 8 10 35 PM" src="https://user-images.githubusercontent.com/15718174/101676596-0af68200-3a64-11eb-9e2c-5087e1c8f6d9.png">
<img width="1025" src="https://user-images.githubusercontent.com/15718174/101676596-0af68200-3a64-11eb-9e2c-5087e1c8f6d9.png">

1. Here you can also choose the word to add by pressing the "Add word" link, which will redirect you to the page for adding a new word (with its text auto-filled):

<img width="329" alt="Screen Shot 2020-12-09 at 8 20 00 PM" src="https://user-images.githubusercontent.com/15718174/101676791-527d0e00-3a64-11eb-9937-a8299f4cd7e8.png">
<img width="329" src="https://user-images.githubusercontent.com/15718174/101676791-527d0e00-3a64-11eb-9937-a8299f4cd7e8.png">

1. Then, proceed as described above in ["Adding Words"](#adding-words).
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@

import org.springframework.dao.DataAccessException;

public interface LetterSoundCorrespondenceDao extends GenericDao<LetterSoundCorrespondence> {
public interface LetterSoundDao extends GenericDao<LetterSoundCorrespondence> {

LetterSoundCorrespondence read(List<Letter> letters, List<Sound> sounds) throws DataAccessException;

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,9 +6,9 @@
import java.util.List;
import java.util.stream.Collectors;
import org.springframework.dao.DataAccessException;
import ai.elimu.dao.LetterSoundCorrespondenceDao;
import ai.elimu.dao.LetterSoundDao;

public class LetterSoundCorrespondenceDaoJpa extends GenericDaoJpa<LetterSoundCorrespondence> implements LetterSoundCorrespondenceDao {
public class LetterSoundDaoJpa extends GenericDaoJpa<LetterSoundCorrespondence> implements LetterSoundDao {

@Override
public LetterSoundCorrespondence read(List<Letter> letters, List<Sound> sounds) throws DataAccessException {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -4,12 +4,12 @@
import ai.elimu.model.content.LetterSoundCorrespondence;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.core.convert.converter.Converter;
import ai.elimu.dao.LetterSoundCorrespondenceDao;
import ai.elimu.dao.LetterSoundDao;

public class StringToLetterSoundCorrespondenceConverter implements Converter<String, LetterSoundCorrespondence> {

@Autowired
private LetterSoundCorrespondenceDao letterSoundCorrespondenceDao;
private LetterSoundDao letterSoundDao;

/**
* Convert LetterSoundCorrespondence id to LetterSoundCorrespondence entity
Expand All @@ -19,7 +19,7 @@ public LetterSoundCorrespondence convert(String id) {
return null;
} else {
Long letterSoundCorrespondenceId = Long.parseLong(id);
LetterSoundCorrespondence letterSoundCorrespondence = letterSoundCorrespondenceDao.read(letterSoundCorrespondenceId);
LetterSoundCorrespondence letterSoundCorrespondence = letterSoundDao.read(letterSoundCorrespondenceId);
return letterSoundCorrespondence;
}
}
Expand Down
32 changes: 16 additions & 16 deletions src/main/java/ai/elimu/rest/v2/JpaToGsonConverter.java
Original file line number Diff line number Diff line change
Expand Up @@ -88,31 +88,31 @@ public static SoundGson getSoundGson(Sound sound) {
}
}

public static LetterSoundCorrespondenceGson getLetterSoundCorrespondenceGson(LetterSoundCorrespondence letterSoundCorrespondence) {
if (letterSoundCorrespondence == null) {
public static LetterSoundCorrespondenceGson getLetterSoundGson(LetterSoundCorrespondence letterSound) {
if (letterSound == null) {
return null;
} else {
LetterSoundCorrespondenceGson letterSoundCorrespondenceGson = new LetterSoundCorrespondenceGson();
LetterSoundCorrespondenceGson letterSoundGson = new LetterSoundCorrespondenceGson();

// BaseEntity
letterSoundCorrespondenceGson.setId(letterSoundCorrespondence.getId());
letterSoundGson.setId(letterSound.getId());

// LetterSoundCorrespondence
// LetterSound
List<LetterGson> letters = new ArrayList<>();
for (Letter letter : letterSoundCorrespondence.getLetters()) {
for (Letter letter : letterSound.getLetters()) {
LetterGson letterGson = getLetterGson(letter);
letters.add(letterGson);
}
letterSoundCorrespondenceGson.setLetters(letters);
letterSoundGson.setLetters(letters);
List<SoundGson> sounds = new ArrayList<>();
for (Sound sound : letterSoundCorrespondence.getSounds()) {
for (Sound sound : letterSound.getSounds()) {
SoundGson soundGson = getSoundGson(sound);
sounds.add(soundGson);
}
letterSoundCorrespondenceGson.setSounds(sounds);
letterSoundCorrespondenceGson.setUsageCount(letterSoundCorrespondence.getUsageCount());
letterSoundGson.setSounds(sounds);
letterSoundGson.setUsageCount(letterSound.getUsageCount());

return letterSoundCorrespondenceGson;
return letterSoundGson;
}
}

Expand All @@ -131,12 +131,12 @@ public static WordGson getWordGson(Word word) {

// Word
wordGson.setText(word.getText());
List<LetterSoundCorrespondenceGson> letterSoundCorrespondences = new ArrayList<>();
for (LetterSoundCorrespondence letterSoundCorrespondence : word.getLetterSoundCorrespondences()) {
LetterSoundCorrespondenceGson letterSoundCorrespondenceGson = getLetterSoundCorrespondenceGson(letterSoundCorrespondence);
letterSoundCorrespondences.add(letterSoundCorrespondenceGson);
List<LetterSoundCorrespondenceGson> letterSounds = new ArrayList<>();
for (LetterSoundCorrespondence letterSound : word.getLetterSoundCorrespondences()) {
LetterSoundCorrespondenceGson letterSoundGson = getLetterSoundGson(letterSound);
letterSounds.add(letterSoundGson);
}
wordGson.setLetterSoundCorrespondences(letterSoundCorrespondences);
wordGson.setLetterSoundCorrespondences(letterSounds);
wordGson.setWordType(word.getWordType());

return wordGson;
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,77 @@
package ai.elimu.rest.v2.analytics;

import java.io.File;
import java.io.IOException;

import javax.servlet.http.HttpServletResponse;

import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
import org.json.JSONObject;
import org.springframework.http.HttpStatus;
import org.springframework.http.MediaType;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestMethod;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.RestController;
import org.springframework.web.multipart.MultipartFile;

import ai.elimu.model.v2.enums.Language;
import ai.elimu.util.AnalyticsHelper;
import ai.elimu.util.ConfigHelper;

/**
* REST API endpoint for receiving letter-sound learning events from the
* <a href="https://github.com/elimu-ai/analytics">Analytics</a> application.
*/
@RestController
@RequestMapping(value = "/rest/v2/analytics/letter-sound-learning-events", produces = MediaType.APPLICATION_JSON_UTF8_VALUE)
public class LetterSoundLearningEventsRestController {

private Logger logger = LogManager.getLogger();

@RequestMapping(value = "/csv", method = RequestMethod.POST)
public String handleUploadCsvRequest(
@RequestParam("file") MultipartFile multipartFile,
HttpServletResponse response
) {
logger.info("handleUploadCsvRequest");

// Expected format: "7161a85a0e4751cd_3001017_letter-sound-learning-events_2023-10-27.csv"
String originalFilename = multipartFile.getOriginalFilename();
logger.info("originalFilename: " + originalFilename);

String androidIdExtractedFromFilename = AnalyticsHelper.extractAndroidIdFromCsvFilename(originalFilename);
logger.info("androidIdExtractedFromFilename: \"" + androidIdExtractedFromFilename + "\"");

Integer versionCodeExtractedFromFilename = AnalyticsHelper.extractVersionCodeFromCsvFilename(originalFilename);
logger.info("versionCodeExtractedFromFilename: " + versionCodeExtractedFromFilename);

// Store the original CSV file on the filesystem, e.g.
// ~/.elimuai/lang-HIN/analytics/android-id-7161a85a0e4751cd/version-code-3001017/letter-sound-learning-events/7161a85a0e4751cd_3001017_letter-sound-learning-events_2023-10-27.csv"
File elimuAiDir = new File(System.getProperty("user.home"), ".elimu-ai");
File languageDir = new File(elimuAiDir, "lang-" + Language.valueOf(ConfigHelper.getProperty("content.language")));
File analyticsDir = new File(languageDir, "analytics");
File androidIdDir = new File(analyticsDir, "android-id-" + androidIdExtractedFromFilename);
File versionCodeDir = new File(androidIdDir, "version-code-" + versionCodeExtractedFromFilename);
File letterSoundLearningEventsDir = new File(versionCodeDir, "letter-sound-learning-events");
letterSoundLearningEventsDir.mkdirs();
File csvFile = new File(letterSoundLearningEventsDir, originalFilename);
logger.info("Storing CSV file at " + csvFile);

JSONObject jsonObject = new JSONObject();
try {
multipartFile.transferTo(csvFile);
jsonObject.put("result", "success");
jsonObject.put("successMessage", "The CSV file was successfully uploaded");
} catch (IllegalStateException | IOException e) {
logger.error(e);
jsonObject.put("result", "error");
jsonObject.put("errorMessage", e.getMessage());
response.setStatus(HttpStatus.INTERNAL_SERVER_ERROR.value());
}
String jsonResponse = jsonObject.toString();
logger.info("jsonResponse: " + jsonResponse);
return jsonResponse;
}
}
Loading

0 comments on commit 6e81204

Please sign in to comment.