Skip to content

Commit

Permalink
feat: make output folder structure configurable with OUTPUT_SUBFOLDER…
Browse files Browse the repository at this point in the history 
…_TEMPLATE env variable

Signed-off-by: Gustav Grusell <[email protected]>
  • Loading branch information
@grusell
grusell committed Sep 3, 2024
1 parent a70521e commit beb0595
Show file tree
Hide file tree
Showing 3 changed files with 30 additions and 22 deletions.
37 changes: 19 additions & 18 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -17,24 +17,25 @@ package the output of the transcoding job referenced by the message.

#### Environment variables

| Variable | Description | Default value |
|-------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------|----------------|
| `REDIS_URL` | URL to the redis server | `redis://localhost:6379` |
| `REDIS_QUEUE` | Name of the redis queue to listen to | `packaging-queue` |
| `HOST` | Hostname or IP address to bind to for healtchechk endpoint | `0.0.0.0` |
| `PORT` | Port to bind to for healtchechk endpoint | `8000` |
| `DISABLE_HEALTCHECK` | Disable the healthcheck endpoint | `false` |
| `SHAKA_PACKAGER_EXECUTABLE` | Path to the shaka packager executable | `packager` |
| `PACKAGE_OUTPUT_FOLDER` | Base folder for output, actual output will be in a subfolder named from the job id | `packaged` |
| `PACKAGE_CONCURRENCY` | Number of concurrent packaging jobs | `1` |
| `PACKAGE_LISTENER_PLUGIN` | Optional path to a javascript file containing a custom listener for packaging event, see below | |
| `PACKAGE_FORMAT_OPTIONS_JSON` | Optional JSON string with format options for shaka packager, format as defined in https://github.com/Eyevinn/shaka-packager-s3/blob/main/src/packager.ts |
| `VIDEO_STREAM_KEY_TEMPLATE` | Optional template for video stream key, see below for supported keywords | `$VIDEOIDX$_$BITRATE$` |
| `AUDIO_STREAM_KEY_TEMPLATE` | Optional template for video stream key, see below for supported keywords | `$AUDIOIDX$` |
| `ENCORE_PASSWORD` | Optional password for the encore instance `user` user | |
| `OSC_ACCESS_TOKEN` | Optional OSC access token for accessing Encore instance in OSC | |
| `AWS_ACCESS_KEY_ID` | Optional AWS access key id when `PACKAGE_OUTPUT_FOLDER` is an AWS S3 bucket | |
| `AWS_SECRET_ACCESS_KEY` | Optional AWS secret access key when `PACKAGE_OUTPUT_FOLDER` is an AWS S3 bucket | |
| Variable | Description | Default value |
|-------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------|
| `REDIS_URL` | URL to the redis server | `redis://localhost:6379` |
| `REDIS_QUEUE` | Name of the redis queue to listen to | `packaging-queue` |
| `HOST` | Hostname or IP address to bind to for healtchechk endpoint | `0.0.0.0` |
| `PORT` | Port to bind to for healtchechk endpoint | `8000` |
| `DISABLE_HEALTCHECK` | Disable the healthcheck endpoint | `false` |
| `SHAKA_PACKAGER_EXECUTABLE` | Path to the shaka packager executable | `packager` |
| `PACKAGE_OUTPUT_FOLDER` | Base folder for output, actual output will be in a subfolder according to `OUTPUT_SUBFOLDER_TEMPLATE` | `packaged` |
| `PACKAGE_CONCURRENCY` | Number of concurrent packaging jobs | `1` |
| `PACKAGE_LISTENER_PLUGIN` | Optional path to a javascript file containing a custom listener for packaging event, see below | |
| `PACKAGE_FORMAT_OPTIONS_JSON` | Optional JSON string with format options for shaka packager, format as defined in https://github.com/Eyevinn/shaka-packager-s3/blob/main/src/packager.ts |
| `VIDEO_STREAM_KEY_TEMPLATE` | Optional template for video stream key, see below for supported keywords | `$VIDEOIDX$_$BITRATE$` |
| `AUDIO_STREAM_KEY_TEMPLATE` | Optional template for video stream key, see below for supported keywords | `$AUDIOIDX$` |
| `OUTPUT_SUBFOLDER_TEMPLATE`| Template for subfolder relative to `PACKAGE_OUTPUT_FOLDER` where output will be stored. Keywords `$INPUTNAME$` and `$JOBID$` will be replaced with basename of input, and id of encore job respectively | `$INPUTNAME$/$JOBID$` |
| `ENCORE_PASSWORD` | Optional password for the encore instance `user` user | |
| `OSC_ACCESS_TOKEN` | Optional OSC access token for accessing Encore instance in OSC | |
| `AWS_ACCESS_KEY_ID` | Optional AWS access key id when `PACKAGE_OUTPUT_FOLDER` is an AWS S3 bucket | |
| `AWS_SECRET_ACCESS_KEY` | Optional AWS secret access key when `PACKAGE_OUTPUT_FOLDER` is an AWS S3 bucket | |

##### Stream key templates
Stream key templates can be used to set the 'key' for each stream, which decides how the stream is identified in the packaged manifest.
Expand Down
6 changes: 6 additions & 0 deletions src/config.ts
View file
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@ export interface RedisConfig {

export interface PackagingConfig {
outputFolder: string;
outputSubfolderTemplate: string;
concurrency: number;
shakaExecutable?: string;
packageListenerPlugin?: string;
Expand All @@ -30,6 +31,8 @@ export interface PackagingConfig {
streamKeysConfig: StreamKeyTemplates;
}

export const DEFAULT_OUTPUT_SUBFOLDER_TEMPLATE = '$INPUTNAME$/$JOBID$';

export interface StreamKeyTemplates {
video: string;
audio: string;
Expand Down Expand Up @@ -67,6 +70,9 @@ function readPackagingConfig(): PackagingConfig {
outputFolder: process.env.PACKAGE_OUTPUT_FOLDER?.match(/^s3:/)
? new URL(process.env.PACKAGE_OUTPUT_FOLDER).toString()
: resolve(process.env.PACKAGE_OUTPUT_FOLDER || 'packaged'),
outputSubfolderTemplate:
process.env.OUTPUT_SUBFOLDER_TEMPLATE ||
DEFAULT_OUTPUT_SUBFOLDER_TEMPLATE,
shakaExecutable: process.env.SHAKA_PACKAGER_EXECUTABLE,
concurrency: parseInt(process.env.PACKAGE_CONCURRENCY || '1'),
packageListenerPlugin: process.env.PACKAGE_LISTENER_PLUGIN,
Expand Down
9 changes: 5 additions & 4 deletions src/encorePackager.ts
View file
Original file line number Diff line number Diff line change
Expand Up @@ -57,12 +57,13 @@ export class EncorePackager {
getPackageDestination(job: EncoreJob) {
const inputUri = job.inputs[0].uri;
const inputBasename = basename(inputUri, extname(inputUri));
const subfolder = this.config.outputSubfolderTemplate
.replaceAll('$INPUTNAME$', inputBasename)
.replaceAll('$JOBID$', job.id);
if (this.config.outputFolder.match(/^s3:/)) {
return new URL(
this.config.outputFolder + inputBasename + '/' + job.id
).toString();
return new URL(this.config.outputFolder + subfolder).toString();
} else {
return resolve(this.config.outputFolder, inputBasename, job.id);
return resolve(this.config.outputFolder, subfolder);
}
}

Expand Down

0 comments on commit beb0595

Please sign in to comment.