Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Resource limit #55

Draft
wants to merge 15 commits into
base: master
Choose a base branch
from
Draft

Resource limit #55

wants to merge 15 commits into from

Commits on Oct 16, 2020

  1. Configuration menu
    Copy the full SHA
    240cea6 View commit details
    Browse the repository at this point in the history
  2. got more tests to pass

    LogJobStart was not being logged - FIXED
    saurabhnanda committed Oct 16, 2020
    Configuration menu
    Copy the full SHA
    1ff4676 View commit details
    Browse the repository at this point in the history

Commits on Oct 18, 2020

  1. Updated .gitignore to ignore some Cabal artifacts

    David Ellis committed Oct 18, 2020
    Configuration menu
    Copy the full SHA
    3e2186e View commit details
    Browse the repository at this point in the history
  2. Draft implementation for core functionality for resource limited jobs (

    …saurabhnanda#38)
    
    Many small changes were required, but the majority were mechanical
    changes. With the exception of the `createJob`/`scheduleJob` family of
    function, this should be mostly backwards compatible with existing
    user code that does not use "internal" functions. Further discussion
    about how to handle backwards compatibility is needed.
    
    Note also that this commit by itself is NOT compatible with existing
    databases. A future commit will address migration from existing to new
    database definitions.
    
    The intention is for this feature to have a negligable impact on
    performance when the resource limiting features are not being used,
    and only a minor impact when they are used. However, performance
    testing has not yet been done.
    
    Material changes inclue:
    
    - New resource table
    - New `resource_id` column in to job table
    - New `cfgDefaultResourceLimit` configuration value
    - New `createResourceJob` and `scheduleResourceJob` functions
    - New `TableNames` type to carry the now multiple values needed
    - Updated existing functions and queries to account for these changes
    - Updated dequeueing queries to implement resource concurrency limits
    - New `DevelDb` module to the `devel` executable to help ad hoc testing
    
    TODO:
    - Resource record management functions
    - Migration functionality (need to discuss database versioning)
    - Unit tests
    - Performance tests
    - Documentation updates
    - Resource management UI
    David Ellis committed Oct 18, 2020
    Configuration menu
    Copy the full SHA
    baeff32 View commit details
    Browse the repository at this point in the history

Commits on Oct 19, 2020

  1. Fix import path in the doc (saurabhnanda#49)

    kanagarajkm authored and David Ellis committed Oct 19, 2020
    Configuration menu
    Copy the full SHA
    05bd9cc View commit details
    Browse the repository at this point in the history
  2. Setup CI using Github Actions (saurabhnanda#50)

    * setup a simple CI using GitHub actions
    
    * describe a simple dev setup process
    
    * build branches with pull requests against master, run build on multiple operating systems
    
    * trigger CI for all branches
    
    * always checkout the ref
    
    * build only on pull_request events not to duplicate builds
    
    * build only on Ubuntu
    mjarosie authored and David Ellis committed Oct 19, 2020
    Configuration menu
    Copy the full SHA
    ac25d5b View commit details
    Browse the repository at this point in the history
  3. Reduced database notification payload (saurabhnanda#43)

    Because PostgreSQL limits notification payloads to 8000 bytes by
    default, passing the full job record in the notification artificially
    limits the job payload size.  By passing only required fields in the
    notification, this limitation is removed.
    mrputty authored and David Ellis committed Oct 19, 2020
    Configuration menu
    Copy the full SHA
    cec5558 View commit details
    Browse the repository at this point in the history
  4. Updated .gitignore to ignore some Cabal artifacts

    David Ellis committed Oct 19, 2020
    Configuration menu
    Copy the full SHA
    265d06d View commit details
    Browse the repository at this point in the history
  5. Draft implementation for core functionality for resource limited jobs (

    …saurabhnanda#38)
    
    Many small changes were required, but the majority were mechanical
    changes. With the exception of the `createJob`/`scheduleJob` family of
    function, this should be mostly backwards compatible with existing
    user code that does not use "internal" functions. Further discussion
    about how to handle backwards compatibility is needed.
    
    Note also that this commit by itself is NOT compatible with existing
    databases. A future commit will address migration from existing to new
    database definitions.
    
    The intention is for this feature to have a negligable impact on
    performance when the resource limiting features are not being used,
    and only a minor impact when they are used. However, performance
    testing has not yet been done.
    
    Material changes inclue:
    
    - New resource table
    - New `resource_id` column in to job table
    - New `cfgDefaultResourceLimit` configuration value
    - New `createResourceJob` and `scheduleResourceJob` functions
    - New `TableNames` type to carry the now multiple values needed
    - Updated existing functions and queries to account for these changes
    - Updated dequeueing queries to implement resource concurrency limits
    - New `DevelDb` module to the `devel` executable to help ad hoc testing
    
    TODO:
    - Resource record management functions
    - Migration functionality (need to discuss database versioning)
    - Unit tests
    - Performance tests
    - Documentation updates
    - Resource management UI
    David Ellis committed Oct 19, 2020
    Configuration menu
    Copy the full SHA
    ee7d216 View commit details
    Browse the repository at this point in the history

Commits on Oct 20, 2020

  1. Introduced dropJobTables and refactored Migrations.hs

    Because there are unit tests that delete tabes, and now that there are
    multiple table names in play, it seemed useful to explicitly implement
    a deletion function. Becuase this deserved a comment, I also added
    comments to the other externally useful functions in this module, and
    rearranged the file a bit to highlight these defintions.
    David Ellis committed Oct 20, 2020
    Configuration menu
    Copy the full SHA
    8595a3c View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    1a1c56b View commit details
    Browse the repository at this point in the history
  3. Updated tests for resource-limited jobs

    David Ellis committed Oct 20, 2020
    Configuration menu
    Copy the full SHA
    7c727b0 View commit details
    Browse the repository at this point in the history
  4. Organized imports in test/Test.hs

    David Ellis committed Oct 20, 2020
    Configuration menu
    Copy the full SHA
    0b6738f View commit details
    Browse the repository at this point in the history
  5. Updated tests to allow env var specification of database connection

    Now, if the environment variable "ODD_JOBS_TEST_DB_CONNECT" is set, it
    will be used as a connection string for unit test.  If not, the
    existing hard-coded default connection will be used.
    
    This is useful when developing in a containerized environment.
    David Ellis committed Oct 20, 2020
    Configuration menu
    Copy the full SHA
    bb06cdf View commit details
    Browse the repository at this point in the history
  6. Merge branch 'resource-limit-tests' into resource-limit

    This includes the changes that are on `fix/tests` branch.
    David Ellis committed Oct 20, 2020
    Configuration menu
    Copy the full SHA
    984da57 View commit details
    Browse the repository at this point in the history