Skip to content

5.0.1 - Finally here and in style
Compare
Choose a tag to compare
@kata198 kata198 released this 01 May 23:40
· 88 commits to master since this release
5.0.1
8fa513a

5.0.1 - Mon May 1 2017

A conversion document has been created, CONVERTING_TO_5.0.0 in the main
directory of this package. Also available online, at
https://github.com/kata198/indexedredis/blob/5.0branch/CONVERTING_TO_5.0.0

Use this document to prepare your code for the changes in IndexedRedis 5.0.0.
Also please review this Changelog to see new features, bug fixes, etc which
will not be listed in the conversion document.

  • Improve IRPickleField to work with ALL types (including now, strings and
    bytes)

    THIS WAS PREVIOUSLY BACKWARDS-INCOMPATIBLE, BUT HAS BEEN MADE COMPATIBLE
    WITHOUT CHANGES TO YOUR CODE. DO NOT RUN THE COMPAT FUNCTION SHIPPED WITH
    4.1.3 - 4.1.4

  • Fixup the way connecting to Redis works. Now, instead of having to define
    REDIS_CONNECTION_PARAMS on each model, you can call
    "setDefaultRedisConnectionParams" to set a default connection configuration.
    This configuration will be used on all models unless REDIS_CONNECTION_PARAMS has been
    defined and is not empty.

  • Rename "connect" method to "connectAlt" on IndexedRedisModel (to better
    convey what it does). Also fix it up to better support multiple alternate
    connection models

  • Remove formerly deprecated IndexedRedisModel.BASE64_FIELDS - You've been
    warned for a while now to use fields.IRField.IRBase64Field
    BACKWARDS INCOMPATIBLE
    To convert, remove the BASE64_FIELDS array from your model. For any items
    that are in it, change it in the FIELDS array to be:
    IRBase64Field
    as found in
    from IndexedRedis.fields import IRBase64Field

    If you already have a type, and were using it to base64 encode that type
    (like maybe you are compressing then base64 encoding), use a IRFieldChain
    with base64 as the last element to retain same functionality.

    Like:
    IRFieldChain('fieldName', [IRUnicodeField(encoding='utf-16'),
    IRBase64Field()])

    Models with BASE64_FIELDS defined will generate an error upon validation.

  • Remove IndexedRedis.BINARY_FIELDS - Use the new IRBytesField to act the same
    way as a BINARY_FIELD did on python3 or python2, use IRField(valueType=bytes)
    [python 3 only], or use IRRawField for no encoding/decoding at all. You can
    combine these in a chain just like IRBase64Field.
    BACKWARDS INCOMPATIBLE

    Models with BINARY_FIELDS defined will generate an error upon validation.

  • Make copy.copy and copy.deepcopy on an IndexedRedisModel call
    IndexedRedisModel.copy(False), in order to copy the data, but NOT to have the
    copy be linked to the database. A save on the copy will be an insert, the
    parent will remain an update.
    BACKWARDS INCOMPATIBLE

  • Removed the old, deprecated, compat global "defaultEncoding", retaining just
    defaultIREncoding. Also, removed deprecated getEncoding and setEncoding, use getDefaultIREncoding and setDefaultIREncoding
    BACKWARDS INCOMPATIBLE

    You shouldn't notice this. You should be using setDefaultIREncoding or
    getDefaultIREncoding for default encodings

  • Introduce a new IRField method called "fromInput", which is like
    "convert", but is used when the input is guarenteed to NOT come from storage.
    This will simplify a lot of things, remove some conditional and weird garbage,
    etc. Constructing objects is now a lot cheapear.

  • Store "was validated" result for model by the class itself. This allows models with the same keyname (like for conversion, tests, etc) to be validated for each unique class.

  • Add IRClassicField as a replacement for the plain-string name in FIELDS array. validateModel will automatically convert these to IRClassicField and warn you that these may go away. I have not yet decided if 5.0.0 will generate error or just warning for now.

  • Default IRField's which have not been set to irNull (instead of empty string).
    Classic string fields (just a string name, not an IRField object) will retain
    the empty string.
    This allows there to be a difference from not-set and empty-string value items
    (which can be meaningful in several contexts).

You can override this by setting defaultValue='something else' (like empty
string for previous default on some types, irNull was previous default on
others)
BACKWARDS INCOMPATIBLE

Consider if you are testing like
if myObj.someField == ''
that this will stop working on newly created objects if you aren't setting
the value, and expecting that to test such a condition. You can use:
if not myObj.someField
or
if (myObj.someField in ('', b'', irNull)
to support both.

There is no "compat convert" method for this, because it's unknown if an
item was set to default string, or empty value. It's on you to make sure
your code either handles both, or you convert your datasets accordingly.

You can override the defaultValue by passing defaultValue=X to any IRField
or subclass, so to retain old behaviour set defaultValue='' on your fields.

  • Allow specifying a defaultValue on IRFields. This specifies what the default
    value will be (was not set) for each field. Every IRField type (Except
    IRClassicField) supports specifying a default value.

  • repr on FIELDS now shows the field type and any properties specific to that object. asDict now takes a new param, (strKeys, default False), which will str the keys instead of using the IRField (so if you are pretty-printing the asDict repr, you'll want to use this to prevent the field types from creeping into the printout)

  • Use "fromInput" every time a field is set on an IndexedRedisModel
    ( via myModel.xfield = someVal ), same as if it were done via the constructor.
    This ensures that everything always has an expected type, and can fix some
    weird issues when folks set unconverted values.
    ** Backwards Incompatible, if you were doing it wrong before... ;) **

  • Rename "convert" to "fromStorage" on IRField. Instead of implementing
    convert/toStorage/fromInput, the subclasses should implement the
    underscore-prefixed version,

    • _fromStorage
    • _fromInput
    • _toStorage

The non-underscore prefixed versions have been reworked to handle irNull and
such.

  • Change the way irNull is handled, pull pretty well all of the work and
    flip-flopping out of IndexedRedisModel and make it part of IRField

  • Ensure that "toIndex" is ALWAYS called when setting indexes. This fixes some
    issues with hashIndex=True (like that irNull would not previously hash).

  • Fix some cases where irNull would equal empty string on indexes

  • Work around a strange "bug" in python that could cause some builtin types to
    be overriden within the "fields.init" module

  • Make IRBytesField able to index (by hashing the bytes)

  • Force IRUnicodeField to always have a hashed index, when used as an index
    BACKWARDS INCOMPATIBLE
    If you were using an IRUnicodeField as an index before, you will need to
    reindex your data. This is because without hasing, there are issues with
    many codecs at obtaining the index key. But it's okay, it's simple to
    reindex and move forward.

    For any model that has an index on an IRUnicodeField, you must call:

    MyModel.objects.compat_convertHashedIndexes()

    This existing method will take care of ensuring that ALL fields are using
    the proper index format, be it hashed or unhashed, and even takes care of
    if you accidently let your application run and have a mix of both hashed
    and unhashed values within the index.

    Please ensure that your application is offline before executing this
    function, for each model that may need index conversion.

  • Update runTests.py to the latest provided with the new version of
    GoodTests.py

  • Some minor to moderate cleanups/fixups/optimizations

  • Add pprint method to both IndexedRedisModel and IRQueryableList. This will
    pretty-print a dict representation of the model/collection of models, to a
    given stream (default sys.stdout)

  • Add eq and ne to IndexedRedisModel, to test if the models are equal
    (including ID).

  • Add "hasSameValues" method to IndexedRedisModel, to test if two objects have
    the same field values, even if the _id primary key is different (i.e.
    different objects in the database)

  • Fix where on insert the forStorage data would be put in _origData instead of
    the converted data, which would cause that field to be listed as needing to be
    updated if that same object was saved again

  • Fix a strange bug where some builtin types could be overriden in the
    IndexedRedis.fields.init.py file

  • Allow specifying an explicit encoding on IRBytesField, to use an alternative
    than the default encoding

  • Make IRField's copyable

  • Add "copyModel" method to IndexedRedisModel as a class method which will
    return a copy of the model class. This way when converting field types etc you
    can easily have the same model, copy it, and change one.

  • Add "name" property to IRField to make it more obvious how to access ( you
    can still use str(fieldObj) )

  • Improve compat_convertHashedIndexes method to work with all field types, and
    to only touch fields which can both hash and not hash

  • Update IRFixedPointField to ensure the value is always rounded to
    #decimalPlaces property, not just after fetch from storage.

  • Ensure IRRawField always actually does no encoding/decoding, does not support
    irNull (well, it just treats it same as empty string)

  • Ensure that IRBytesField always encodes to bytes as soon as you set the
    value on the object

  • Ensure that every field type converts the value to its consumable form (so
    same before-save and after-fetch) when setting the attribute through the
    dot-operator (myobj.value = X ) or constructor.

  • Fix bug where toIndex could be calling toStorage twice on a value (which
    affected things like base64 fields, which could be base64-of-base64 on index).
    Didn't matter before because base64 fields couldn't index before, but maybe
    some case hit this bug.

  • Allow IRFieldChain to index if the right-most field in the chain supports
    indexing

  • Added many more tests, like for each field type, lots more usage scenarios,
    etc.

  • Make IRCompressedField indexable.

The only non-indexable fields now are
IRPickleField (Different py2 vs py3 repr), IRRawField (because, no
encoding/decoding), and IRField(valueType= (float or dict or json) ) (float because
floats are machine-dependent, use IRFixedPointField to define a precision and
support indexes. And dicts are not ordered. And json has too many nesting
issues. )

  • Support "lzma" ( aka "xz" ) compression in IRCompressedField. This is
    provided by the core in python3. For python2, IndexedRedis also supports
    backports.lzma and lzmaffi as alternative implementations. You can also
    explicitly set IndexedRedis.fields.compressed._lzmaMod if you have some other
    external implementation for python2

  • Some cleanups, refactoring, optimizations, simplifications

  • Rename the fields.bytes fields.unicode and fields.pickle module names to not
    conflict with builtins. You should be inheriting all your fields like:

    from IndexedRedis.fields import IRPickleField
    anyway, so should not notice this change.

  • More updates on the way Redis connection parameters work.

    Now -- instead of using the "fixed" defaults of host="127.0.0.1",
    port=6379, db=0 for connections, IndexedRedis will use what you set
    through setDefaultRedisConnectionParams. The old defaults remain the
    initial version of the defaultRedisConnectionParams, so existing code
    will not break.

    This means now that models that define their own REDIS_CONNECTION_PARAMS
    are filled in by the defaultRedisConnectionParams for any missing slots,
    rather than the aforementioned hard defaults.

    So for example, for the following:

      setDefaultRedisConnectionParams( { 'host' : '192.168.1.1' , 'port' :
  15000, 'db' : 0 } )

  class MySpecialModel(IndexedRedisModel):
      ...

      REDIS_CONNECTION_PARAMS = { 'db' : 1 }

    The default connection for everything will use the first dict
    (host='192.168.1.1', port=15000, db=0 ).

    MySpecialModel will override the db portion, but inherit host and port,
    so MySpecialModel will use (host='192.168.1.1', port=15000, db=1)

    This only affects connection pools that IndexedRedis manages. If you
    define your own (by setting the 'connection_pool' key on your connection
    params), as before you will not get any of the inheritance,
    connect/disconnect, size management, etc that IndexedRedis would
    otherwise provide.

    Some other changes because of this:

    • getDefaultRedisConnectionParams will return a COPY of the default
      params, to prevent updates (such that IndexedRedis can track and apply
      them properly). setDefaultRedisConnectionParams already made copies of
      the values and set them on the global instance.

    • Introduce as "clearRedisPools" method which will disconnect and clear
      the connection_pool ConnectionPools off of all non-managed connections.

    • Calling setDefaultRedisConnectionParams will now call clearRedisPools,
      disconnecting any active managed connections and clearing the cached
      connection_pool associated with each model (except managed pools). This
      allows inheritance to happen again with the new properties, for example
      on the above example code, if 'host' was changed in the global,
      MySpecialModel on next connection will inherit that value (and cache the
      inherited settings and connection pool).

4.1.4 - Sat Apr 15 2017

  • Update READMEs a bit

4.1.3 - Wed Apr 12 2017

** IMPORTANT -- UPGRADE TO THIS, ADJUST YOUR CODE IN DEV TO PREPARE FOR 5.0.0
**

  • Add IRBytesField to work the same as BINARY_FIELDS on both python2 and 3.
    BINARY_FIELDS is going away in python 5.0.0

  • Add the new Pickle field implementation from 5.0.0 (which supports pickling
    ALL types, but is not compatible). This is named IRNewPickleField and can be
    found in
    from IndexedRedis.fields.new_pickle import IRNewPickleField

You can change your models to use this field, then on each model call
myModel.compat_convertPickleFields()

to convert the data form the old form to the new form.

Please see the 5.0.0 preview release also released today.

At 4.1.3 you can
make all your code forwards-compatible (with the exception of pickle fields,
which need to be converted post-update)

Several things are changing backwards-incompatible, but if you start making
changes now, you can run on 4.1.3 and then upgrade to 5.0.0 without issue.

4.1.2 - Wed Apr 12 2017

  • Fix some potential encoding issues on python2
  • Fixup some potential issues with IRBase64Field on python2 ( keep in mind,
    BASE64_FIELDS is going away in IndexedRedis 5.0.0! )
  • Some minor performance updates in encoding/decoding paths

4.1.1 - Wed Apr 12 2017

  • Ensure that the "origData" structure gets updated after a save which
    performs an insert. It was previously only updated after an "update", but not
    an insert. This prevents those fields as being seen as changed and updated
    again if you save the inserted object.
  • Update "runTests.py" to be version 1.2.3 from GoodTests.py

4.1.0 - Mon Jan 9 2017

  • Fix issue where fields that implemented toStorage (complex IRFields) would sometimes have problems
    doing updates
  • Fix issue where field values that work as-reference (like lists) would be seen to not update. We now make
    a copy in the "_origData" dict, if possible. If doesn't support copy.copy, we fallback to value assignment.
  • Add tests, specifically for IRPickleField with lists where these issues came up
Assets 2
Loading