Is the models' object.objects.bulk_update() and bulk_create() function supported?

[jchambers2012]

We have a few scripts that do large bulk operations like synchronizing a hundred VLANs to interfaces with tags or pull in IPAM data from external sources. Today we use the object.save() takes some time to run (like .3 sec per vlan per interface – one run took 60+ minutes). We wanted to start using the function bulk_update() to help speedup this operation based on some limited testing in our development environment. Reading the docs for bulk_update the update() function is called and the pre_save and post_save signals aren’t sent. Will this be an issue if we convert our scripts to use bulk_update vs a loop of saves?

https://docs.djangoproject.com/en/5.0/ref/models/querysets/#bulk-update

Each model’s save() method isn’t called, and the pre_save and post_save signals aren’t sent.

https://docs.djangoproject.com/en/5.0/ref/models/querysets/#bulk-create

The model’s save() method will not be called, and the pre_save and post_save signals will not be sent.

[jchambers2012]

We do see bulk_update is use in the code base - few examples:

• netbox/netbox/ipam/signals.py

  Line 16 in 36df922

  Prefix.objects.bulk_update(parents, ['_children'], batch_size=100)

• netbox/netbox/ipam/signals.py

  Line 26 in 36df922

  Prefix.objects.bulk_update(children, ['_depth'], batch_size=100)

• netbox/netbox/core/models/data.py

  Line 216 in 36df922

  updated_count = DataFile.objects.bulk_update(updated_files, ('last_updated', 'size', 'hash', 'data'))

• netbox/netbox/dcim/models/devices.py

  Line 1263 in 36df922

  component_model.objects.bulk_update(update_instances, update_fields)

• netbox/netbox/extras/models/customfields.py

  Line 275 in 36df922

  model.objects.bulk_update(instances, ['custom_field_data'], batch_size=100)

And bulk_create is used in the testing files - few examples:

• netbox/netbox/netbox/tests/test_search.py

  Line 44 in 36df922

  Site.objects.bulk_create(sites)

• netbox/netbox/virtualization/tests/test_api.py

  Line 163 in 36df922

  Cluster.objects.bulk_create(clusters)

• netbox/netbox/ipam/tests/test_views.py

  Line 409 in 36df922

  Role.objects.bulk_create(roles)

• netbox/netbox/circuits/tests/test_filtersets.py

  Line 466 in 36df922

  Provider.objects.bulk_create(providers)

[candlerb]

Reading the docs for bulk_update the update() function is called and the pre_save and post_save signals aren’t sent. Will this be an issue if we convert our scripts to use bulk_update vs a loop of saves?

It can be.

Example: I wrote a custom script which used bulk_create() to add missing components to devices (from the device type). However, because the hooks aren't called, it didn't update the cached counters of the number of each type of component on a device, which meant that the tabs in the GUI either showed the wrong count, or the tabs were missing completely (if the count == 0)

Actually the pull request which fixes that hasn't been merged. It changes:

                    for i in items:
                        i.full_clean()
                    klass.objects.bulk_create(items)

to:

                    for i in items:
                        i.full_clean()
                        i.save()

[jchambers2012]

That what I was thinking as I was reading the source code more. It seems like it might an unsafe operation, might have to submit a feature request to add support for update() in models that support custom save().

Guessing we will have to go model-by-model to see what one might be safe and what might not be...... or just live with 60+ minute run time for some of our deployment scripts. 😄