Core Engine

The core engine of DiffSync is meant to be transparent for most users but in some cases it’s important to have the ability to change its behavior to adjust to some specific use cases. For these use cases, there are several ways to customize its behavior:

Global and Model Flags

Diff class

Global and Model Flags

These flags offer a powerful way to instruct the core engine how to handle some specific situation without changing the data. One way to think of the flags is to represent them as configuration for the core engine. Currently 2 sets of flags are supported:

global flags: applicable to all data.
model flags: applicable to a specific model or to individual instances of a model.

The flags are stored in binary format which allows storing multiple flags in a single variable. See the section below `Working with flags <#working-with-flags>`_ to learn how to manage them.

The list of supported flags is expected to grow over time as more use cases are identified. If you think some additional flags should be supported, please reach out via Github to start a discussion.

Global flags

Global flags can be defined at runtime when calling one of these functions : diff_to ,diff_from, sync_to or sync_from

from diffsync.enum import DiffSyncFlags
flags = DiffSyncFlags.SKIP_UNMATCHED_DST
diff = nautobot.diff_from(local, flags=flags)

Supported Global Flags

Name	Description	Binary Value
CONTINUE_ON_FAILURE	Continue synchronizing even if failures are encountered when syncing individual models.	0b1
SKIP_UNMATCHED_SRC	Ignore objects that only exist in the source/”from” DiffSync when determining diffs and syncing. If this flag is set, no new objects will be created in the target/”to” DiffSync.	0b10
SKIP_UNMATCHED_DST	Ignore objects that only exist in the target/”to” DiffSync when determining diffs and syncing. If this flag is set, no objects will be deleted from the target/”to” DiffSync.	0b100
SKIP_UNMATCHED_BOTH	Convenience value combining both SKIP_UNMATCHED_SRC and SKIP_UNMATCHED_DST into a single flag	0b110
LOG_UNCHANGED_RECORDS	If this flag is set, a log message will be generated during synchronization for each model, even unchanged ones.	0b1000

Model flags

Model flags are stored in the attribute model_flags of each model and are usually set when the data is being loaded into the adapter.

from diffsync import Adapter
from diffsync.enum import DiffSyncModelFlags
from model import MyDeviceModel


class MyAdapter(Adapter):
    device = MyDeviceModel

    def load(self, data):
        """Load all devices into the adapter and add the flag IGNORE to all firewall devices."""
        for device in data.get("devices"):
            obj = self.device(name=device["name"])
            if "firewall" in device["name"]:
                obj.model_flags = DiffSyncModelFlags.IGNORE
            self.add(obj)

Supported Model Flags

Name	Description	Binary Value
IGNORE	Do not render diffs containing this model; do not make any changes to this model when synchronizing. Can be used to indicate a model instance that exists but should not be changed by DiffSync.	0b1
SKIP_CHILDREN_ON_DELETE	When deleting this model, do not recursively delete its children. Can be used for the case where deletion of a model results in the automatic deletion of all its children.	0b10
SKIP_UNMATCHED_SRC	Ignore the model if it only exists in the source/”from” DiffSync when determining diffs and syncing. If this flag is set, no new model will be created in the target/”to” DiffSync.	0b100
SKIP_UNMATCHED_DST	Ignore the model if it only exists in the target/”to” DiffSync when determining diffs and syncing. If this flag is set, the model will not be deleted from the target/”to” DiffSync.	0b1000
SKIP_UNMATCHED_BOTH	Convenience value combining both SKIP_UNMATCHED_SRC and SKIP_UNMATCHED_DST into a single flag	0b1100
NATURAL_DELETION_ORDER	When deleting, delete children before instances of this model.	0b10000

Working with flags

Flags are stored in binary format. In binary format, each bit of a variable represents 1 flag which allow us to have up to many flags stored in a single variable. Using binary flags provides more flexibility to add support for more flags in the future without redefining the current interfaces and the current DiffSync API.

Enable a flag (Bitwise OR)

Enabling a flag is possible with the bitwise OR operator |=. It’s important to use the bitwise operator OR when enabling a flags to ensure that the value of other flags remains unchanged.

>>> from diffsync.enum import DiffSyncFlags
>>> flags = DiffSyncFlags.CONTINUE_ON_FAILURE
>>> flags
<DiffSyncFlags.CONTINUE_ON_FAILURE: 1>
>>> bin(flags.value)
'0b1'
>>> flags |= DiffSyncFlags.SKIP_UNMATCHED_DST
>>> flags
<DiffSyncFlags.SKIP_UNMATCHED_DST|CONTINUE_ON_FAILURE: 5>
>>> bin(flags.value)
'0b101'

Checking the value of a specific flag (bitwise AND)

Validating if a flag is enabled is possible with the bitwise operator AND: &. The AND operator will return 0 if the flag is not set and the binary value of the flag if it’s enabled. To convert the result of the test into a proper conditional it’s possible to wrap the bitwise AND operator into a bool function.

>>> from diffsync.enum import DiffSyncFlags
>>> flags = DiffSyncFlags.NONE
>>> bool(flags & DiffSyncFlags.CONTINUE_ON_FAILURE)
False
>>> flags |= DiffSyncFlags.CONTINUE_ON_FAILURE
>>> bool(flags & DiffSyncFlags.CONTINUE_ON_FAILURE)
True

Disable a flag (bitwise NOT)

After a flag has been enabled, it’s possible to disable it with a bitwise AND NOT operator : &= ~

>>> from diffsync.enum import DiffSyncFlags
>>> flags = DiffSyncFlags.NONE
# Setting the flags SKIP_UNMATCHED_DST and CONTINUE_ON_FAILURE
>>> flags |= DiffSyncFlags.SKIP_UNMATCHED_DST | DiffSyncFlags.CONTINUE_ON_FAILURE
>>> flags
<DiffSyncFlags.SKIP_UNMATCHED_DST|CONTINUE_ON_FAILURE: 5>
>>> bool(flags & DiffSyncFlags.SKIP_UNMATCHED_DST)
True
# Unsetting the flag SKIP_UNMATCHED_DST; CONTINUE_ON_FAILURE remains set
>>> flags &= ~DiffSyncFlags.SKIP_UNMATCHED_DST
>>> flags
<DiffSyncFlags.CONTINUE_ON_FAILURE: 1>
>>> bool(flags & DiffSyncFlags.SKIP_UNMATCHED_DST)
False

Custom Diff Class

When performing a diff or a sync operation, a diff object is generated. A diff object is itself composed of DiffElement objects representing the different elements of the original datasets with their differences.

The diff object helps to access all the DiffElements. It’s possible to provide your own Diff class in order to customize some of its capabilities the main one being the order in which the elements are processed.

Using your own Diff class

To use your own diff class, you need to provide it at runtime when calling one of these functions : diff_to, diff_from, sync_to or sync_from.

>>> from diffsync.enum import DiffSyncFlags
>>> from diff import AlphabeticalOrderDiff
>>> diff = remote_adapter.diff_from(local_adapter, diff_class=AlphabeticalOrderDiff)
>>> type(diff)
<class 'AlphabeticalOrderDiff'>

Change the order in which the element are being processed

By default, all objects of the same type will be stored in a dictionary and as such the order in which they will be processed during a diff or a sync operation is not guaranteed (although in most cases, it will match the order in which they were initially loaded and added to the adapter). When the order in which a given group of object should be processed is important, it’s possible to define your own ordering inside a custom Diff class.

When iterating over a list of objects, either at the top level or as a group of children of a given object, the core engine is looking for a function named after the type of the object order_children_<type> and if none is found it will rely on the default function order_children_default. Either function need to be present and need to return an Iterator of DiffElement.

In the example below, by default all devices will be sorted per type of CRUD operations (order_children_device) while all other objects will be sorted alphabetically (order_children_default)

class MixedOrderingDiff(Diff):
    """Alternate diff class to list children in alphabetical order, except devices to be ordered by CRUD action."""

    @classmethod
    def order_children_default(cls, children):
        """Simple diff to return all children in alphabetical order."""
        for child_name, child in sorted(children.items()):
            yield children[child_name]

    @classmethod
    def order_children_device(cls, children):
        """Return a list of device sorted by CRUD action and alphabetically."""
        children_by_type = defaultdict(list)

        # Organize the children's name by action create, update or delete
        for child_name, child in children.items():
            action = child.action or "skip"
            children_by_type[action].append(child_name)

        # Create a global list, organized per action
        sorted_children = sorted(children_by_type["create"])
        sorted_children += sorted(children_by_type["update"])
        sorted_children += sorted(children_by_type["delete"])
        sorted_children += sorted(children_by_type["skip"])

        for name in sorted_children:
            yield children[name]

Store backends

By default, Diffsync supports a local memory storage. All the loaded models from the adapters will be stored in memory, and become available for the diff calculation and sync process. This default behavior works well when executing all the steps in the same process, having access to the same memory space. However, if you want to scale out the execution of the tasks, running it in different processes or in totally different workers, a more distributed memory support is necessary.

The store is a class attribute in the Adapter class, but all the store operations in that class are abstracted in the following methods: get_all_model_names, get, get_by_uids, add, update, remove, get_or_instantiate, update_or_instantiate and count.

Use the `LocalStore` Backend

When you initialize the Diffsync Adapter class, there is an optional keyed-argument, internal_storage_engine, defaulting to the LocalStore class.

>> > from diffsync import Adapter
>> > adapter = Adapter()
>> > type(adapter.store)
<

class 'diffsync.store.local.LocalStore'>

Use the `RedisStore` Backend

To get it, you have to install diffsync package with the “redis” extra option: pip install diffsync[redis]

The RedisStore backend, as the name suggests, connects to an external Redis service, to store data loaded by the Adapter tasks. The biggest change is that it requires to initialize the Redis store class, before using it in the Adapter adapter class.

>> > from diffsync import Adapter
>> > from diffsync.store.redis import RedisStore
>> > store = RedisStore(host="redis host")
>> > adapter = Adapter(internal_storage_engine=store)
>> > type(adapter.store)
<

class 'diffsync.store.local.RedisStore'>

Notice that the RedisStore will validate, when initialized, that there is a reachability to the Redis host, and if not, will raise an exception:

>>> from diffsync.store.redis import RedisStore
>>> store = RedisStore(host="redis host")
redis.exceptions.ConnectionError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/chadell/github.ntc/diffsync/diffsync/store/redis.py", line 34, in __init__
    raise ObjectStoreException("Redis store is unavailable.") from RedisConnectionError
diffsync.exceptions.ObjectStoreException: Redis store is unavailable.

Using RedisStore, every adapter uses a specific Redis label, generated automatically, if not provided via the store_id keyed-argument. This store_id can be used to point an adapter to the specific memory state needed for diffsync operations.