Nautobot Apps - Lab 11: Syncing with External Systems¶
For this lab, we'll learn how to sync the Nautobot database with an external source.
- Nautobot Apps - Lab 11: Syncing with External Systems
- Task 1: Install the
nautobot-ssotapp - Task 2: Adding data sync capability
- Task 3: Add Locations to Sync job
Task 1: Install the nautobot-ssot app¶
The goal of this lab is to create a job that can read data from one (source) system and update another (target) system to reflect changes from the source. The approach we will use relies on the nautobot-ssot app. We will leverage the components provided to build the following components:
- Data models that represent the individual data records that we want to sync between systems
- Adapters that represent the source and target systems
- Jobs that orchestrate the synchronizing data between the two systems
Step 1-0 - Prepare for the Lab¶
For this and all remaining labs, it is advisable to open two terminal windows, one for running the Nautobot development server, and another for executing various commands.
The steps in this lab require that you have a terminal open and have switched to the nautobot user. Verify your username with the whoami command and switch user to nautobot if needed.
ntc@nautobot:~$ whoami
ntc
ntc@nautobot:~$ sudo -iu nautobot
nautobot@nautobot:~$ whoami
nautobot
nautobot@nautobot:~$
If you have the Nautobot development server running from a previous lab, please continue to the next step. Otherwise we need to start the server with the command nautobot-server runserver 0.0.0.0:8080 --insecure.
nautobot@nautobot:~$ nautobot-server runserver 0.0.0.0:8080 --insecure
Watching for file changes with StatReloader
Performing system checks...
System check identified no issues (0 silenced).
April 24, 2022 - 05:19:10
Django version 3.1.14, using settings 'nautobot_config'
Starting development server at http://0.0.0.0:8080/
Quit the server with CONTROL-C.
Step 1-1 - Install the nautobot-ssot Plugin¶
Install the plugin's Python package using pip. Issue the command pip install nautobot-ssot==1.3.2
nautobot@nautobot:~$ pip install nautobot-ssot==1.3.2
Collecting nautobot-ssot==1.3.2
<---OMITTED--->
Note: To ensure compatibility and a stable environment, Nautobot plugins usually have their version pinned to a specific version. In the case of
nautobot-ssot, we have pinned version1.3.2.
Step 1-2 - Update Local Requirements¶
The Nautobot upgrade process involves rebuilding the Python virtual environment. To ensure that the plugin persists after an upgrade, we'll add it to the local list of required Python packages, local_requirements.txt.
Note: Use of the double arrow (
>>) will append contents to an existing file or create a new file if one does not exist. A single arrow (>) in the above command would overwrite the file contents if the file already exists.
Step 1-3 - Enable the Plugin¶
Now that the nautobot-ssot plugin package is installed, we need to add it to the existing list of installed plugins.
Open the /opt/nautobot/nautobot_config.py file in VSCode and edit the PLUGINS configuration list to include the nautobot_ssot plugin. Do not remove any of the other plugins when adding this one.
PLUGINS = [
...
"nautobot_plugin_nornir",
"nautobot_golden_config",
"nautobot_ssot",
"maintenance_notices",
]
Note: When enabling a plugin, be sure to use the Python package name, not the name of the plugin itself, which may be different. (Python package names cannot contain hyphens.)
Save the file.
Step 1-4 - Run the Post Upgrade Command¶
The nautobot_ssot plugin includes additional data models that require the execution of database migrations before the plugin will work correctly.
Open the terminal where your development server is running and hit <CTRL> C to stop the server.
Next, run the nautobot-server post_upgrade command.
nautobot@nautobot:~$ nautobot-server post_upgrade
Performing database migrations...
Operations to perform:
Apply all migrations: admin, auth, circuits, contenttypes, database, dcim, django_celery_beat, django_rq, extras, ipam, maintenance_notices, nautobot_data_validation_engine, nautobot_device_lifecycle_mgmt, nautobot_golden_config, nautobot_ssot, sessions, social_django, taggit, tenancy, users, virtualization
Running migrations:
Applying nautobot_ssot.0001_initial... OK
Applying nautobot_ssot.0002_performance_metrics... OK
Applying nautobot_ssot.0003_alter_synclogentry_textfields... OK
Applying nautobot_ssot.0004_sync_summary... OK
...
15:58:34.205 INFO nautobot.extras.utils :
Created Job "SSoT Examples: Example Data Source" from <plugins: ExampleDataSource>
15:58:34.219 INFO nautobot.extras.utils :
Created Job "SSoT Examples: Example Data Target" from <plugins: ExampleDataTarget>
Generating cable paths...
...
Invalidating cache...
nautobot@nautobot:~$
Start your server after the post_upgrade command completes.
nautobot@nautobot:~$ nautobot-server runserver 0.0.0.0:8080 --insecure
Watching for file changes with StatReloader
Performing system checks...
...
Step 1-5 - Verify the Installed Plugin¶
Navigate to the Plugins menu in Nautobot UI and select Dashboard under Single Source of Truth
The Single Source of Truth dashboard is shown.
Step 1-6 - Run an example job¶
The nautobot_ssot plugin includes two example jobs for demonstration purposes. You can click the Sync button on either example to run the job.
We need to restart the Nautobot worker process to ensure it loads the current Python site packages.
You can restart the nautobot-worker service as the nautobot user and you will be prompted to authenticate with the ntc credentials that you used to ssh into the student pod.
nautobot@nautobot:~$ systemctl restart nautobot-worker
==== AUTHENTICATING FOR org.freedesktop.systemd1.manage-units ===
Authentication is required to restart 'nautobot-worker.service'.
Authenticating as: ntc
Password:
==== AUTHENTICATION COMPLETE ===
nautobot@nautobot:~$
Recall from Lab 8 that newly installed jobs are disabled by default and will need to be enabled before they can be queued for the nautobot-worker. See Lab 8, Step 2-8 "Enable the job" for details on enabling the job.
Step 1-7 - Hide the example jobs¶
In a production deployment we don't want to display the example jobs within the Nautobot UI. Edit the /opt/nautobot/nautobot_config.py file in VSCode and edit the PLUGINS_CONFIG dictionary. Add a key for "nautobot_ssot" and set the value to {"hide_example_jobs": True}.
Since the nautobot_config.py file is not monitored by the nautobot-server, we can induce the server to restart.
In your browser, reload The Single Source of Truth dashboard.
Note: The jobs will still show in the Job list view until a post-migration jobs signal is triggered, such as by running
nautobot-server migrate.
Task 2: Adding data sync capability¶
The goal of this task is to create a data sync job that will read in locations from a json file and sync that data to the Nautobot Location and LocationType models.
We will start by building an SSoT job that syncs data from a remote source (file upload) to the Nautobot LocationType model. Later, we will expand this to include syncing the Location model objects.
Step 2-1 - Create DiffSync files¶
Create a folder called diffsync within our project and create three files within the folder.
nautobot@nautobot:~$ mkdir /opt/nautobot/plugin/maintenance_notices/diffsync
nautobot@nautobot:~$ touch /opt/nautobot/plugin/maintenance_notices/diffsync/__init__.py
nautobot@nautobot:~$ touch /opt/nautobot/plugin/maintenance_notices/diffsync/adapters.py
nautobot@nautobot:~$ touch /opt/nautobot/plugin/maintenance_notices/diffsync/models.py
Step 2-2 - Create DiffSync model for LocationType¶
The DiffSync models are based on the Pydantic data validation models and, as such, they make use of Python type hints to specify the type of data that is expected for each named attribute.
Each instance of a DiffSyncModel represents a single record in a single system.
Our goal is to sync Location objects from an external source, this source will also include LocationType information.
Open the diffsync/models.py file in your editor and add the following code:
"""diffsync/models.py"""
from typing import List, Optional
from nautobot.dcim.models import LocationType
from diffsync import DiffSyncModel
class LocationTypeDiffSyncModel(DiffSyncModel):
"""DiffSync LocationType model."""
_modelname = "diffsync_location_type_model"
_identifiers = ("name",) # Note the comma that forces this to be a tuple.
name: str
In the above code we are creating a class that inherits from the DiffSyncModel class. We need to define a few class attributes in order to use this data model.
_modelname: Used by DiffSync to identify common models between different systems_identifiers: Attribute names used as primary keys for this object- The
nameattribute is enough to uniquely identify aLocationTypeobject
The name: str is in type hint notation and is used by the Pydantic BaseMmodel to know that this data model will have a name attribute whose value must be a string.
Lets add the following "write" methods to this class:
@classmethod
def create(cls, diffsync, ids, attrs):
name = ids["name"]
obj = LocationType.objects.create(name=name)
return super().create(diffsync=diffsync, ids=ids, attrs=attrs)
def update(self, attrs):
raise NotImplemented("Update not supported on LocationTypeDiffSyncModel")
def delete(self):
obj = LocationType.objects.get(name=self.name)
self.diffsync.job.log_warning(obj=obj, message="Deleting object.")
obj.delete()
return super().delete()
The create method does two things: It creates a Nautobot LocationType object and it creates and instance of itself where the object attributes are assigned from the ids and attrs.
The update method would normally be used to update an existing Nautobot object instance, however this class will not have updates due to the lack of any defined _attributes.
The delete method is called when the source of truth system does not have data that exists in the target system.
Step 2-3 - Create DiffSync Adapters for LocationType¶
DiffSync adapters represent the source and destination systems. The adapter can read in the relevant data from that system and compare its data against that of another adapter. It can also orchestrate the synchronization of the two data sources.
Open the diffsync/adapters.py file in your editor and add the following import statements:
"""diffsync/adapters.py"""
from nautobot.dcim.models import LocationType
from diffsync import DiffSync
from .models import LocationTypeDiffSyncModel
Here is the first of our two adapters. This will define our source of truth for this data, which will be a file upload.
class LocationSourceAdapter(DiffSync):
"""Remote Source DiffSync Adapter for Locations."""
diffsync_location_type_model = LocationTypeDiffSyncModel
top_level = ["diffsync_location_type_model"]
def __init__(self, *, data, job=None, **kwargs):
self.job = job
self.data = data
super().__init__(**kwargs)
def load(self):
for location_type_name, locations in self.data.items():
location_type_model = self.diffsync_location_type_model(
name=location_type_name
)
self.add(location_type_model)
location_type_count = len(self.dict()["diffsync_location_type_model"])
self.job.log_info(
message=f"Source adapter loaded {location_type_count:,} Location Type records from remote system"
)
The LocationSourceAdapter inherits from DiffSync. We are defining two attributes here:
diffsync_location_type_modelis assigned theLocationTypeDiffSyncModelclass that we created in the previous step.- The attribute name here must match the name defined in the model, ie:
LocationTypeDiffSyncModel._modelname - Any other models that will be handled by the adapter would be specified here as well. We will add another in the next task.
top_levelspecifies the names for theDiffSyncModelclasses that will be synchronized by this adapter.
We are overriding the inherited __init__ method so that we can capture two variables that will be passed to the adapter by the SSoT DataSource job. Here we assign the job and data keyword arguments to instance variables with the same names. By having a reference to the running job object we can write log messages at run time. The data instance variable will contain a dict of the data read in from the uploaded file.
The load method reads in the data from self.data. The method loops over each data source record and creates a DiffSyncModel instance contains the data. Each model instance gets added to the adapter until all records (model instances) are contained in the adapter.
We need to add another adapter to handle the target system. Since we are updating Nautobot using an external source of truth, then this target adapter will represent the data as seen by Nautobot.
class LocationTargetAdapter(DiffSync):
"""Nautobot DiffSync Adapter for Locations."""
diffsync_location_type_model = LocationTypeDiffSyncModel
top_level = ["diffsync_location_type_model"]
def __init__(self, *, job=None, **kwargs):
self.job = job
super().__init__(**kwargs)
def load(self):
queryset = LocationType.objects.only("name")
for obj in queryset:
name = obj.name
location_type_model = self.diffsync_location_type_model(name=name)
self.add(location_type_model)
queryset_count = queryset.count()
self.job.log_info(
message=f"Target adapter loaded {queryset_count:,} Location Type records from local system"
)
As you can see the target adapter is similar to the source adapter. Here the __init__ method is only assigning the job object to an instance variable.
The load method works a little differently, as it uses the Django ORM to read data from Nautobot.
Step 2-4 - Create SSoT DataSource Job¶
Next step is to create a job that will orchestrate the load, comparison, and synchronization between the two adapters. The Nautobot SSoT app provides two such base classes: DataSource and DataTarget. Which one you use depends on which system is considered to be the Source of Truth.
DataSource: Used when the remote system is the Source of Truth- All write operations will be written locally to Nautobot
DataTarget: Used when Nautobot is the Source of Truth- All write operations will be written to the remote system
The choice of base class will determine if this job gets listed under Data Sources or Data Targets within the Nautobot SSoT Dashboard.
Since our external system is the Source of Record, we will inherit from the DataSource base class.
Open the jobs.py file in your editor and modify your imports to match the following:
import json
from io import StringIO
from django.urls import reverse
from nautobot.dcim.models import Device
from nautobot.extras.jobs import FileVar, IntegerVar, Job, MultiObjectVar
from nautobot.utilities.utils import csv_format
from nautobot_ssot.jobs.base import DataMapping, DataSource
from .diffsync.adapters import LocationSourceAdapter, LocationTargetAdapter
from .models import MaintenanceNotice
Now add a new class definition below the existing job that was created in Lab 8.
class LocationSync(DataSource, Job):
"""Sync Locations from external source."""
file = FileVar(description="Location source data file (json)")
class Meta:
name = "Location Data Source"
description = "Sync Location data from JSON file upload"
data_source = "Location data upload file (remote)"
source_name = "Location data upload file"
target_name = "Nautobot Locations"
FileVar provides a form field on the job run page that allows the user to upload a file. The file handle will be available in self.kwargs["file"]
Attributes defined in class Meta will be displayed on the DataSource job page.
@classmethod
def data_mappings(cls):
location_type_data_map = DataMapping(
"Location Type data upload file (remote)",
None,
"Location Types (local)",
reverse("dcim:locationtype_list"),
)
return (location_type_data_map,)
Data mappings provide a nice table on the Sync job detail view and includes a link to the Nautobot object.
def load_source_adapter(self):
"""Load data from file upload."""
_file = self.kwargs["file"].read().decode("utf-8-sig")
data = json.load(StringIO(_file))
self.source_adapter = LocationSourceAdapter(data=data, job=self)
self.source_adapter.load()
def load_target_adapter(self):
"""Load data from Nautobot."""
self.target_adapter = LocationTargetAdapter(job=self)
self.target_adapter.load()
Here is where the action happens.
- The
load_source_adaptermethod reads the uploaded json file. Then it instantiates theLocationSourceAdapterand executes the.load()method. - The
load_target_adaptermethod simply instantiates theLocationTargetAdapterand executes the.load()method. - The built-in
sync_datamethod (not shown) will calculate the diff between the two adapters and writes the summary to the log. If this is not a dry run, the data synchronization is initiated.
As a final edit to this file, modify the last line to include the LocationSync job in the jobs variable.
Note: Be sure to save all of the above files before continuing.
Step 2-5 - Create source data file¶
The source data for this lab will be json data read from a file that we will upload to the job. Create a new file on your workstation where you are running your browser. Give the file a name a name like locations.json and add the following content to the file.
{
"Call Center": [
{"name": "Charlotte", "status": "active"},
{"name": "Fort Worth", "status": "active"}
],
"Data Center": [
{"name": "Jersey City", "status": "retired"},
{"name": "Reston", "status": "active"},
{"name": "San Jose", "status": "active"}
],
"Remote Office": [
{"name": "Atlanta", "status": "active"},
{"name": "Chicago", "status": "active"},
{"name": "New York", "status": "active"},
{"name": "Pittsburgh", "status": "planned"}
]
}
This data could have come from the result of an API call, an SQL query, or even a spreadsheet. Keep this file handy, we will upload it when we run the sync job.
Step 2-6 - Register and enable the new job¶
Anytime a new job is added to Nautobot, the job must get registered by a post-migration signal. Therefore we need to stop the Nautobot service and apply migrations. This can be done by running the nautobot-server migrate command, or with the more inclusive nautobot-server post_upgrade command.
nautobot@nautobot:~$ nautobot-server post_upgrade
Performing database migrations...
...
Invalidating cache...
nautobot@nautobot:~$
Start your server after the post_upgrade command completes.
nautobot@nautobot:~$ nautobot-server runserver 0.0.0.0:8080 --insecure
Watching for file changes with StatReloader
Performing system checks...
...
Navigate to the Jobs view in the Nautobot UI and located the Location Data Source job. Click the yellow pencil icon to edit the job.
Check the Enabled box to enable the job and click Update to save the changes.
Our new job should also be visible from SSoT Dashboard. Navigate to the Plugins menu in Nautobot UI and select Dashboard under Single Source of Truth
When we modify code that is used by our jobs we need to restart the Nautobot workers. This is due to the fact that the code is read in by the worker service at the time the service is started.
nautobot@nautobot:~$ systemctl restart nautobot-worker
==== AUTHENTICATING FOR org.freedesktop.systemd1.manage-units ===
Authentication is required to restart 'nautobot-worker.service'.
Authenticating as: ntc
Password:
==== AUTHENTICATION COMPLETE ===
nautobot@nautobot:~$
Step 2-7 - Run SSoT DataSource job¶
Now try running the job.
- You can toggle the
Dry Runflag to specify whether the sync should be performed. - Click the
Choose Filebutton to upload thelocations.jsonfile that you created in the previous step. - Click
Run Job Nowto execute.
Your Dry Run job results should look similar to the following:
Run the job again but this time deselect the Dry Run checkbox. When the job completes you should have three new LocationType objects when you navigate to Organization > Location Types in the Nautobot UI.
Task 3: Add Locations to Sync job¶
Our goal is to sync Locations so let's add that to our sync job.
Step 3-1 - Modify diffsync/models.py file¶
Replace the existing imports with the following.
from functools import lru_cache
from typing import List, Optional
from nautobot.dcim.models import Location, LocationType
from nautobot.extras.models import Status
from diffsync import DiffSyncModel
Add the _children and locations attributes to the LocationTypeDiffSyncModel class.
# Location Type model (parent)
class LocationTypeDiffSyncModel(DiffSyncModel):
"""DiffSync LocationType model."""
_modelname = "diffsync_location_type_model"
_identifiers = ("name",) # Note the comma that forces this to be a tuple.
name: str
locations: List = []
We have added two class attributes to the LocationTypeDiffSyncModel.
_children: This references another DiffSync modeldiffsync_location_model- Instances of that model will be stored in a container with the name
locations locations: is defined as aListand assigned to a new list.
Let's add another DiffSyncModel, this one is for the Location objects. We will also make use of two helper functions to perform Nautobot ORM lookups.
# Location Type model (parent)
class LocationTypeDiffSyncModel(DiffSyncModel):
"""DiffSync LocationType model."""
_modelname = "diffsync_location_type_model"
...
# Location model (child)
class LocationDiffSyncModel(DiffSyncModel):
"""DiffSync Location model."""
_modelname = "diffsync_location_model"
_children = {"diffsync_location_model": "locations"}
_identifiers = ("name", "location_type")
_attributes = ("status", "description")
name: str
description: Optional[str]
status: str
location_type: str
Our LocationType model had only one attribute (name) that we used for our sync operation. Here we have four attributes of the Location model that we want to use for comparison between our data sources: name, description, status, and location_type.
Our class attributes are divided into the two groups:
_identifiersare those attributes that identify the record- These attributes are used to match records between the two data sources
- These attributes will not trigger an
update _attributesare those attributes that are compared between the the matching data source records.- Any difference found in these attributes will trigger an update to the target system
Add the create method to the new class.
@classmethod
def create(cls, diffsync, ids, attrs):
"""Create an object in Nautobot."""
name = ids["name"]
location_type = ids["location_type"]
status_name = attrs["status"]
description = attrs["description"]
location_type_id = _get_location_type_id(location_type)
status_id = _get_status_id(status_name)
obj = Location.objects.create(
name=name,
location_type_id=location_type_id,
status_id=status_id,
description=description,
)
return super().create(diffsync=diffsync, ids=ids, attrs=attrs)
The create method creates and object on the target system, in this case a Nautobot Location object instance, and a DiffSync model instance, which is retuned to the method caller. The method shown here uses some helper functions to lookup object IDs from the Nautobot ORM.
Add the update method to the new class.
def update(self, attrs):
"""Update an object in Nautobot."""
queryset = Location.objects.filter(name=self.name)
if "description" in attrs:
description = attrs["description"]
queryset.update(description=description)
if "status" in attrs:
status_name = attrs["status"]
status_id = _get_status_id(status_name)
queryset.update(status_id=status_id)
return super().update(attrs=attrs)
The update method updates the taget system attribute values that differ with the source system values. The attrs variable that is passed to the method is a dict object containing the attribute names and values for only those attributes where the values differ.
Add the delete method to the new class.
def delete(self):
"""Delete an object in Nautobot."""
obj = Location.objects.get(name=self.name)
obj.delete()
return super().delete()
The delete method removes an object from the target system. This method is called when there is no matching record in the source of truth system.
Lastly, add the helper functions.
@lru_cache()
def _get_location_type_id(name):
obj = LocationType.objects.get(name=name)
return obj.id
@lru_cache()
def _get_status_id(name):
slug = name.lower()
obj = Status.objects.get(slug=slug)
return obj.id
These helper functions are used to lookup the object IDs for objects inthe Nautobot database. Since most of the same objects are expected to be retrieved, the @lru_cache() decorator will return values for names that have already been resolve to an ID.
Step 3-2 - Modify diffsync/adapters.py file¶
Open the diffsync/adapters.py file in your editor. Again we will start with the modifying required imports to match what we have below.
from nautobot.dcim.models import Location, LocationType
from diffsync import DiffSync
from .models import LocationDiffSyncModel, LocationTypeDiffSyncModel
Next, modify the LocationSourceAdapter to match the code snippet below.
class LocationSourceAdapter(DiffSync):
"""Remote Source DiffSync Adapter for Locations."""
diffsync_location_type_model = LocationTypeDiffSyncModel
diffsync_location_model = LocationDiffSyncModel
top_level = ["diffsync_location_type_model"]
def __init__(self, *, data, job=None, **kwargs):
self.job = job
self.data = data
super().__init__(**kwargs)
def load(self):
for location_type_name, locations in self.data.items():
location_type_model = self.diffsync_location_type_model(
name=location_type_name
)
self.add(location_type_model)
for location in locations:
location_name = location["name"]
description = location.get("description", "")
location_status_name = location["status"]
location_model = self.diffsync_location_model(
name=location_name,
description=description,
location_type=location_type_name,
status=location_status_name,
)
self.add(location_model)
location_type_model.add_child(location_model)
location_type_count = len(self.dict()["diffsync_location_type_model"])
self.job.log_info(
message=f"Source adapter loaded {location_type_count:,} Location Type records from remote system"
)
location_count = len(self.dict()["diffsync_location_model"])
self.job.log_info(
message=f"Source adapter loaded {location_count:,} Location records from remote system"
)
Notice that we have added a second model (LocationDiffSyncModel) to our adapter. Since LocationDiffSyncModel is listed as a child of LocationTypeDiffSyncModel, then we do not add it to the top_level list.
The load method has expanded to process the additional data that was previously ignored from our source json file.
Now, modify the LocationTargetAdapter to match the code snippet below.
class LocationTargetAdapter(DiffSync):
"""Nautobot DiffSync Adapter for Locations."""
diffsync_location_type_model = LocationTypeDiffSyncModel
diffsync_location_model = LocationDiffSyncModel
top_level = ["diffsync_location_type_model"]
def __init__(self, *, job=None, **kwargs):
self.job = job
super().__init__(**kwargs)
def load(self):
queryset = LocationType.objects.only("name")
for obj in queryset:
name = obj.name
location_type_model = self.diffsync_location_type_model(name=name)
self.add(location_type_model)
queryset_count = queryset.count()
self.job.log_info(
message=f"Target adapter loaded {queryset_count:,} Location Type records from local system"
)
queryset = Location.objects.only(
"name", "description", "location_type", "status"
)
for obj in queryset:
name = obj.name
description = obj.description
status = obj.status.slug
location_type_name = obj.location_type.name
location_model = self.diffsync_location_model(
name=name,
description=description,
location_type=location_type_name,
status=status,
)
self.add(location_model)
location_type_model = self.get(
obj=self.diffsync_location_type_model, identifier=location_type_name
)
location_type_model.add_child(location_model)
queryset_count = queryset.count()
self.job.log_info(
message=f"Target adapter loaded {queryset_count:,} Location records from local system"
)
Changes to this adapter are similar to previous changes: we add a second DiffSync model and modify the load method to read data from the Location objects.
Step 3-3 - Modify jobs.py file¶
Open up the jobs.py file in your editor. We will modifying the data_mappings method of the LocationSync class.
@classmethod
def data_mappings(cls):
location_type_data_map = DataMapping(
"Location Type data upload file (remote)",
None,
"Location Types (local)",
reverse("dcim:locationtype_list"),
)
location_data_map = DataMapping(
"Location data upload file (remote)",
None,
"Locations (local)",
reverse("dcim:location_list"),
)
return (location_type_data_map, location_data_map)
Output of the data_mappings method is displayed on the detail view page for the SSoT job. To see this, navigate to the Plugins menu in Nautobot UI and select Dashboard under Single Source of Truth. Then click on the link for Location Data Source.
Note: Be sure to save all of the above files before continuing.
Step 3-4 - Run the updated SSoT DataSource job¶
Once again we need to restart the Nautobot workers so they can read in the latest changes to the code. As a reminder, you can use the same password that you used for your ssh login.
nautobot@nautobot:~$ systemctl restart nautobot-worker
==== AUTHENTICATING FOR org.freedesktop.systemd1.manage-units ===
Authentication is required to restart 'nautobot-worker.service'.
Authenticating as: ntc
Password:
==== AUTHENTICATION COMPLETE ===
nautobot@nautobot:~$
Before running the job again, take a look at the LocationType and Location list views with Nautobot. At this point you should see three LocationType objects and no Location objects.
Let's run the job again using the same file that we uploaded previously. Be sure to uncheck the Dry run checkbox.
Step 3-5 - Simulate changing data¶
We can tell form the previous results that our job is able to create new Location objects, but can it update or delete existing objects based on changes to the source data? Let's find out.
Create another new file on your workstation where you are running your browser. Give the file a name a name like locations-updated.json and add the following content to the file.
{
"Call Center": [
{"name": "Charlotte", "status": "active"},
{"name": "Fort Worth", "status": "active"}
],
"Data Center": [
{"name": "Jersey City", "status": "retired"},
{"name": "Reston", "status": "active"},
{"name": "San Jose", "status": "active"}
],
"Remote Office": [
{"name": "Atlanta", "status": "active", "description": "VIP location"},
{"name": "Chicago", "status": "active"},
{"name": "Pittsburgh", "status": "active"},
{"name": "Trenton", "status": "planned"},
{"name": "Zebulon", "status": "planned"}
]
}
Notice the following changes have been made to the source data:
- A description has been added to the Atlanta office.
- Pittsburgh office status has changed from
plannedtoactive. - Both the Trenton and Zebulon offices have been added with the
plannedstatus. - The New York office has been removed from the data set.
We expect changes #1 and #2 to trigger the update method for the description and status fields, respectively. Change #3 should trigger the create method twice and change #4 should trigger the delete method.
Now run the job again with the newly created locations-updated.json. Uncheck the Dry run checkbox. Check the job logs to see that the results are as expected.
Check the Locations list view in Nautobot to verify that the expected changes are reflected in the view.
The Sync job we created keeps Nautobot perfectly in sync with the external data source, for those models and attributes that we have defined.
Try the following to verify the data sync:
- Run the Sync job again with the
locations-updated.jsonfile. - You should see no changes to the data in Nautobot.
- Run the Sync job again with the original data file.
- You should see that Nautobot is completely restored to the previous
Locationobjects.
Step 3-6 - Commit Your Changes¶
Remember to commit your changes to git:
nautobot@nautobot:~$
nautobot@nautobot:plugin/maintenance_notices$ git add -A
nautobot@nautobot:plugin/maintenance_notices$ git commit -m "Completed Lab 11."
This lab is now complete.