Nautobot Apps - Lab 08: Creating Jobs¶
This lab will walk through how to create a custom Job in Nautobot, which allows code to interact directly with Nautobot data. It's important to note that these jobs are completely outside of the Nautobot code base and can be modified without affecting Nautobot.
The possibilities of custom jobs are virtually infinite. This can become a challenge when implementing a job, as it's very easy to include a lot of business logic into one script. However, it's good practice to have a well defined scope, with a clear goal in mind for what the job is going to accomplish. In the end, jobs are just code and it should be treated as such. Think of it as a function that can be broken up into several smaller functions. It will be easier to read and maintain in the long run.
- Nautobot Apps - Lab 08: Creating Jobs
- Task 1: Create the Jobs File
- Task 2: Create a Job
- Task 3: Create
post_run()function.
Task 1: Create the Jobs File¶
For our example job, we will query the database with a list of devices and a specific date. Our job will determine if any of the devices identified are associated with a maintenance event on this date.
Step 1-0 - Prepare for the Lab¶
For this and all remaining labs, it is advisable to open two terminal windows, one for running the Nautobot development server, and another for executing various commands.
The steps in this lab require that you have a terminal open and have switched to the nautobot user. Verify your username with the whoami command and switch user to nautobot if needed.
ntc@nautobot:~$ whoami
ntc
ntc@nautobot:~$ sudo -iu nautobot
nautobot@nautobot:~$ whoami
nautobot
nautobot@nautobot:~$
If you have the Nautobot development server running from a previous lab, please continue to the next step. Otherwise we need to start the server with the command nautobot-server runserver 0.0.0.0:8080 --insecure.
nautobot@ntc-nautobot-apps:~$ nautobot-server runserver 0.0.0.0:8080 --insecure
Watching for file changes with StatReloader
Performing system checks...
System check identified no issues (0 silenced).
April 24, 2022 - 05:19:10
Django version 3.1.14, using settings 'nautobot_config'
Starting development server at http://0.0.0.0:8080/
Quit the server with CONTROL-C.
Step 1-1 - Create the jobs.py File¶
Create a jobs.py file at the root of our plugin directory. This should be at the same level as our models.py file. For Nautobot App developers, it's important to understand the different structure of how jobs are brought into Nautobot once the application is installed. Review the documentation for Jobs in Nautobot Apps.
nautobot@nautobot:~$ pwd
/opt/nautobot
nautobot@nautobot:~$ cd /opt/nautobot/plugin/maintenance_notices
nautobot@nautobot:~/plugin/maintenance_notices$ touch jobs.py
Nautobot provides the ability to import jobs from a GitRepository. However, during this lab we will not be using this approach.
Step 1-2 - Understand Variables¶
Variables are not a requirement for creating a job, but they do allow for total flexibility in how the Job is executed.
There are several types of variable options available:
StringVar: String textTextVar: Arbitrary text of any length, renders as multi-line text input field.IntegerVar: Stores a numeric integer.BooleanVar: True/FalseChoiceVar: A set of choices as a list.MultiChoiceVar: Similar toChoiceVarbut allows multiple choices to be selected.ObjectVar: Select a object from NautobotMultiObjectVar: Similar toObjectVarbut allows multiple objects to be selected.FileVar: Upload a file.IPAddressVar: IPv4/6 without a mask.IPAddressWithMaskVar: IPv4/6 with a mask.IPNetworkVar: IPv4/6 Network with a mask.
To get more details on the available options please visit the official documentation for Default Variable Options
Task 2: Create a Job¶
Step 2-1 - Write the Job Code¶
Using VSCode, open the newly created jobs.py file.
Each job class will implement some or all of the following components:
- Module and class attributes, mostly for documentation and human readability a set of variables for user input via the Nautobot UI (if your job requires any user inputs)
- A
run()method, which is executed first and receives the user input values, if any number of test_*() methods, which will be invoked next in order of declaration. Log messages generated by the job will be grouped together by the test method they were invoked from. - A
post_run()method, which is executed last and can be used to handle any necessary cleanup or final events (such as sending an email or triggering a webhook). The status of the overall job is available at this time as self.failed and the JobResult data object is available as self.result.
Import the job class and create a new class that inherits from Job. Assign Job Metadata as well to include, name, description and read_only boolean value. Additionally, we will add a module level attribute name to define the display name for the collection of job that can be defined under this python file, in the Nautobot UI.
"""Maintenance Jobs."""
from nautobot.extras.jobs import Job
name = "Maintenance Event Jobs"
class CheckDeviceMaintenanceEvents(Job):
"""Job that validates if a given date has an associated maintenance event."""
class Meta:
"""Meta class for CheckDeviceMaintenanceEvents"""
name = "Check Device Maintenance Events"
description = "Verify if a maintenance event is associated to a device for a given date."
read_only = True
def run(self, data=None, commit=None):
"""Executes the job"""
pass
Step 2-2 - Define the Variables¶
We know the scope of our job at this point. The goal is define if a device is impacted on any given date by any maintenance that we are aware of. This gives us an idea of what options are necessary, such as Devices and a given Date as our variables. To make this simple we will introduce three IntegerVar fields to capture Year, Month and Day.
Import IntegerVar and MultiObjectVar from nautobot.extras.jobs (Same place the initial Job class is from). We will also need to import the core model, Device from nautobot.
"""Maintenance Jobs."""
from nautobot.extras.jobs import Job, IntegerVar, MultiObjectVar
from nautobot.dcim.models import Device
from .models import MaintenanceNotice
name = "Maintenance Event Jobs"
class CheckDeviceMaintenanceEvents(Job):
"""Job that validates if a given date has an associated maintenance event."""
devices = MultiObjectVar(model=Device)
year = IntegerVar()
month = IntegerVar()
day = IntegerVar()
class Meta:
Step 2-3 - Get the Variables¶
It's not uncommon for jobs to have functions that make up the run function. This makes it easy to maintain, read and extend. For this simple use case, we will have all our logic inside the run function.
Inside the run function, lets use the devices variable inputted by the user. It's not a requirement to do this, but it's an example of how to access them within a job.
Gather variables within the run function. This method should be added after the class Meta: code block.
def run(self, data=None, commit=None):
"""Executes the job"""
devices = data["devices"]
year = data["year"]
month = data["month"]
day = data["day"]
Step 2-4 - Generate Results¶
With the variables ready for us to use, we will simply filter through our available MaintenanceNotice objects in the database that have been filtered down by year, month and day. Once we have this list, we will access the associated devices from the many-to-many relationship in our data model. If any of the devices that we have passed into the job as user input are found within the associated devices to the maintenance, we have found an impacted device and will add it to our list of results.
Import the MaintenanceNotice model from our models.py file at the top of our jobs.py file
Below the variables inside the run method, add the logic to create the results
def run(self, data=None, commit=None):
"""Executes the job"""
devices = data["devices"]
year = data["year"]
month = data["month"]
day = data["day"]
result = []
for maintenance in MaintenanceNotice.objects.filter(
start_time__year=year,
start_time__month=month,
start_time__day=day,
):
for device in devices:
if not maintenance.devices.filter(name=device.name).exists():
continue
result.append(device)
self.log_warning(
obj=device,
message=f"Impacted for {maintenance.duration} Minutes",
)
if not result:
self.log_success(message=f"No devices impacted on this date.")
return
self.log_success(message=f"Job complete. A total of {len(result)} devices are impacted on this date.")
For now, this is enough foundation for a helpful job that can provide detailed reporting with just a few lines of python code. These read only jobs can become beneficial within cross functioning teams to be aware of any events happening that are impacting specific devices of interest and for how long without ever needing to contact the network or change management team.
Step 2-5 - Register the Job¶
Nautobot will look inside of the jobs.py file and specifically search for an iterable (a list in this case) named jobs. We must register our job classes inside of this iterable.
At the end of the jobs.py file, add the following line
Step 2-6 - Run database migrations¶
In order for jobs to show up we must run database migrations.
nautobot@nautobot-1:~$ nautobot-server migrate
Operations to perform:
Apply all migrations: admin, auth, circuits, contenttypes, database, dcim, django_celery_beat, django_rq, extras, ipam, maintenance_notices, nautobot_golden_config, sessions, social_django, taggit, tenancy, users, virtualization
Running migrations:
No migrations to apply.
00:09:59.553 INFO nautobot.extras.utils :
Refreshed Job "Golden Configuration: Backup Configurations" from <plugins: BackupJob>
00:09:59.558 INFO nautobot.extras.utils :
Refreshed Job "Golden Configuration: Generate Intended Configurations" from <plugins: IntendedJob>
00:09:59.562 INFO nautobot.extras.utils :
Refreshed Job "Golden Configuration: Perform Configuration Compliance" from <plugins: ComplianceJob>
00:09:59.566 INFO nautobot.extras.utils :
Refreshed Job "Golden Configuration: Execute All Golden Configuration Jobs - Single Device" from <plugins: AllGoldenConfig>
00:09:59.570 INFO nautobot.extras.utils :
Refreshed Job "Golden Configuration: Execute All Golden Configuration Jobs - Multiple Device" from <plugins: AllDevicesGoldenConfig>
00:09:59.574 INFO nautobot.extras.utils :
Refreshed Job "Maintenance Event Jobs: Check Device Maintenance Events" from <plugins: CheckDeviceMaintenanceEvents>
nautobot@nautobot-1:~$
Step 2-7 - Restart the Nautobot Server and Worker Service¶
Restart the nautobot-server so that it loads the jobs.py file.
The Nautobot UI will render and register the job, but the worker service needs to be restarted in order to actually execute the job.
You can restart the nautobot-worker service as the nautobot user and authenticate with the ntc credentials used to ssh into the server. Or you can exit back to the ntc user, run the command and switch back to the nautobot user.
nautobot@nautobot:~$ systemctl restart nautobot-worker
==== AUTHENTICATING FOR org.freedesktop.systemd1.manage-units ===
Authentication is required to restart 'nautobot-worker.service'.
Authenticating as: ntc
Password:
==== AUTHENTICATION COMPLETE ===
nautobot@nautobot:~$
Now, head on over to the Nautobot UI -> Jobs -> Jobs. Once the page loads, you should see a new Job added to the list.
Step 2-8 - Enable the job¶
Newly created jobs are not enabled by default. From the Nautobot UI -> Jobs -> Jobs page, click the yellow button on the Check Device Maintenance Events Job. This will take you to job's setting page where you can enable the job.
Enable the job and save by clicking the update button at the bottom of the page.
Step 2-9 - Running the job¶
Back on the Nautobot UI -> Jobs -> Jobs page, you can now click the blue play button. This will take you to the job Click on the job and you will be taken to the form entry to execute the job with user inputted variables. Fill in the form and locate a device that will be impacted on a given date!
Example Job Output
Task 3: Create post_run() function.¶
Let's take advantage of the post_run() method to create a CSV file of the data found during our job run. In this example we will just be writing the file to the disk. However, we could easily send this file to a shared folder or another appliance to consume the data.
Step 3-1 - Checkpoint¶
Before we begin, make sure your models.py file matches the output below. This should be how we left it off in Lab 03.
"""Model definition for maintenance_notices."""
from datetime import timedelta
from django.contrib.auth import get_user_model
from django.db import models
from django.shortcuts import reverse
from nautobot.core.models import BaseModel
class MaintenanceNotice(BaseModel):
UserModel = get_user_model()
start_time = models.DateTimeField()
end_time = models.DateTimeField(editable=False)
duration = models.PositiveSmallIntegerField(help_text="Duration (in minutes)")
comments = models.CharField(blank=True, max_length=200)
created_by = models.ForeignKey(
to=UserModel, on_delete=models.SET_NULL, blank=True, null=True, editable=False
)
devices = models.ManyToManyField(
to="dcim.Device", related_name="maintenance_notices"
)
class Meta:
ordering = ("start_time", "pk")
verbose_name = "Maintenance Notice"
verbose_name_plural = "Maintenance Notices"
def __str__(self):
return f"{self.start_time:%Y-%m-%d %H:%M} ({self.duration} minutes)"
def get_absolute_url(self):
"""Return absolute URL for instance."""
return reverse("plugins:maintenance_notices:maintenancenotice", args=[self.pk])
def save(self, *args, **kwargs):
self.end_time = self.start_time + timedelta(minutes=self.duration)
super().save(*args, **kwargs)
Step 3-2 - Update models.py¶
Nautobot provides some constructs around converting a defined model to csv. To take advantage of that, you need to add a few lines of code to your models.py file. You need to first define a class variable named csv_headers. This will tell us what headers to use when dealing with the Maintenance Notice model. Here we are defining headers that directly correlate with the models attributes.
...
class MaintenanceNotice(BaseModel):
csv_headers = ["Start Time", "End Time", "Devices", "Duration", "Comments"]
UserModel = get_user_model()
...
You also need to define a function called to_csv(). This helper function defines what attributes of a Maintenance Notice we are interesting in working with. You'll notice we have some logic to gather a string of the devices associated with a Maintenance Notice. If we just returned self.devices we'd get a django QuerySet back instead of an actual list of the devices.
...
def get_absolute_url(self):
"""Return absolute URL for instance."""
return reverse("plugins:maintenance_notices:maintenancenotice", args=[self.pk])
def to_csv(self):
"""Indicates model fields to return as csv."""
device_list = [device.get("name") for device in self.devices.values("name")]
device_comma_separated_string = ",".join(device_list)
return (
self.start_time,
self.end_time,
device_comma_separated_string,
self.duration,
self.comments if self.comments else "",
)
...
Step 3-3 - Update jobs.py run() method¶
Add the self.data = data line into our run() method.
...
def run(self, data=None, commit=None):
"""Executes the job"""
devices = data["devices"]
year = data["year"]
month = data["month"]
day = data["day"]
self.data = data
...
As mentioned at the beginning of this lab, the post_run() method is executed after the run() method. One thing to be aware of is that the post_run() method cannot accept any parameters. Because of this, if we want to use passed in data from the GUI in our post_run() method we need to assign that data to the class variable data in the run() method.
Step 3-4 - Add jobs.py post_run() method¶
Next, you'll add our post_run() function.
from nautobot.utilities.utils import csv_format
...
def post_run(self):
"""Create CSV file and save on disk."""
devices = self.data["devices"]
year = self.data["year"]
month = self.data["month"]
day = self.data["day"]
csv_data = []
headers = MaintenanceNotice.csv_headers.copy()
csv_data.append(",".join(headers))
for maintenance in MaintenanceNotice.objects.filter(start_time__year=year, start_time__month=month, start_time__day=day):
for device in devices:
if not maintenance.devices.filter(name=device.name).exists():
continue
data = maintenance.to_csv()
csv_data.append(csv_format(data))
with open("maintenance_notice_job_results.csv", "w") as csv_file:
for line in "\n".join(csv_data):
csv_file.write(line)
self.log_success(message=f"Excel sheet created in post_run()")
jobs = [CheckDeviceMaintenanceEvents]
Note the new import at the top of the code example
from nautobot.utilities.utils import csv_format
Let's break this down some. The first four lines do the same thing as the first four lines in run(). However, because we are in the post_run() method, we have to get the information we want from the class variable self.data.
The next three lines utilize the variable csv_headers that we created on our model. First, you create an empty list which is used to track the data we will be writing to the csv. Next you grab those headers you defined on the Maintenance Notice model in the previous step. Lastly, you take those headers and add them as an entry to our csv_data list.
The next part uses the same logic used in the run() method. The difference is what we do with the matching data. For every maintenance item that exists for the device you got from the user you call the to_csv() method and store the returned information into the data vriable and append it to our csv_data list.
for maintenance in MaintenanceNotice.objects.filter(start_time__year=year, start_time__month=month, start_time__day=day):
for device in devices:
if not maintenance.devices.filter(name=device.name).exists():
continue
data = maintenance.to_csv()
csv_data.append(csv_format(data))
Lastly, we write the data to a csv file called maintenance_notice_job_results.csv and send a log message to the GUI.
Note: Depending on what Maintenance Notices you created, you'll notice that the devices column contains all the devices a certain Maintenance Notice is associated with.
Step 3-5 - Run the updated job¶
We need to restart the worker like we did in step 2-6. We do not have to do database migrations since we are editing an existing job.
You can restart the nautobot-worker service as the nautobot user and authenticate with the ntc credentials used to ssh into the server. Or you can exit back to the ntc user, run the command and switch back to the nautobot user.
nautobot@nautobot:~$ systemctl restart nautobot-worker
==== AUTHENTICATING FOR org.freedesktop.systemd1.manage-units ===
Authentication is required to restart 'nautobot-worker.service'.
Authenticating as: ntc
Password:
==== AUTHENTICATION COMPLETE ===
nautobot@nautobot:~$
Now, head on over to the Nautobot UI -> Jobs. Once the page loads, you should see a new Job added to the list.
Click on the job and you will be taken to the form entry to execute the job with user inputted variables. Fill in the form and locate a device that will be impacted on a given date!
Example Job Output
You should now see your maintenance_notice_job_results.csv file located at /opt/nautobot/maintenance_notice_job_results.csv.
Step 3-6 - Commit Your Changes¶
Remember to commit your changes to git:
nautobot@nautobot:~$ cd ~/plugin/
nautobot@nautobot:~/plugin$ git status
On branch master
Untracked files:
(use "git add <file>..." to include in what will be committed)
maintenance_notices/jobs.py
nothing added to commit but untracked files present (use "git add" to track)
nautobot@nautobot:plugin$ git add -A
nautobot@nautobot:plugin$ git commit -m "Completed Lab 8."
Summary
As we learned above, creating a job can be very straight forward and only takes a few lines of code. It's important to maintain a well defined scope for a job, in order to have maintainable and easy to read code. There are many types of variable options available for application developers, which give jobs greater flexibility. Ensure to review documentation and example jobs from the Nautobot documentation for more assistance.