Skip to content

Netutils

Warning

This requires the following dependencies and has been tested at these specified versions: 

requests==2.31.0
netutils==1.5.0
PyYAML==6.0.1

Netutils overview

Netutils is a library created by Network to Code that has a series of functions that are aggregated in a single location. The intention is to find things that network engineers commonly have to do in their job and provide them in one place. The high-level buckets include:

Functions are grouped with like functions, such as IP or MAC address based functions. Included to date are groupings of:

  • ASN - Provides the ability to convert BGP ASN from integer to dot notation and back.
  • Bandwidth - Provides the ability to convert between various bandwidth values.
  • Banner - Provides the ability to normalize the various banner delimiters.
  • Configuration
    • Cleaning - Provides the ability to remove or replace lines based on regex matches.
    • Compliance - Provides the ability to compare two configurations to sanely understand the differences.
    • Conversion - Provides the ability to convert between different syntax's within the same OS.
    • Parsing - Provides the ability to parse configuration for the minor differences that are there.
  • DNS - Provides the ability to work with DNS, such as validating that a FQDN is resolvable.
  • Interface - Provides the ability to work with interface names, expanding, abbreviating, and splitting the names.
  • IP Address - Provides the ability to work with IP addresses, primarily exposing Python ipaddress functionality.
  • Library Helpers - Provides helpers to pull useful information, e.g. NAPALM getters.
  • Library Mapper - Provides mappings in expected vendor names between Netmiko, NAPALM, pyntc, ntc-templates, pyats, and scrapli.
  • MAC Address - Provides the ability to work with MAC addresses such as validating or converting to integer.
  • OS Version - Provides the ability to work with OS version, such as defining an upgrade path.
  • Password - Provides the ability to compare and encrypt common password schemas such as type5 and type7 Cisco passwords.
  • Ping - Provides the ability to ping, currently only tcp ping.
  • Protocol Mapper - Provides a mapping for protocol names to numbers and vice versa.
  • Regex - Provide convenience methods for regex to be used in Jinja2.
  • Route - Provides the ability to provide a list of routes and an IP Address and return the longest prefix matched route.
  • Time -Provides the ability to convert between integer time and string times.
  • VLANs - Provide the ability to convert configuration into lists or lists into configuration.

The primary mechanism of documentation is within the form of docstrings on the functions that provide examples for how to use them. These functions are tested via doctest, and thus ensured to be accurate.

Config Parsing

from netutils.config.parser import IOSConfigParser

raw_config = """!
interface Loopback0
 ip address 10.0.10.3 255.255.255.255
!
interface GigabitEthernet1
 description MANAGEMENT_DO_NOT_CHANGE
 ip address 10.0.0.15 255.255.255.0
 negotiation auto
 no mop enabled
 no mop sysid
!
router bgp 65250
  router-id 10.0.10.4
  log-neighbor-changes
  neighbor 10.10.10.6
    remote-as 65250
    address-family ipv4 unicast
!
banner exec ^C
**************************************************************************
* IOSv is strictly limited to use for evaluation, demonstration and IOS  *
* education. IOSv is provided as-is and is not supported by Cisco's      *
* Technical Advisory Center. Any use or disclosure, in whole or in part, *
* of the IOSv Software or Documentation to any third party for any       *
* purposes is expressly prohibited except as otherwise authorized by     *
* Cisco in writing.                                                      *
**************************************************************************^C
!
"""

parsed_config = IOSConfigParser(raw_config)

The list of items in property of config_lines are each NamedTuple's and as such you can access their attributes via the .config_line/.parents or via a dictionary with ._asdict().

for line in parsed_config.config_lines:
    if "interface GigabitEthernet1" in line.parents:
        print(line.config_line)

for line in parsed_config.config_lines:
    if "interface GigabitEthernet1" in line._asdict()['parents']:
        print(line._asdict()['config_line'])

One of the key components of parsers is the ability to identify the configuration based on their parents. We have already observed the the first use case, of a single parent. Let's observe how multiple parents work.

for line in parsed_config.config_lines:
    if "router bgp 65250" in line.parents and "  neighbor 10.10.10.6" in line.parents:
        print(line.config_line)

Notice that the secondary parent contains spaces, as in the configuration. Alternatively you can get configurations with the find_all_children method. 

parsed_config.find_all_children(pattern="router bgp 65250")

The parser needs to handle abnormal configurations, most notably is the banner. Instead of having a single line per configuration, the banner is rolled up into a single item in the config_lines attribute.

>>> raw_config = """!
... banner exec ^C
... **************************************************************************
... * IOSv is strictly limited to use for evaluation, demonstration and IOS  *
... * education. IOSv is provided as-is and is not supported by Cisco's      *
... * Technical Advisory Center. Any use or disclosure, in whole or in part, *
... * of the IOSv Software or Documentation to any third party for any       *
... * purposes is expressly prohibited except as otherwise authorized by     *
... * Cisco in writing.                                                      *
... **************************************************************************^C
... !
... """
>>>
>>> parsed_config = IOSConfigParser(raw_config)
>>> parsed_config.config_lines[1]
ConfigLine(config_line="**************************************************************************\n* IOSv is strictly limited to use for evaluation, demonstration and IOS  *\n* education. IOSv is provided as-is and is not supported by Cisco's      *\n* Technical Advisory Center. Any use or disclosure, in whole or in part, *\n* of the IOSv Software or Documentation to any third party for any       *\n* purposes is expressly prohibited except as otherwise authorized by     *\n* Cisco in writing.                                                      *\n**************************************************************************^C", parents=('banner exec ^C',))
>>>

Parsing Errors

There are several challenges with parsing the configuration. The configurations are only intended to be configurations that show up in the show run or equivalent. If as an example, the below impossible configuration is used, it will error out.

>>> from netutils.config.parser import IOSConfigParser
>>>
>>> raw_config = """!
... ntp server 1.1.1.1
... ntp server 1.1.1.1
... !
... """
>>> parsed_config = IOSConfigParser(raw_config)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.11/site-packages/netutils/config/parser.py", line 585, in __init__
    super(IOSConfigParser, self).__init__(config)
  File "/usr/local/lib/python3.11/site-packages/netutils/config/parser.py", line 520, in __init__
    super(CiscoConfigParser, self).__init__(config)
  File "/usr/local/lib/python3.11/site-packages/netutils/config/parser.py", line 64, in __init__
    super(BaseSpaceConfigParser, self).__init__(config)
  File "/usr/local/lib/python3.11/site-packages/netutils/config/parser.py", line 31, in __init__
    self.build_config_relationship()
  File "/usr/local/lib/python3.11/site-packages/netutils/config/parser.py", line 655, in build_config_relationship
    self._update_same_line_children_configs()
  File "/usr/local/lib/python3.11/site-packages/netutils/config/parser.py", line 607, in _update_same_line_children_configs
    previous_line = new_config_lines[-1]
                    ~~~~~~~~~~~~~~~~^^^^
IndexError: list index out of range
>>>

Another issue is an improper formatted banner configuration. 

>>> raw_config = """!
... banner exec ^C
... **************************************************************************
... * IOSv is strictly limited to use for evaluation, demonstration and IOS  *
... * education. IOSv is provided as-is and is not supported by Cisco's      *
... * Technical Advisory Center. Any use or disclosure, in whole or in part, *
... * of the IOSv Software or Documentation to any third party for any       *
... * purposes is expressly prohibited except as otherwise authorized by     *
... * Cisco in writing.                                                      *
... **************************************************************************
... !
... """
>>>
>>> parsed_config = IOSConfigParser(raw_config)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.11/site-packages/netutils/config/parser.py", line 585, in __init__
    super(IOSConfigParser, self).__init__(config)
  File "/usr/local/lib/python3.11/site-packages/netutils/config/parser.py", line 520, in __init__
    super(CiscoConfigParser, self).__init__(config)
  File "/usr/local/lib/python3.11/site-packages/netutils/config/parser.py", line 64, in __init__
    super(BaseSpaceConfigParser, self).__init__(config)
  File "/usr/local/lib/python3.11/site-packages/netutils/config/parser.py", line 31, in __init__
    self.build_config_relationship()
  File "/usr/local/lib/python3.11/site-packages/netutils/config/parser.py", line 654, in build_config_relationship
    super(IOSConfigParser, self).build_config_relationship()
  File "/usr/local/lib/python3.11/site-packages/netutils/config/parser.py", line 311, in build_config_relationship
    line = self._build_banner(line)  # type: ignore
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/netutils/config/parser.py", line 600, in _build_banner
    return super(IOSConfigParser, self)._build_banner(config_line)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/netutils/config/parser.py", line 540, in _build_banner
    return super(CiscoConfigParser, self)._build_banner(config_line)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/netutils/config/parser.py", line 228, in _build_banner
    raise ValueError("Unable to parse banner end.")
ValueError: Unable to parse banner end.
>>>

Testing Mock Strategy

The mock data strategy is to take configurations that are provided in raw format and ensure the resulting data is the same. This strategy is used in several libraries:

  • NTC Templates
  • NAPALM
  • Netutils

The pattern is essentially to glob the raw files and the respective mock results files. You can view how this accomplished in this code snippet.

import os

import pytest
from netutils.config import compliance

MOCK_DIR = os.path.join(os.path.dirname(os.path.realpath(__file__)), "mock", "config", "parser")
TXT_FILE = "_sent.txt"

base_parameters = []
find_all_children_parameters = []
find_children_w_parents_parameters = []
for network_os in list(compliance.parser_map.keys()):
    for _file in glob.glob(f"{MOCK_DIR}/base/{network_os}/*{TXT_FILE}"):
        base_parameters.append([_file, network_os])
    for _file in glob.glob(f"{MOCK_DIR}/find_all_children/{network_os}/*{TXT_FILE}"):
        find_all_children_parameters.append([_file, network_os])
    for _file in glob.glob(f"{MOCK_DIR}/find_children_w_parents/{network_os}/*{TXT_FILE}"):
        find_children_w_parents_parameters.append([_file, network_os])


@pytest.mark.parametrize("_file, network_os", base_parameters)
def test_parser(_file, network_os, get_text_data, get_python_data):  # pylint: disable=redefined-outer-name
    truncate_file = os.path.join(MOCK_DIR, "base", _file[: -len(TXT_FILE)])

    device_cfg = get_text_data(os.path.join(MOCK_DIR, "base", _file))
    received_data = get_python_data(truncate_file + "_received.py", "data")
    os_parser = compliance.parser_map[network_os]
    assert os_parser(device_cfg).config_lines == received_data

Config Compliance

The configuration compliance has the ability to compare configuration from one configuration to another. Generally, the intended configuration vs the actual configuration.

The set of features is defined by matching the configurations that startswith() in the root key (meaning, not nested configuration), and their respective children.

  • ntp - would match all configuration starting with ntp, such as ntp server and ntp authenticate.
  • interface - would match all interface configuration, including nested configuration.
  • router - would match all router protocol configuration, including router ospf and router bgp, including nested configuration.

The feature must also indicate that the configuration is to be ordered or not. Take for example access lists, the ordering matters, whereas ntp configurations, ordering is irrelevant.

The operating systems are mapped to the parsers, which supports the compliance process.

>>> from netutils.config.compliance import compliance
>>>
>>> features = [
...     {
...         "name": "hostname",
...         "ordered": True,
...         "section": [
...             "hostname"
...         ]
...     },
...     {
...         "name": "bgp",
...         "ordered": True,
...         "section": [
...             "router bgp"
...         ]
...     },
...     {
...         "name": "ntp",
...         "ordered": True,
...         "section": [
...             "ntp"
...         ]
...     }
... ]
>>>
>>> backup = "ntp server 192.168.1.1\nntp server 192.168.1.2 prefer\nhostname nyc-rt01"
>>>
>>> intended = "ntp server 192.168.1.1\nntp server 192.168.1.5 prefer\nhostname nyc-rt01"
>>>
>>> network_os = "cisco_ios"
>>>
>>> compliance(features, backup, intended, network_os, "string")
{'hostname': {'compliant': True,
  'missing': '',
  'extra': '',
  'cannot_parse': True,
  'unordered_compliant': True,
  'ordered_compliant': True,
  'actual': 'hostname nyc-rt01',
  'intended': 'hostname nyc-rt01'},
 'bgp': {'compliant': True,
  'missing': '',
  'extra': '',
  'cannot_parse': True,
  'unordered_compliant': True,
  'ordered_compliant': True,
  'actual': '',
  'intended': ''},
 'ntp': {'compliant': False,
  'missing': 'ntp server 192.168.1.5 prefer',
  'extra': 'ntp server 192.168.1.2 prefer',
  'cannot_parse': True,
  'unordered_compliant': False,
  'ordered_compliant': False,
  'actual': 'ntp server 192.168.1.1\nntp server 192.168.1.2 prefer',
  'intended': 'ntp server 192.168.1.1\nntp server 192.168.1.5 prefer'}}
>>>

As you will note, the response provided shows:

Configuration state:

  • actual - The actual configuration.
  • intended - The intended configuration.
  • missing - Configuration that should be on the actual configuration but is not.
  • extra - Configuration that is on the actual configuration but should not be.

Compliant state:

  • compliant - Whether or not it was compliant.
  • unordered_compliant - Whether or not it was compliant but unordered between the configs.
  • ordered_compliant- Whether or not it was compliant and ordered the same.

Errors:

  • cannot_parse - Set if there was a parsing issue.

Configuration Compliance - LAB

Generate configurations that that demonstrate each of the following:

  1. compliant being true
  2. compliant being false
  3. ordered_compliant being false and compliant being true on the same instance
  4. missing configuration provided
  5. extra configuration provided

The below should work if you assign each of these to a variable named as compliance_<test_num> and you use ntp configuration for each test.

assert compliance_1["ntp"]["compliant"] == True
assert compliance_2["ntp"]["compliant"] == False
assert compliance_3["ntp"]["ordered_compliant"] == False & compliance_3["ntp"]["compliant"] == True
assert compliance_4["ntp"]["missing"] != ""
assert compliance_5["ntp"]["extra"] != ""

Review functions

https://netutils.readthedocs.io/en/latest/dev/code_reference/

def asn_to_int(asplain: str) -> int:
    """Convert AS Number to standardized asplain notation as an integer.

    Args:
        asplain: An `asplain` notated BGP ASN with community.

    Returns:
        Integer value within of the given asplain value provided.

    Examples:
        >>> from netutils.asn import asn_to_int
        >>> asn_to_int("65000")
        65000
        >>> asn_to_int("65000.111")
        4259840111
        >>>
    """
    # ASN is in asdot notation
    if "." in asplain:
        asn_list = asplain.split(".")
        asn = int(f"{int(asn_list[0]):016b}{int(asn_list[1]):016b}", 2)
        return asn
    return int(asplain)

Challenge Lab - 1

  • Determine the routes that are within the path
    • Hint: You will need to convert the prefix_length to subnet mask
  • Provide the interface that the route matches on, in shortened format.
  • Provide the range of IPs that the IP is in when matched

After using this started file, the output should be able to use an f-string like:

f"IP {ip} found in range of {ip_range} on interface {interface}"

With the output like:

IP 10.0.1.10 found in range of 10.0.1.1 - 10.0.1.254 on interface Se0/0

Leverage this starter script. 

import requests
import yaml

# URL to the YAML file
url = "https://raw.githubusercontent.com/networktocode/ntc-templates/master/tests/cisco_ios/show_ip_route/cisco_ios_show_ip_route.yml"

route_data = yaml.safe_load(requests.get(url).text)["parsed_sample"]

ip_data = ["10.0.1.125", "10.0.5.125", "192.168.12.125", "10.63.187.10", "2.2.2.0"]

Challenge Lab - 2

  • Convert mac addresses to format of "aa:aa:aa:aa:aa"
  • Build the uplink vlan configuration you would need to support all vlans programmatically

The output should be a list of mac addresses and the vlan configuration snippets.

import requests
import yaml

# URL to the YAML file
url = "https://raw.githubusercontent.com/networktocode/ntc-templates/master/tests/arista_eos/show_mac_address-table/arista_eos_show_mac_address-table.yml"

mac_data = yaml.safe_load(requests.get(url).text)["parsed_sample"]