Reliable Config Generation with Python

2018-07-02

Config Generation is a quite simple task. I wrote about it several times in past (e.g. here and here). You can use python (with Jinja2 or Mako), Ansible playbooks (which is at the end also Jinja2) and so on and it will work. Whatever, the challenging part isn’t the process of generating the configuration using a template language. From my perspective, it’s also not the rollout of the template. There are also a ton of tools available to push a piece of text to a device using SSH, e.g. using the netmiko library of Kirk Byers.

The critical part: the data

At the end, the challenging part is the data that you are working with. In many cases it must be collected, normalized and cleaned and if they contain incomplete, invalid or missing entries you have a problem. No template engine produces any usable results if the input data is nonsense. You can manually verify every template afterwards (or the data before), but this doesn’t scale very well for the automation pipeline that you’re creating (think of the rollout of 2500 devices…).

The following two options obviously address this problem:

use (or code) something to collect and sanitize the data before you generate your templates
use (or code) something to verify that your device is still working after the template was thrown at the device

Depending on the complexity and variety of your template/device/network, the first option can be associated to a huge amount of work. Furthermore, if the implementation is incomplete (missing verification on a data field), you also run into the same problems as before. Therefore, you need a validation engine to verify the results after the rollout of the device. Sounds right? Well, if the IP address of you device is for whatever reason rubbish, you’re not able to validate the results, the device is broken and at the end the customer isn’t happy…¹

I thought about this for quite some time and came to the conclusion that the template engine should tell me, if something is missing, broken or just not useful. As a side effect, the author of the template can also include some basic checks to avoid the worst case (device unreachable) when using the template. Don’t get me wrong, this won’t prevent all issues, but I think it’s a useful safeguarding to make the process itself more reliable. Such a feature combined with a structured data collection and a validation after the rollout may lead into a powerful automation pipeline for your network.

Reliable Template Engine for Network Engineers

Based on these thoughts, I created a little library in 2017 that allows a more reliable process of generating configurations: networkconfgen. It’s based on Jinja2 and provides an interface for the author of a template to signal that something is wrong with the data. As a little extra, it extends the Jinja2 syntax with some functions that are required to generate configuration files for network devices (e.g. create hostmasks/wildcard masks from a subnetmask or convert a dotted decimal representation to a prefix length and vice versa).

The library is available on the python package index (pypi) and can be installed using the following command:

pip install networkconfgen

The code is available at GitHub and it’s tested with python 2.7 and >3.4.

The simple part: config generation

networkconfgen contains a single class that is used as the configuration generator. The following lines of code create a new instance:

from networkconfgen import NetworkConfGen

confgen = NetworkConfGen()

The initialization process will create a new Jinja2 template environment, register the custom filters and enable the Jinja2 do extension (see the Jinja2 documentation for more details). This extension is quite useful from time to time because it allows the creation of some logic within your templates (create additional variables, add/remove entries from a list etc.).

The following lines of code shows the basic usage of the library with a template as a string:

template = """
!
hostname {{ hostname }}
!
"""
parameters = {
    "hostname": "demo"
}
result = confgen.render_from_string(
    template_content=template, 
    parameters=parameters
)

You can also use a directory with templates, which is also provided by Jinja2. Please visit the networkconfgen README to get further instructions how this can be implemented.

The result object is not a string, it’s an instance of the NetworkConfGenResult class that can be used to get further information about the configuration process. The most basic check, is to verify that there are no issues with the rendering of the template (e.g. Jinja2 syntax errors). The following lines of code show how to verify that there are no rendering errors and how to get the results from Jinja2:

if not result.render_error:
    print(result.template_result)
   
else: 
    print("Something went wrong: %s" % result.error_text)

The error_text variable contains a string representation from the Jinja2 exception if something went wrong. The “raw” template result is stored in the template_result variable. The content can be a little bit confusing depending on your formatting standards within the template (indentation etc.). For this reason, there is a cleaned_template_result() method to produce a more readable output. It will remove 4 consecutive blanks from the lines to provide some readability to the outputs while maintaining the readability in the templates. The following examples show the difference:

Template:

!
{% if something %}
    something is defined
{% endif %}
!
interface Ethernet 0/1
{% if something_else %}
    {# use 5 blanks to maintain readablility in the clean output#}
     ip address dhcp
{% endif %}
!

contents of the template_result:

!
    something is defined
!
interface Ethernet 0/1
     ip address dhcp
!

result of the cleaned_template_result() method:

!
something is defined
!
interface Ethernet 0/1
 ip address dhcp
!

By default, some additional options are enabled within the Jinja2 environment to provide a cleaner response in the template_result (e.g. trim_blocks and lstrip_blocks, see the High Level API in the Jinja2 documentation. It depends on the size and complexity (e.g. nested loops, many conditions) of the template, but in many cases the clean result provides some additional readability to the result.

The critical part: Content Error Checks

Now, let’s take a look on the original issue of this post: the content error checks. As mentioned in the introduction, the main problem that this library should solve is to provide a mechanism to allow the author to signal an error with the data that is used within the template. For this reason, networkconfgen defines certain error strings. After Jinja2 has rendered the template, networkconfgen will check if an error string is part of the output. If this is the case, the content_error flag is set on the NetworkConfGenResult. It’s not a complex feature, but it’s quite useful when creating many configurations from (not so) cleaned data.

The following template example demonstrates the use of the error flags:

Template:

!
hostname {{ hostname|default(_ERROR_.invalid_value)}}
!

>>> result = confgen.render_from_string(template_content=template, parameters={})
>>> result.render_error
False
>>> result.content_error
True

At the end, it’s just comparing strings. To make the templates more readable a _ERROR_ dictionary is added to the template parameters, as shown in the last example. If you provide a hostname as a parameter of the template, everything is fine. If the parameter is missing, the Jinja2 filter |default() will add an error string to the template result and the NetworkConfGenResult will set the content_error flag. At this point, the render process is successful, but you know that there is something wrong with the data.

You can also add a value from the _ERROR_ dictionary to the output of the template, if certain criteria is matched (e.g. something should be enabled which is not part of the template).

There are multiple keys within the _ERROR_ dictionary (invalid_value for invalid/missing values, template for something that isn’t handled by the template etc.), but they are not used to generate an error message. It’s just to improve the readability of the templates. There is a list with all defined keys available in the networkconfgen README.

JSON Export

The last notable function of the library is to convert the NetworkConfGenResult object to a JSON format. This allows from my perspective a quite easy integration of the data with other tools:

>>> result.to_json()
{'content_error': True,
 'error_text': None,
 'from_string': True,
 'render_error': False,
 'search_path': None,
 'template_file_name': None,
 'template_result': '!\nhostname $$INVALID_VALUE$$\n!'}

The keys are the attributes of the NetworkConfGenResult instance. There are some additional values that provides information on how the template was generated (using a template string or a template file, if so with the search path etc.).

Library and Example Code

You find the full example code as a jupyter notebook in the networkconfgen directory within my python script examples repository on GitHub.

The code of the library is also available on GitHub in the networkconfgen GitHub repository. Thanks for reading.