Configuration Generator with Python and Jinja2

This week, I will show you how to build a simple python based configuration generator using the Jinja2 library. As you will see in this post, the technical implementation is not really a big deal. From my point of view, the real challenge is the definition of the data model (or the structure of the parameters) that is used to generate the configurations. Please don’t get me wrong, I don’t talk about a plain one dimensional substitution of strings, its more like ”here is a list of VLANs with all information, I require a config for my distribution and access switches”.

After reading this post, you will get a basic understanding, how you can implement a configuration generator using a simple python script utilizing the Jinja2. As always, you can find the code from this post on my python script examples repository on Github.

The basic concept of the configuration generator

At first, we will look at a simple an quite common use case. Consider the following scenario: you have a list with parameters and created a configuration template, which includes references to the parameters from the list. At first, we will start with a CSV file, where one line is a set of parameters for a single device (I know, that’s one dimensional but hang on, I will get there). The configurations should be generated and stored in a separate directory. The following figure illustrates the concept using the files from the example repository.

The CSV file from the example has the following content:

hostname;domain_name;management_ip
switch-a;domain.local;10.0.0.1
switch-b;domain.local;10.0.0.2
switch-c;domain.local;10.0.0.3
switch-d;domain.local;10.0.0.4

One solution that will work could be a simple “search & replace” function of the parameters within the template file. Sounds easy and quick, but the configuration parameters are usually not only key-value pairs. You have normally various lists of parameters for you configuration (e.g. VLANs, IP networks, static routes, OSPF areas, BGP neighborships etc.). You will run very fast into limitations with such an implementation, therefore I always recommend the use of a full-blown template language. The benefit is from my perspective quite clear: it gives you more capabilities out of the box. I prefer Jinja2 for this case.

What is Jinja2?

Jinja2 is a modern template language, which offers many functions and capabilities within a single library. For our use case we will not use every piece of functionality, but there are some quite interesting concepts in it as we will see later in this post.

To run the example, you need to install Jinja2 to your python environment, using the following command:

$ pip install jinja2

The library itself is also straightforward to use. The first step is to create a Template file. In our case this is a prepared Cisco configuration template, which looks exactly like the following snippet (file switch.j2 from the repository):

!
hostname {{ hostname }}
!
ip domain-name {{ domain_name }}
!
interface Loopback0
 description Management Interface
 ip address {{ management_ip }} 255.255.255.255
!
interface FastEthernet 0/0
 description external interface
 ip address dhcp
!

Within the configuration template, you’ll find expressions like {{ hostname }}. These are references to the variables, which will instruct the Jinja2 parser to replace it with the appropriate value form the given parameter list. The parameter list is in this case one line within the CSV file. If a variable is not found, it is simply replaced by nothing.

That’s all what we need, now lets go to the python side of the example.

The python script and the CSV parameters

You will find the example for this post in my python script examples repository on GitHub. I will not explain in detail, what the script will do. I’ll recommend that you now download the repository and start the script using the following command:

$ python3 csv_based_config_generator.py

The script will execute at a high level the following four steps:

  • Load the content of the CSV file parameter.csv
  • Convert one line of the CSV file to a dictionary and add it to a parameter list
  • Create the Jinja2 execution environment
  • Combine every dictionary from the parameter list with the Jinja2 template and write the result to a file

After the script was executed, you should find a _output folder with the resulting configurations named <hostname>.config. You can also extend the CSV file with additional header values and insert these into the template. You will see that the script will also work with this modification. After you tried it, come back and read the rest of this post.

As you might see, this script has around 40 lines (including comments and whitespace), but the basic problem what I described in the introduction is solved. But wait, I think you need a bit more than this to generate configurations at scale.

Add a hierarchy to the parameters

Within the script, you maybe noticed the eight lines of code, where we convert the CSV file to a dictionary. This is required, because the Jinja2 template engine will work with (multi-dimensional) python dictionaries. As you might know, the python dictionary structure can be converted to and imported from JSON formatted files. Within my last post about RESTful APIs, we already heard about the data format JSON. As you might remember: JSON stands for JavaScript Object Notation and is one way to present data to an application.

Why should I use JSON instead CSV files? The answer is quite easy: CSV files have just one flat hierarchy, whereas JSON files can express lists and nested dictionaries. These can be used as a single parameter set for the script. Along with Jinja2, you now have the ability to implement more complex but also more powerful scripts.

Lets extend our use case: We will now add VLANs with a name parameter to our configuration and I don’t care how many VLANs are defined per device. For this purpose, we need to rewrite our CSV file to a JSON formatted file:

[
  {
    "hostname": "switch-a",
    "domain_name": "domain.local",
    "management_ip": "10.0.0.1",
    "vlans": [
      {
        "name": "Data",
        "id": 100
      },
      {
        "name": "Voice",
        "id": 200
      },
      {
        "name": "Server",
        "id": 300
      }
    ]
  }
]

Within the JSON parameter file, you can also extend the list of VLANs with additional items, if you like to. Now, we will extend the configuration script with the VLAN configuration using the parameters from the VLAN list. To iterate over it, the template will use a for-loop. To accomplish this, we add the following lines to it:

{% for vlan in vlans %}
vlan {{ vlan.id }}
 name {{ vlan.name }}
{% endfor %}
!

The template file in the repository is named switch_with_vlans.j2. You might now see why I found Jinja2 more useful than just a plain “search & replace” function on a string. The {% and %} expressions are control signs for Jinja2. With this you can call various control-structures and functions within the template. There are many control structures, filters and other functions available within Jinja2, as you can see for example in the List of Control Structures section in the Template Designer documentation part of the documentation.

The python code example with JSON parameters

Because you have already downloaded the example, you can now just run the JSON based configuration generator and see what happens.

$ python3 json_based_config_generator.py

After the script was executed, you should find the following files in the _output folder:

  • switch-a_with_vlans.config
  • switch-b_with_vlans.config
  • switch-c_with_vlans.config
  • switch-d_with_vlans.config

These scripts contains now all the VLANs and names that are defined in the JSON parameter file. Try to extend the VLAN list, but be careful about the formatting. If there is an error, the script will raise an exception and stop working.

Conclusion

Okay, 80 lines of code at all and we have a quite useful configuration generation script for two use cases. I know that there is some room for improvements from a coding and usability perspective, but you should now have a basic understanding how you can use Jinja2 and python to build a configuration generator.

If you extend the concept, you will see that there are some limitations in terms of readability of the configuration templates and also in terms of, lets say “network specific functionality”, but these should be topics for other posts.

That’s it for this week. I hope you learned something and tried the example. Tell me what you think about the small scripts. Probably, you find some other use cases for it or you have any suggestion how to improve the code. If so, please let me know. Thanks for reading.