Post Image

Network Automation

If you are a network administrator managing Cisco devices you know the monumental task it is to manage all network devices through SSH manually, and realistically it is not practical at scale. In this post I will be going over at a high level the challenges network administrators face without automation, the various degrees of automation, and provide a recommendation and direction for automating the network. I also make the assumption that your network devices are able to be administered via SSH and because of my particular background this is geared toward Cisco products.

 

Example Scenario

Lets take the example of an organization with 15 remote sites and 1 datacenter, making the assumption of the remote site having 1 switch and 1 router, and the datacenter having 2 switches and 2 routers that leaves us with 17 routers and 17 switches for a total of 34 network devices to manage. Now lets say that you have a very basic change to make disabling the http server by issuing the command no ip http server. An organization with 34 is small enough where you can entertain the idea of brute forcing it and doing them one by one but think if the organization was double the size and had 68 network devices, we are now entering the scale where doing them manually one by one becomes clearly an inefficient use of time and will eat up a good chunk of the day on a mundane task.

Lets re focus to our 34 device organization with 15 remote sites and break this problem down and its various solutions.

 

What needs to be done

On each network device the admin needs to do the following 5 step process

  1. Log into the device.
  2. Enter configuration mode.
  3. Issue command no ip http server.
  4. Exit into privilege exec mode.
  5. Save configuration.

 

Possible Solutions to Accomplish Task and Time Estimation

Completely Manually

We could manually find the devices we want to change, log in, and type each command manually. This is the most time intensive task and inefficent use of time by far, the time breakdown for this I have below

Step Time Estimation (seconds) Note
Finding device and logging in  15 Factoring in authentication errors over time
Typing the commands and logging out  45

Factoring in mistyped commands over time

Time Calculation Formula: (15+45) * device_count

I estimate 1 minute per device on average which when adding 20% gives us 33 minutes 36 seconds to accomplish this task.

It is easy to say that there must be a better way than taking the better part of an hour to do this manually

 

Rudimentary Partial Automation

Because of the nature of this change in particular the configuration commands don't change from device to device so we can write the commands in a notepad and copy and paste them to each device one at a time. Below I have my time estimation breakdown.

Step Time Estimation (seconds) Note
Writing out the configuration 30 Only need to do this once
Finding device and logging in  15 Factoring in authentication errors over time
Pasting in the commands 5

 

Time Calculation Formula: ( (15+5) * device_count) + 30

So in this scenario the per device time is 20 seconds with a 1 time 30 second investment. So expanding this and adding 20% for error factor it will take us approximately 15 minutes 24 seconds to accomplish this task. It is clear to see that the manual process is out of the door. But we can do this faster if we look at enhancing this a bit with the right tool kit.

 

Enhanced Partial Automation

A tool like MobaXterm's multi execution allows you to type/paste commands into multiple windows at once, so lets say we can do 10 devices at once I have a time estimation breakdown below.

Step Time Estimation (seconds) Note
Writing out the configuration 30 Only need to do this once
Finding device and logging in  60 Only do this 3 times in our example, using MobaXterm it takes slightly longer to select multiple devices to log into.
Pasting in the commands 5

Only do this 3 times in our example

Time Calculation Formula: ( (60+5) * (device_count/10) ) + 30

Adding up the time here with an extra 20%  we have reduced the entire task to 5 minutes and 24 seconds. The one caveat here that I really dislike is that it hinges on a paid software that could be a barrier to entry for users. However 5 and a half minutes for this task is pretty good!

 

Completely Automated

Using a tool like ansible or a programming language such as Python with a library for automating against network devices such as CiscoAutomationFramework or NetMiko allows you to completely automate the task. The up front cost of writing the script is a bit heavier and depending on the technologies the time estimation can be way different so I will need to make some assumptions and those assumptions will be related to my experience.

Here I assume that you have a script that reads a file, and then takes the contents of that file and types it into all of your devices which it got the IP's from either another file or was hard coded. With a script like that already written the time breakdown will look something like below.

Step Time Estimation (seconds) Note
Writing out the configuration 30 Only need to do this once
Running script 10 Only need to do this once.

Time Calculation Formula: (30+10) * times_run)

Now we are cooking with fire, with a ~20% time added we have our task down to 48 seconds! I don't believe that we should figure in the time to build the script because if made generic enough can be used in multiple scenarios so each time it is used the time cost goes down. If I were to give an estimate I could make a script as described above in about 5-10 minutes.

 

Scaled Up

With the estimates made above using the formulas listed I make time estimations for this change by device count based on the estimated time per device listed above plus 20%

The times are in H:MM:SS format

  50 Devices 100 Devices 150 Devices 200 Devices 250 Devices
Manual 0:45:18 1:30:18 2:15:18 3:00:18 3:45:18
Partial Automation 0:20:36 0:40:36 1:00:36 1:20:36 1:40:36
Enhanced Automation 0:07:06 0:13:36 0:20:06 0:26:36 0:33:06
Complete Automation 0:00:48 0:00:48 0:00:48 0:00:48 0:00:48

 

As you can see it is clear that as things scale up the complete automation is where you save the most time, and the greater your device count the greater your time saved. I would argue that it is clear at just 100 devices it makes sense to do nothing other than the complete automation solution.

 

A Few Caveats

The time estimations I give on the completely automated section I assume that you are familiar with the automation technology you are using and confident in your abilities. If you are beginning in your journey automation against the network you will not see time savings like this but such as any technology there is a learning curve that will make some things take longer than you are used to. I believe you should be able to get to a point where these estimations are accurate within a few months of consistently writing scripts and using a framework that abstracts away a lot of the lower level programming things so you can focus on the task you are trying to accomplish.

Also the time estimations is only of actual execution of a task not all the parts around it, investigating, double checking, due diligence etc. The other business factors vary wildly from business to business and these estimations are meant to be more of an illustration of time savings between the methodologies. 

 

What Solution is Right for You?

Well I can for sure tell you that doing it completely manually is NOT right for you, but that's obvious.

But realistically I believe the "complete" automation part is where every network administrator should be moving to at this point. While my example above is very basic and each device has the same configuration to be pasted you need to also consider that you can build conditional logic into your scripts which allows you to not only automate the entering of configuration, but the decision tree of figuring out when to enter the configuration (ex. only enter the configuration if it is missing). In addition you can also automate the generation of said configuration, lets say you need to add a second IP address to a specific interface on a bunch of devices for a migration, you can have your script log into the device, gather the IP address, and dynamically generate the target IP address.

So realistically you can automate

  • Configuration generation
  • Decision to input configuration
  • The input of the configuration

One you expand upon that you can build really amazing tools like something to ensure configuration compliance which checks the base configuration and alerts and/or automatically remediates devices that are out of compliance!

The options are nearly endless as to the things you can do, you will find your limitation will be your imagination as to what you can automate. As an example I have several automation scripts that automate all 3 of the points above.

 

Where to Start

Finding a starting point can be very hard, when I started automating my networks back in 2016 I did not want to go with a product like ansible and I wanted the power of a programming language that was easy so Python with the NetMiko library was about the only option and while I found it is great for a broad base to start with, I wanted something specifically for Cisco devices that abstracted things away like hardware type and provided a consistent interface for getting data from the device, issuing commands, navigating the various CLI levels, concurrent execution, and more. Unfortunately for me there was not thing out there at that time so I began developing my own framework which I then released as the CiscoAutomationFramework which provides exactly what I described above. While I am partial to it, I have not found a better way to automate against Cisco devices using Python so I would highly advise checking it out!

 



Comments (0)
Leave a Comment