I noticed some discrepancies in logs on a few of my network devices so I needed to check and see what devices clock's were set incorrectly and get them all properly set to the NTP server. Versus doing this all one by one I wrote a script using Python and the CiscoAutomationFramework to audit all of my network devices and display their clock in a list. 

Requirements

• Be able to easily tell if a clock is off by more than 5 seconds from another
  • Because of that I will need to issue the command at the same time, so I will need to use multiple threads.
    • This is built into the CiscoAutomationFramework
• List out the hostname and IP address of each devices

Script

Import required libraries

This script was fairly easy to write, I start by importing the libraries required.

from CiscoAutomationFramework.ThreadLib import SSH, start_threads

Define an Object that Abstracts the Network Device

I then define a new object to represent a network device that inherits from CiscoAutomationFramework.ThreadLib.SSH which is what makes it run in multiple threads.

class NetworkDevice(SSH):

    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.raw_clock_time = []

    @property
    def clock_time(self):
        return ' '.join(x for x in self.raw_clock_time if x != '')

    def during_login(self, ssh):
        self.raw_clock_time = ssh.send_command_get_output('show clock')[1:-1]

Breaking down the NetworkDevice object above

Looking at the __init__ method:

def __init__(self, *args, **kwargs):
    super().__init__(*args, **kwargs)
    self.raw_clock_time = []

I extend the __init__ method so I can store data in the runtime of the thread, this is the raw data returned when issuing the raw clock command which you can see on line 5.

I override the during_login method which accepts ssh as an argument to define what I want to do when logged into the device.

def during_login(self, ssh):
    self.raw_clock_time = ssh.send_command_get_output('show clock')[1:-1]

I remove the first and last lines of output because the first line will be the previous command, and the last line will be the prompt and set the instance variable that I previously defined as the output.

And I also create an attribute that will analyze the contents of the instance variable raw_clock_time and return a string that is the clock output as returned from the network device.

@property
def clock_time(self):
    return ' '.join(x for x in self.raw_clock_time if x != '')

This attribute is designed to be referenced after the runtime of the object has been completed.

Scripting it Together

I now have a multi threaded object that can be run, I just need to write a script which uses that object to connect to the network devices and print the data they receive.

if __name__ == '__main__':
    from getpass import getpass

    username = input('Enter Username: ')
    password = getpass(f'Enter Password for {username}: ')
    ips = ['', '']  # replace with your IP addresses or get them programmatically

    print('Reaching out to network devices!')
    threads = start_threads(NetworkDevice, ips, username, password)

    print('Waiting for threads to complete.')
    for t in threads:
        t.join()

    print('Hostname     IP          Time')
    for thread in threads:
        print(f'{thread.hostname}  {thread.ip}  {thread.clock_time}')

Breakdown

We first import getpass so we can securly get the users password, and then get the username and password from the user.

from getpass import getpass

username = input('Enter Username: ')
password = getpass(f'Enter Password for {username}: ')

Because we do not get an enable password it is implied that the credentials will have access directly to privilege exec mode.

In this example I have 2 blank strings defined as the IP addresses, but this is where we would either define the IP addresses manually or get them programmatically.

ips = ['', '']  # replace with your IP addresses or get them programmatically

We then pass the NetworkDevice object along with the IP addresses, username, and password to the start_threads helper function to start all of the threads and return them to the user.

print('Reaching out to network devices!')
threads = start_threads(NetworkDevice, ips, username, password)

print('Waiting for threads to complete.')
for t in threads:
    t.join()

I chose to not automatically wait for their completion so I could update the user when the waiting starts and then manually wait for their execution with the for loop.

And lastly we iterate through all of the completed NetworkDevice threads and call the clock_time to display the clock time that was gathered during login.

print('Hostname     IP          Time')
for thread in threads:
    print(f'{thread.hostname}  {thread.ip}  {thread.clock_time}')

An important note is because we gather the output and store it as an instance variable, it does not display the time of the device at the time of the print statement, but the time that was gathered when the device  was logged into, likely a couple seconds behind when you are viewing it.

Output

When running the script I get output similar to below.

$ python myscript.py
Enter Username: myuser
Enter Password for myuser:
Reaching out to network devices!
Waiting for threads to complete.
Hostname     IP      Time
switch1  192.168.1.2  11:21:27.640 CST Thu Sep 8 2022
switch2  192.168.1.3  11:21:27.588 CST Thu Sep 8 2022

$

As you can see the clock is less than 100 milliseconds different from the other, which is what I am looking for. I ran the script shortly after 11:21 CST and in the real output I had 80 devices showing within 2 seconds of each other. It is important to understand that the drift in a script like this is likely due to execution timing between the underlying threads being off between threads by up to 2 seconds more so than the clocks being off from each other because I know they are syncing the same timeserver.

With this script you will see a list of all of your devices clock outputs and if one of them is more than 5 seconds off from the rest you can investigate why.

Full Script

Below I have the full script in its entirety for you to copy and paste and run in your environment.

from CiscoAutomationFramework.ThreadLib import SSH, start_threads


class NetworkDevice(SSH):

    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.raw_clock_time = []

    @property
    def clock_time(self):
        return ' '.join(x for x in self.raw_clock_time if x != '')

    def during_login(self, ssh):
        self.raw_clock_time = ssh.send_command_get_output('show clock')[1:-1]


if __name__ == '__main__':
    from getpass import getpass

    username = input('Enter Username: ')
    password = getpass(f'Enter Password for {username}: ')
    ips = ['', '']  # replace with your IP addresses or get them programmatically

    print('Reaching out to network devices!')
    threads = start_threads(NetworkDevice, ips, username, password)

    print('Waiting for threads to complete.')
    for t in threads:
        t.join()

    print('Hostname     IP          Time')
    for thread in threads:
        print(f'{thread.hostname}  {thread.ip}  {thread.clock_time}')