NX-OS show interface status parser overflows in some cases

[blue212121]

Issue is with p1_1 for me, example input, and result:
input:

Eth1/2        --                 xcvrAbsen 1         auto    auto    --         
Eth1/3        testdesc           notconnec 1         auto    auto    1000base-T 
Eth1/4        testdesc           connected 1         full    1000    1000base-T 

result:

{'interface': 'Eth1/2', 'name': '--                 xcvrAbsen', 'status': '1', 'vlan': 'auto', 'duplex_code': 'auto', 'port_speed': '--', 'type': ''}
{'interface': 'Eth1/3', 'name': 'testdesc           notconnec', 'status': '1', 'vlan': 'auto', 'duplex_code': 'auto', 'port_speed': '1000base-T', 'type': ''}
{'interface': 'Eth1/4', 'name': 'testdesc', 'status': 'connected', 'vlan': '1', 'duplex_code': 'full', 'port_speed': '1000', 'type': '1000base-T '}

'connected' state is working i suppose because the port speed is num in that case, hence not matchin [a-z] in duplex_code

Working regex for me by defining the possible vlan values, so that the <name> wont overflow and match the <status> variable as well (since <type> can match whitespace only as well, it will match even a single space at the end after the actual type is what im guessing):

(?P<interface>(\S+))\s+
(?P<name>([\S\s]+))(?<! )\s+
(?P<status>([a-zA-Z]+))\s+
(?P<vlan>(\d+|routed|trunk|f-path|--))\s+
(?P<duplex_code>([a-z]+))\s+
(?P<port_speed>(\S+))
(\s+(?P<type>([\S]+)))?\s*$

but may be better regex solution (not sure if there are other values for vlan field) thats why i opened an issue and not a pull request

hardware: N3K-C3064PQ-10GE
software: NXOS: version 7.0(3)I7(9)
genie==23.2
pyats==23.2

[sumitsehgal123]

Hi .
Thanks for reaching out piestack support team, I will be looking at your ticket and reviewing it . Please make sure add the logs / pyats version / package list .
Recommend is to always use the latest pyats version (i.e. Current released version 23.3)

[blue212121]

Hello,
current released version was 23.2 at time of reporting, as indicated in post. Is this issue confirmed fixed in 23.3?
thanks

[sumitsehgal123]

Hi blue212121,
Thanks for your patience. I reviewed your ticket and found that the result you framed is not in a proper format as the key value paired wrongly. Kindly frame that in a proper format and try use this regex.

p1 = re.compile(r'(?P(\S+))\s+(?P(\S+))?\s'
r'+(?P(\S+))?\s+(?P(\S+))'
r' +(?P<duplex_code>(\S+))\s'
r'+(?P<port_speed>(\S+))(\s*(?P(\S*)))?$')

p1_1 = re.compile(r'(?P(\S+))\s+'
r'(?P([\S\s]+))(?<! )\s+'
r'(?P(\S+))\s+'
r'(?P(\S+))\s+'
r'(?P<duplex_code>([a-z]+))\s+'
r'(?P<port_speed>(\S+))\s*'
r'(?P([\S\s]*))$')

[sumitsehgal123]

Hi blue212121,
I hope your issue has been resolved with above solution . Please let me know are you still facing any issue or not ?

[blue212121]

Hi sumit, havent got time to test the above solution yet, but looks similar to the original p1_1 which produced the issue, but will check eventually, thank you for your patience

[blue212121]

Hi @sumitsehgal123 ,

sorry, accidentally closed issue, but it is still not fixed, proposed solution is syntactically incorrect as it is missing the names from the named groups:

p1_1 = re.compile(r'(?P(\S+))\s+'
r'(?P([\S\s]+))(?<! )\s+'
r'(?P(\S+))\s+'
r'(?P(\S+))\s+'
r'(?P<duplex_code>([a-z]+))\s+'
r'(?P<port_speed>(\S+))\s*'
r'(?P([\S\s]*))$')

but even when filling in from source code, the result still produces the same wrong groupdict from the original post from this issue, where description and status merge together.

problem regex:

        p1_1 = re.compile(r'(?P<interface>(\S+))\s+'
                          r'(?P<name>([\S\s]+))(?<! )\s+'
                          r'(?P<status>(\S+))\s+'
                          r'(?P<vlan>(\S+))\s+'
                          r'(?P<duplex_code>([a-z]+))\s+'
                          r'(?P<port_speed>(\S+))\s*'
                          r'(?P<type>([\S\s]*))$')

so unfortunely this issue still exists

[sumitsehgal123]

Hi blue212121,
I am working on your ticket ,I will get back to you .

[sumitsehgal123]

Hi Blue212121,
I tried your input and I am getting the below output . Please try to do from your end .
the output should be as the below format.

dev.parse("show interface status", output=output)
{'interfaces': {'Ethernet1/2': {'status': 'xcvrAbsen', 'vlan': '1', 'duplex_code': 'auto', 'port_speed': 'auto'}, 'Ethernet1/3': {'name': 'testdesc', 'status': 'notconnec', 'vlan': '1', 'duplex_code': 'auto', 'port_speed': 'auto', 'type': '1000base-T'}, 'Ethernet1/4': {'name': 'testdesc', 'status': 'connected', 'vlan': '1', 'duplex_code': 'full', 'port_speed': '1000', 'type': '1000base-T'}}}

[sumitsehgal123]

After make changes in the source file , you have to give make develop and make json in the genieparser directory. After that please try to execute the parser.it will give the excepted output .

[blue212121]

Hi Blue212121, I tried your input and I am getting the below output . Please try to do from your end . the output should be as the below format.

dev.parse("show interface status", output=output) {'interfaces': {'Ethernet1/2': {'status': 'xcvrAbsen', 'vlan': '1', 'duplex_code': 'auto', 'port_speed': 'auto'}, 'Ethernet1/3': {'name': 'testdesc', 'status': 'notconnec', 'vlan': '1', 'duplex_code': 'auto', 'port_speed': 'auto', 'type': '1000base-T'}, 'Ethernet1/4': {'name': 'testdesc', 'status': 'connected', 'vlan': '1', 'duplex_code': 'full', 'port_speed': '1000', 'type': '1000base-T'}}}

Hi @sumitsehgal123

apparently this issue is reproducible when using the nxos parser, when there is a space in the description, see below on Ethernet1/3:

>>> print(json.dumps(dev.parse("show interface status", output=teststring), indent=4))
{
    "interfaces": {
        "Ethernet1/2": {
            "status": "xcvrAbsen",
            "vlan": "1",
            "duplex_code": "auto",
            "port_speed": "auto"
        },
        "Ethernet1/3": {
            "name": "testdesc is 1      notconnec",
            "status": "1",
            "vlan": "auto",
            "duplex_code": "auto",
            "port_speed": "1000base-T"
        },
        "Ethernet1/4": {
            "name": "testdesc is 2",
            "status": "connected",
            "vlan": "1",
            "duplex_code": "full",
            "port_speed": "1000",
            "type": "1000base-T"
        }
    }
}

teststring:

Eth1/2        --                 xcvrAbsen 1         auto    auto    --         
Eth1/3        testdesc is 1      notconnec 1         auto    auto    1000base-T 
Eth1/4        testdesc is 2      connected 1         full    1000    1000base-T 

[sumitsehgal123]

Hi Blue212121,
I am not able to reproduce the issue from my side ,

[blue212121]

Hi Blue212121, I am not able to reproduce the issue from my side ,

Hello @sumitsehgal123,

that is strange indeed... with provided teststring, the resulting json is correct for you without the description regex group overflowing into the status?

here is my pip freeze (only installed pyats[full] inside venv):

aiofiles==23.1.0
aiohttp==3.8.4
aiohttp-swagger==1.0.16
aiosignal==1.3.1
arrow==1.2.3
async-lru==2.0.2
async-timeout==4.0.2
attrs==23.1.0
backports.ssl==0.0.9
bcrypt==4.0.1
binaryornot==0.4.4
certifi==2022.12.7
cffi==1.15.1
chardet==4.0.0
charset-normalizer==3.1.0
click==8.1.3
cookiecutter==2.1.1
cryptography==40.0.2
dill==0.3.6
distro==1.8.0
frozenlist==1.3.3
genie==23.3
genie.libs.clean==23.3
genie.libs.conf==23.3
genie.libs.filetransferutils==23.3
genie.libs.health==23.3
genie.libs.ops==23.3
genie.libs.parser==23.3
genie.libs.robot==23.3
genie.libs.sdk==23.3
genie.telemetry==23.3
genie.trafficgen==23.3
gitdb==4.0.10
GitPython==3.1.31
grpcio==1.54.0
idna==3.4
IxNetwork==9.0.1915.16
ixnetwork-restpy==1.1.9
Jinja2==3.1.2
jinja2-time==0.2.0
jsonpickle==3.0.1
junit-xml==1.9
lxml==4.9.2
MarkupSafe==2.1.2
multidict==6.0.4
ncclient==0.6.13
netaddr==0.8.0
packaging==23.1
paramiko==3.1.0
pathspec==0.11.1
prettytable==3.7.0
protobuf==3.20.3
psutil==5.9.5
pyats==23.3
pyats.aereport==23.3
pyats.aetest==23.3
pyats.async==23.3
pyats.connections==23.3
pyats.contrib==23.3
pyats.datastructures==23.3
pyats.easypy==23.3
pyats.kleenex==23.3
pyats.log==23.3
pyats.reporter==23.3
pyats.results==23.3
pyats.robot==23.3
pyats.tcl==23.3
pyats.topology==23.3
pyats.utils==23.3
pycparser==2.21
pyftpdlib==1.5.7
PyNaCl==1.5.0
pyOpenSSL==23.1.1
python-dateutil==2.8.2
python-engineio==3.14.2
python-slugify==8.0.1
python-socketio==4.6.1
PyYAML==6.0
requests==2.28.2
requests-toolbelt==0.10.1
robotframework==6.0.2
ruamel.yaml==0.17.21
ruamel.yaml.clib==0.2.7
six==1.16.0
smmap==5.0.0
text-unidecode==1.3
tftpy==0.8.0
tqdm==4.65.0
typing_extensions==4.5.0
unicon==23.3
unicon.plugins==23.3
urllib3==1.26.15
wcwidth==0.2.6
websocket-client==1.5.1
xlrd==1.2.0
XlsxWriter==3.1.0
xlwt==1.3.0
xmltodict==0.13.0
yamllint==1.31.0
yang.connector==23.3
yarl==1.9.1

[sumitsehgal123]

Hi blue212121,
Based on the output provided, I tried to reproduce the issue by following the below steps,

pyats shell --testbed-file testbed.yaml
dev = testbed.devices['***']
output = ''' Eth1/2 -- xcvrAbsen 1 auto auto --
Eth1/3 testdesc notconnec 1 auto auto 1000base-T
Eth1/4 testdesc connected 1 full 1000 1000base-T '''
dev.parse("show interface status ", output=output)

output:
{'interfaces': {'Ethernet1/2': {'status': 'xcvrAbsen', 'vlan': '1', 'duplex_code': 'auto', 'port_speed': 'auto'}, 'Ethernet1/3': {'name': 'testdesc', 'status': 'notconnec', 'vlan': '1', 'duplex_code': 'auto', 'port_speed': 'auto', 'type': '1000base-T'}, 'Ethernet1/4': {'name': 'testdesc', 'status': 'connected', 'vlan': '1', 'duplex_code': 'full', 'port_speed': '1000', 'type': '1000base-T'}}}

Since I'm unable to reproduce the issue, can you please try again. The pyats version used is 23.3 and python version 3.8.
Let me know if I can close this issue since it is not reproducible in my end.

[blue212121]

Hi blue212121, Based on the output provided, I tried to reproduce the issue by following the below steps,

pyats shell --testbed-file testbed.yaml dev = testbed.devices['***'] output = ''' Eth1/2 -- xcvrAbsen 1 auto auto -- Eth1/3 testdesc notconnec 1 auto auto 1000base-T Eth1/4 testdesc connected 1 full 1000 1000base-T ''' dev.parse("show interface status ", output=output)

output: {'interfaces': {'Ethernet1/2': {'status': 'xcvrAbsen', 'vlan': '1', 'duplex_code': 'auto', 'port_speed': 'auto'}, 'Ethernet1/3': {'name': 'testdesc', 'status': 'notconnec', 'vlan': '1', 'duplex_code': 'auto', 'port_speed': 'auto', 'type': '1000base-T'}, 'Ethernet1/4': {'name': 'testdesc', 'status': 'connected', 'vlan': '1', 'duplex_code': 'full', 'port_speed': '1000', 'type': '1000base-T'}}}

Since I'm unable to reproduce the issue, can you please try again. The pyats version used is 23.3 and python version 3.8. Let me know if I can close this issue since it is not reproducible in my end.

Hello @sumitsehgal123,
you are using wrong test string, as mentioned before, issue occures to me, when there is whitespace in the description.
Could you please try testing with below string?

Eth1/2        --                 xcvrAbsen 1         auto    auto    --         
Eth1/3        testdesc is 1      notconnec 1         auto    auto    1000base-T 
Eth1/4        testdesc is 2      connected 1         full    1000    1000base-T 

[sumitsehgal123]

Hi Blue ,
Please find the below patter which is matching with below output .

output= Eth1/4 testdesc is 2 connected 1 full 1000 1000base-T
Eth1/3 testdesc is 1 notconnec 1 auto auto 1000base-T
Eth1/2 -- xcvrAbsen 1 auto auto --

Pattern :
^(?P[\w/]+)\s+(?P(\w+\s\w+\s\d|[\w-]+|[\w\s()[]_,]+))\s+(?P(\w+\s\d|\w+))\s+(?P(\w+))\s+(?P<duplex_code>(\w+))\s+(?P<port_speed>([\w-]+))(\s*(?P([\w-]+)))?$

[blue212121]

Hello @sumitsehgal123

it is matching, but the parsing itself is incorrect as shown previously... hence why issue is created...

[sumitsehgal123]

Hi ,
Since genieparser is community driven, would you like to contribute to the library by following this documentation.

Please let me know if you need any help during the process of PR creation. If not let me know we will add this to our backlog and let you know once it is fixed

[blue212121]

Hello @sumitsehgal123
i proposed a solution in the original post, which fixes my issue, but not sure if it breaks anything else, as i have no time to thoroughly test it at this time. Hence issue and not pr.

[sumitsehgal123]

Hi
I have raised the PR for issue , The change will reflect from next released of pyats .

[blue212121]

Hello @sumitsehgal123 ,

thank you, looking forward to switching back to main :)

[sumitsehgal123]

Hi ,
The PR is merged related to issue and its not reproducible .I checked from my side , Please check and confirm .

pyats shell --testbed-file testbed.yaml
dev = testbed.devices['***']
output='''Eth1/2 -- xcvrAbsen 1 auto auto --
Eth1/3 testdesc notconnec 1 auto auto 1000base-T
Eth1/4 testdesc connected 1 full 1000 1000base-T '''
dev.parse("show interface status",output=output)

Output=
{'interfaces': {'Ethernet1/2': {'name': '--', 'status': 'xcvrAbsen 1', 'vlan': 'auto', 'duplex_code': 'auto', 'port_speed': '--'},
 'Ethernet1/3': {'name': 'testdesc', 'status': 'notconnec 1', 'vlan': 'auto', 'duplex_code': 'auto', 'port_speed': '1000base-T'}, 
'Ethernet1/4': {'name': 'testdesc', 'status': 'connected 1', 'vlan': 'full', 'duplex_code': '1000', 'port_speed': '1000base-T'}}}

if you need any further assistance .please let me know .