Add Unix and Windows application plugins #851

JSCU-CNI · 2024-09-17T09:39:39Z

This PR adds Unix (snapd and .desktop) and Windows (registry) application plugins to dissect.

Fixes #884, fixes #885 and fixes #886.

Horofic

As mentioned in #852. Please also add the record field documentation to their respective exported functions

dissect/target/plugins/os/windows/regf/applications.py

dissect/target/plugins/os/unix/applications.py

dissect/target/plugins/os/windows/regf/applications.py

dissect/target/plugins/os/unix/applications.py

tests/plugins/os/unix/linux/debian/test_snap.py

tests/plugins/os/windows/regf/test_applications.py

Co-authored-by: Stefan de Reuver <9864602+Horofic@users.noreply.github.com>

Horofic · 2024-09-26T10:29:16Z

As mentioned in #852. Please also add the record field documentation to their respective exported functions

In case you missed it, could you please add this to the exported functions still?

JSCU-CNI · 2024-09-30T07:35:26Z

In case you missed it, could you please add this to the exported functions still?

I missed that, thanks for reminding me. Added in f354b97.

codecov · 2024-10-03T08:02:33Z

Codecov Report

Attention: Patch coverage is 88.88889% with 11 lines in your changes missing coverage. Please review.

Project coverage is 74.30%. Comparing base (38f84b2) to head (7271f3b).

Files with missing lines	Patch %	Lines
...issect/target/plugins/os/unix/linux/debian/snap.py	78.94%	8 Missing ⚠️
...ect/target/plugins/os/windows/regf/applications.py	90.00%	2 Missing ⚠️
dissect/target/plugins/os/unix/applications.py	96.77%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #851      +/-   ##
==========================================
- Coverage   76.26%   74.30%   -1.97%     
==========================================
  Files         312      315       +3     
  Lines       26788    26886      +98     
==========================================
- Hits        20430    19977     -453     
- Misses       6358     6909     +551

Flag	Coverage Δ
unittests	`74.30% <88.88%> (-1.97%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Horofic · 2024-10-03T08:08:23Z

The tests seem to be failing on the Windows applications plugin. Looks like an invalid timestamp is passed. Can you try fix this?

JSCU-CNI · 2024-10-07T09:51:00Z

The tests seem to be failing on the Windows applications plugin. Looks like an invalid timestamp is passed. Can you try fix this?

I cannot reproduce that locally, could you share stdout/err?

Horofic · 2024-10-07T13:07:49Z

Upon looking a little closer the the Windows application plugin fails from Python (and PyPy) version 3.9 up to and including 3.10 on Ubuntu and Windows.

Errors across Python versions and OSes are similar of nature (PyPy 3.10 on Windows):

cls = <class 'flow.record.fieldtypes.datetime'>, date_string = '20240301'

    @classmethod
    def fromisoformat(cls, date_string):
        """Construct a datetime from the output of datetime.isoformat()."""
        if not isinstance(date_string, str):
            raise TypeError('fromisoformat: argument must be str')
    
        # Split this at the separator
        dstr = date_string[0:10]
        tstr = date_string[11:]
    
        try:
>           date_components = _parse_isoformat_date(dstr)

C:\hostedtoolcache\windows\PyPy\3.10.14\x86\Lib\datetime.py:1752: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

dtstr = '20240301'

    def _parse_isoformat_date(dtstr):
        # It is assumed that this function will only be called with a
        # string of length exactly 10, and (though this is not used) ASCII-only
        if len(dtstr) < 10:
>           raise ValueError('isoformat expects a string of length 10')
E           ValueError: isoformat expects a string of length 10

C:\hostedtoolcache\windows\PyPy\3.10.14\x86\Lib\datetime.py:275: ValueError

During handling of the above exception, another exception occurred:

target_win_users = <Target D:\a\dissect.target\dissect.target\.tox\pypy3.10\tmp\test_windows_applications0\MockTarget-yc3dbmav>
hive_hklm = <VirtualHive>

    def test_windows_applications(target_win_users: Target, hive_hklm: VirtualHive) -> None:
        """test if windows applications are detected correctly in the registry"""
    
        firefox_name = "SOFTWARE\\Microsoft\\Windows\\CurrentVersion\\Uninstall\\Mozilla Firefox 123.0.1 (x64 nl)"
        firefox_key = VirtualKey(hive_hklm, firefox_name)
        firefox_key.add_value("Comments", "Mozilla Firefox 123.0.1 (x64 nl)")
        firefox_key.add_value("DisplayIcon", "C:\\Program Files\\Mozilla Firefox\\firefox.exe,0")
        firefox_key.add_value("DisplayName", "Mozilla Firefox (x64 nl)")
        firefox_key.add_value("DisplayVersion", "123.0.1")
        firefox_key.add_value("EstimatedSize", 238271)
        firefox_key.add_value("HelpLink", "https://support.mozilla.org/")
        firefox_key.add_value("InstallLocation", "C:\\Program Files\\Mozilla Firefox")
        firefox_key.add_value("NoModify", 1)
        firefox_key.add_value("NoRepair", 1)
        firefox_key.add_value("Publisher", "Mozilla")
        firefox_key.add_value("URLInfoAbout", "https://www.mozilla.org/")
        firefox_key.add_value("URLUpdateInfo", "https://www.mozilla.org/firefox/123.0.1/releasenotes")
        firefox_key.add_value("UninstallString", '"C:\\Program Files\\Mozilla Firefox\\uninstall\\helper.exe"')
        hive_hklm.map_key(firefox_name, firefox_key)
    
        chrome_name = "SOFTWARE\\Microsoft\\Windows\\CurrentVersion\\Uninstall\\{47FB91DD-98F3-3C87-A963-357B14EAC7C9}"
        chrome_key = VirtualKey(hive_hklm, chrome_name)
        chrome_key.add_value("DisplayVersion", "122.0.6261.95")
        chrome_key.add_value("InstallDate", "20240301")
        chrome_key.add_value("InstallLocation", "")
        chrome_key.add_value("InstallSource", "C:\\Users\\user\\Desktop\\GoogleChromeEnterpriseBundle64\\Installers\\")
        chrome_key.add_value("ModifyPath", "MsiExec.exe /X{47FB91DD-98F3-3C87-A963-357B14EAC7C9}")
        chrome_key.add_value("NoModify", 1)
        chrome_key.add_value("Publisher", "Google LLC")
        chrome_key.add_value("EstimatedSize", 113725)
        chrome_key.add_value("UninstallString", "MsiExec.exe /X{47FB91DD-98F3-3C87-A963-357B14EAC7C9}")
        chrome_key.add_value("VersionMajor", 70)
        chrome_key.add_value("VersionMinor", 29)
        chrome_key.add_value("WindowsInstaller", 1)
        chrome_key.add_value("Version", 1176322143)
        chrome_key.add_value("Language", 1033)
        chrome_key.add_value("DisplayName", "Google Chrome")
        hive_hklm.map_key(chrome_name, chrome_key)
    
        addressbook_name = "SOFTWARE\\Microsoft\\Windows\\CurrentVersion\\Uninstall\\AddressBook"
        addressbook_key = VirtualKey(hive_hklm, addressbook_name)
        addressbook_key.timestamp = datetime(2024, 12, 31, 13, 37, 0, tzinfo=timezone.utc)
        hive_hklm.map_key(addressbook_name, addressbook_key)
    
        msvc_name = "SOFTWARE\\Microsoft\\Windows\\CurrentVersion\\Uninstall\\{D5D19E2F-7189-42FE-8103-92CD1FA457C2}"
        msvc_key = VirtualKey(hive_hklm, msvc_name)
        msvc_key.add_value("DisplayName", "Microsoft Visual C++ 2022 X64 Minimum Runtime - 14.36.32532")
        msvc_key.add_value("InstallDate", "20240301")
        msvc_key.add_value("DisplayVersion", "14.36.32532")
        msvc_key.add_value("Publisher", "Microsoft Corporation")
        msvc_key.add_value(
            "InstallSource",
            "C:\\ProgramData\\Package Cache\\{D5D19E2F-7189-42FE-8103-92CD1FA457C2}v14.36.32532\\packages\\vcRuntimeMinimum_amd64\\",  # noqa: E501
        )
        msvc_key.add_value("SystemComponent", 1)
        hive_hklm.map_key(msvc_name, msvc_key)
    
        target_win_users.add_plugin(WindowsApplicationsPlugin)
>       results = sorted(list(target_win_users.applications()), key=lambda r: r.name)

tests\plugins\os\windows\regf\test_applications.py:68: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
dissect\target\plugins\os\windows\regf\applications.py:49: in applications
    yield WindowsApplicationRecord(
dissect\target\helpers\record.py:78: in __call__
    return super().__call__(*args, **kwargs)
.tox\pypy3.10\lib\site-packages\flow\record\base.py:595: in __call__
    return self.recordType(*args, **kwargs)
<string>:26: in __init__
    ???
.tox\pypy3.10\lib\site-packages\flow\record\base.py:195: in __setattr__
    v = field_type(v)
.tox\pypy3.10\lib\site-packages\flow\record\fieldtypes\__init__.py:323: in __new__
    obj = cls.fromisoformat(arg)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

cls = <class 'flow.record.fieldtypes.datetime'>, date_string = '20240301'

    @classmethod
    def fromisoformat(cls, date_string):
        """Construct a datetime from the output of datetime.isoformat()."""
        if not isinstance(date_string, str):
            raise TypeError('fromisoformat: argument must be str')
    
        # Split this at the separator
        dstr = date_string[0:10]
        tstr = date_string[11:]
    
        try:
            date_components = _parse_isoformat_date(dstr)
        except ValueError:
>           raise ValueError(f'Invalid isoformat string: {date_string!r}')
E           ValueError: Invalid isoformat string: '20240301'

C:\hostedtoolcache\windows\PyPy\3.10.14\x86\Lib\datetime.py:1754: ValueError

Python 3.9 on Ubuntu:

target_win_users = <Target /home/runner/work/dissect.target/dissect.target/.tox/3.9/tmp/test_windows_applications0/MockTarget-kj40a8z8>
hive_hklm = <VirtualHive>

    def test_windows_applications(target_win_users: Target, hive_hklm: VirtualHive) -> None:
        """test if windows applications are detected correctly in the registry"""
    
        firefox_name = "SOFTWARE\\Microsoft\\Windows\\CurrentVersion\\Uninstall\\Mozilla Firefox 123.0.1 (x64 nl)"
        firefox_key = VirtualKey(hive_hklm, firefox_name)
        firefox_key.add_value("Comments", "Mozilla Firefox 123.0.1 (x64 nl)")
        firefox_key.add_value("DisplayIcon", "C:\\Program Files\\Mozilla Firefox\\firefox.exe,0")
        firefox_key.add_value("DisplayName", "Mozilla Firefox (x64 nl)")
        firefox_key.add_value("DisplayVersion", "123.0.1")
        firefox_key.add_value("EstimatedSize", 238271)
        firefox_key.add_value("HelpLink", "https://support.mozilla.org/")
        firefox_key.add_value("InstallLocation", "C:\\Program Files\\Mozilla Firefox")
        firefox_key.add_value("NoModify", 1)
        firefox_key.add_value("NoRepair", 1)
        firefox_key.add_value("Publisher", "Mozilla")
        firefox_key.add_value("URLInfoAbout", "https://www.mozilla.org/")
        firefox_key.add_value("URLUpdateInfo", "https://www.mozilla.org/firefox/123.0.1/releasenotes")
        firefox_key.add_value("UninstallString", '"C:\\Program Files\\Mozilla Firefox\\uninstall\\helper.exe"')
        hive_hklm.map_key(firefox_name, firefox_key)
    
        chrome_name = "SOFTWARE\\Microsoft\\Windows\\CurrentVersion\\Uninstall\\{47FB91DD-98F3-3C87-A963-357B14EAC7C9}"
        chrome_key = VirtualKey(hive_hklm, chrome_name)
        chrome_key.add_value("DisplayVersion", "122.0.6261.95")
        chrome_key.add_value("InstallDate", "20240301")
        chrome_key.add_value("InstallLocation", "")
        chrome_key.add_value("InstallSource", "C:\\Users\\user\\Desktop\\GoogleChromeEnterpriseBundle64\\Installers\\")
        chrome_key.add_value("ModifyPath", "MsiExec.exe /X{47FB91DD-98F3-3C87-A963-357B14EAC7C9}")
        chrome_key.add_value("NoModify", 1)
        chrome_key.add_value("Publisher", "Google LLC")
        chrome_key.add_value("EstimatedSize", 113725)
        chrome_key.add_value("UninstallString", "MsiExec.exe /X{47FB91DD-98F3-3C87-A963-357B14EAC7C9}")
        chrome_key.add_value("VersionMajor", 70)
        chrome_key.add_value("VersionMinor", 29)
        chrome_key.add_value("WindowsInstaller", 1)
        chrome_key.add_value("Version", 1176322143)
        chrome_key.add_value("Language", 1033)
        chrome_key.add_value("DisplayName", "Google Chrome")
        hive_hklm.map_key(chrome_name, chrome_key)
    
        addressbook_name = "SOFTWARE\\Microsoft\\Windows\\CurrentVersion\\Uninstall\\AddressBook"
        addressbook_key = VirtualKey(hive_hklm, addressbook_name)
        addressbook_key.timestamp = datetime(2024, 12, 31, 13, 37, 0, tzinfo=timezone.utc)
        hive_hklm.map_key(addressbook_name, addressbook_key)
    
        msvc_name = "SOFTWARE\\Microsoft\\Windows\\CurrentVersion\\Uninstall\\{D5D19E2F-7189-42FE-8103-92CD1FA457C2}"
        msvc_key = VirtualKey(hive_hklm, msvc_name)
        msvc_key.add_value("DisplayName", "Microsoft Visual C++ 2022 X64 Minimum Runtime - 14.36.32532")
        msvc_key.add_value("InstallDate", "20240301")
        msvc_key.add_value("DisplayVersion", "14.36.32532")
        msvc_key.add_value("Publisher", "Microsoft Corporation")
        msvc_key.add_value(
            "InstallSource",
            "C:\\ProgramData\\Package Cache\\{D5D19E2F-7189-42FE-8103-92CD1FA457C2}v14.36.32532\\packages\\vcRuntimeMinimum_amd64\\",  # noqa: E501
        )
        msvc_key.add_value("SystemComponent", 1)
        hive_hklm.map_key(msvc_name, msvc_key)
    
        target_win_users.add_plugin(WindowsApplicationsPlugin)
>       results = sorted(list(target_win_users.applications()), key=lambda r: r.name)

tests/plugins/os/windows/regf/test_applications.py:68: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
dissect/target/plugins/os/windows/regf/applications.py:49: in applications
    yield WindowsApplicationRecord(
dissect/target/helpers/record.py:78: in __call__
    return super().__call__(*args, **kwargs)
.tox/3.9/lib/python3.9/site-packages/flow/record/base.py:595: in __call__
    return self.recordType(*args, **kwargs)
<string>:26: in __init__
    ???
.tox/3.9/lib/python3.9/site-packages/flow/record/base.py:195: in __setattr__
    v = field_type(v)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

cls = <class 'flow.record.fieldtypes.datetime'>, args = ('20240301',)
kwargs = {}, arg = '20240301', tstr = '20240301', tzstr = '', tzsearch = ''
tzpos = 0, microsecond_pos = 0

    def __new__(cls, *args, **kwargs):
        if len(args) == 1 and not kwargs:
            arg = args[0]
            if isinstance(arg, bytes_type):
                arg = arg.decode("utf-8")
            if isinstance(arg, string_type):
                # If we are on Python 3.11 or newer, we can use fromisoformat() to parse the string (fast path)
                #
                # Else we need to do some manual parsing to fix some issues with the string format:
                # - Python 3.10 and older do not support nanoseconds in fromisoformat()
                # - Python 3.10 and older do not support Z as timezone info in fromisoformat()
                # - Python 3.10 and older do not support +0200 as timezone info in fromisoformat()
                # - Python 3.10 and older requires "T" between date and time in fromisoformat()
                #
                # There are other incompatibilities, but we don't care about those for now.
                if not PY_311_OR_HIGHER:
                    # Convert Z to +00:00 so that fromisoformat() works correctly on Python 3.10 and older
                    if arg[-1] == "Z":
                        arg = arg[:-1] + "+00:00"
    
                    # Find timezone info after the date part. Possible formats, so we use the longest one:
                    #
                    # YYYYmmdd      length: 8
                    # YYYY-mm-dd    length: 10
                    tstr = arg
                    tzstr = ""
                    tzsearch = arg[10:]
                    if tzpos := tzsearch.find("+") + 1 or tzsearch.find("-") + 1:
                        tzstr = arg[10 + tzpos - 1 :]
                        tstr = arg[: 10 + tzpos - 1]
    
                    # Convert +0200 to +02:00 so that fromisoformat() works correctly on Python 3.10 and older
                    if len(tzstr) == 5 and tzstr[3] != ":":
                        tzstr = tzstr[:3] + ":" + tzstr[3:]
    
                    # Python 3.10 and older do not support nanoseconds in fromisoformat()
                    if microsecond_pos := arg.rfind(".") + 1:
                        microseconds = arg[microsecond_pos:]
                        tstr = arg[: microsecond_pos - 1]
                        if tzpos := (microseconds.find("+") + 1 or microseconds.find("-") + 1):
                            microseconds = microseconds[: tzpos - 1]
                        # Pad microseconds to 6 digits, truncate if longer
                        microseconds = microseconds.ljust(6, "0")[:6]
                        arg = tstr + "." + microseconds + tzstr
                    else:
                        arg = tstr + tzstr
    
>               obj = cls.fromisoformat(arg)
E               ValueError: Invalid isoformat string: '20240301'

.tox/3.9/lib/python3.9/site-packages/flow/record/fieldtypes/__init__.py:323: ValueError

JSCU-CNI · 2024-10-23T10:31:19Z

This should now be fixed in ae15c0a.

JSCU-CNI added 2 commits September 17, 2024 11:38

add application plugins

022ae34

add tests

0b61f8d

JSCU-CNI marked this pull request as ready for review September 17, 2024 13:34

fix tests

95a0d05

Horofic requested review from Horofic September 18, 2024 10:35

Horofic self-assigned this Sep 18, 2024

Horofic requested changes Sep 19, 2024

View reviewed changes

JSCU-CNI and others added 5 commits September 19, 2024 16:08

Apply suggestions from code review

d509f5c

Co-authored-by: Stefan de Reuver <9864602+Horofic@users.noreply.github.com>

Update dissect/target/plugins/os/unix/applications.py

6692ae2

Co-authored-by: Stefan de Reuver <9864602+Horofic@users.noreply.github.com>

implement review suggestions

f61ec04

Merge branch 'main' into feature/unix-windows-application-plugins

6e7e0df

Merge branch 'main' into feature/unix-windows-application-plugins

0c36c5e

JSCU-CNI added 2 commits September 30, 2024 09:34

add better docstrings

f354b97

Merge branch 'main' into feature/unix-windows-application-plugins

2941291

Horofic previously approved these changes Oct 3, 2024

View reviewed changes

Merge branch 'main' into feature/unix-windows-application-plugins

7271f3b

Horofic self-requested a review October 3, 2024 13:18

Merge branch 'main' into feature/unix-windows-application-plugins

5ef3a0f

JSCU-CNI added 2 commits October 23, 2024 12:27

Merge branch 'main' into feature/unix-windows-application-plugins

3a3a769

fix InstallDate regf parsing

ae15c0a

JSCU-CNI dismissed Horofic’s stale review via ae15c0a October 23, 2024 10:31

fix rst

31a61ed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Unix and Windows application plugins #851

Add Unix and Windows application plugins #851

JSCU-CNI commented Sep 17, 2024 •

edited

Loading

Horofic left a comment

Horofic commented Sep 26, 2024 •

edited

Loading

JSCU-CNI commented Sep 30, 2024

codecov bot commented Oct 3, 2024 •

edited

Loading

Horofic commented Oct 3, 2024

JSCU-CNI commented Oct 7, 2024

Horofic commented Oct 7, 2024

JSCU-CNI commented Oct 23, 2024

Add Unix and Windows application plugins #851

Are you sure you want to change the base?

Add Unix and Windows application plugins #851

Conversation

JSCU-CNI commented Sep 17, 2024 • edited Loading

Horofic left a comment

Choose a reason for hiding this comment

Horofic commented Sep 26, 2024 • edited Loading

JSCU-CNI commented Sep 30, 2024

codecov bot commented Oct 3, 2024 • edited Loading

Codecov Report

Horofic commented Oct 3, 2024

JSCU-CNI commented Oct 7, 2024

Horofic commented Oct 7, 2024

JSCU-CNI commented Oct 23, 2024

JSCU-CNI commented Sep 17, 2024 •

edited

Loading

Horofic commented Sep 26, 2024 •

edited

Loading

codecov bot commented Oct 3, 2024 •

edited

Loading