Fix tests after password encryption (#466)

* Remove some old TODOs

* Don't remove the Azure resources at the end of the Linux only tests

* Change the variables to work with the 2.0 pipeline

* Run the cluster run workflow to debug the tests

* Run ansible playbook to set fleet

* Log secrets in set_fleet.yml

* Debug setting the environment variables in set_fleet.yml

* Set the debug mode in set_fleet.yml

* Log the Fleet API call details in set_fleet.yml

* Loop through Fleet API calls in set_fleet.yml

* Change the Fleet API call to loop through attempts in set_fleet.yml

* Try to set the Fleet API with retries in set_fleet.yml

* Attempt to output the Fleet API call details in set_fleet.yml

* A new way to handle the Fleet API call in set_fleet.yml

* Export the check_fleet_api.yml file in set_fleet.yml

* Make sure the password is in the check_fleet_api.yml file

* Exit the loop if the Fleet API call succeeds in check_fleet_api.yml

* Wait a little longer for the results to be written to the index

* Update the cluster.yml workflow to wait a little longer for the results to be written to the index

* Delay in a different way

* Attempt a different looping method

* Remove the set-fleet script from the installer

* Reverts to old loop method

* Check that fleet is ready in an external script

* Call the ansible playbook from the install script

* Get the CA fingerprint from the Elasticsearch container

* Adds headers to the curl commands in the set_fleet.yml playbook

* Address the hosts and fleet API issues

* Change the way we login to the Kibana API

* Increase the timeout for the Endpoint Policy API calls

* Increase the timeout for the Endpoint Policy API calls

* Increase the timeout for the Defend Policy API calls

* Only print debug information if debug_mode is true

* Keeps the azure resources on builds

* Fixing Error with certs where the permissions should only be on first
generation!

* Remove sysctl edits to lower privileged ports and add 443 to kibana container

* Add notes on starting vms via azure cli to testing v2

* Fix ansible errors in checking for passwords that are created

* Add debugging commands, and remove references to 443 for kibana from
debug commands

* Update the cluster.yml file to use the new IP address for the Azure instance

* Only allow the ip address of the host to connect to the azure instance

* remove unnecassary script

* Move ansible files to the ansible directory

* Moving Upgrade Readme into upgrade directory

* Add upgrading docs and remove dev notes

* Update main readme docs:

- add section for LME introductory content
- disclaimer around small simple siem
- add Pre-Requisites page
- add Upgrading 1.4 -> 2.0 docs
- note on lme-frontend coming later
- remove references to lmed and make docs accurate to current
  installation process
- add TODOs for docs that still need updated

* Adding updated cloud docs and firewall explanation

* Update FAQ and Pre-requisites

---------

Co-authored-by: Cbaxley and Michael Reeves <mreeve@sandia.gov>

.github/workflows/cluster.yml

@@ -82,7 +82,7 @@ jobs:
  cd /home/lme-user/LME/testing/v2/installers && \
  python3 ./azure/build_azure_linux_network.py \
  -g pipe-${{ env.UNIQUE_ID }} \
- -s 0.0.0.0/0 \
+ -s ${{ env.IP_ADDRESS }}/32 \
  -vs Standard_D8_v4 \
  -l centralus \
  -ast 23:00 \
@@ -251,7 +251,7 @@ jobs:
  env:
  ES_PASSWORD: ${{ env.ES_PASSWORD }}
  run: |
- sleep 120
+ sleep 360
  cd testing/v2/development
  docker compose -p ${{ env.UNIQUE_ID }} exec -T pipeline bash -c "
  ssh -o StrictHostKeyChecking=no lme-user@${{ env.AZURE_IP }} \
@@ -265,9 +265,7 @@ jobs:
  run: |
  cd testing/v2/development
  docker compose -p ${{ env.UNIQUE_ID }} exec -T pipeline bash -c "
- cd /home/lme-user/LME/testing/v2/installers && \
- IP_ADDRESS=\$(cat pipe-${{ env.UNIQUE_ID }}.ip.txt) && \
- ssh lme-user@\$IP_ADDRESS 'cd /home/lme-user/LME/testing/tests && \
+ ssh lme-user@${{ env.AZURE_IP }} 'cd /home/lme-user/LME/testing/tests && \
  echo ELASTIC_PASSWORD=\"$ES_PASSWORD\" >> .env && \
  echo KIBANA_PASSWORD=\"$KIBANA_PASSWORD\" >> .env && \
  echo elastic=\"$ES_PASSWORD\" >> .env && \
@@ -282,14 +280,12 @@ jobs:
  run: |
  cd testing/v2/development
  docker compose -p ${{ env.UNIQUE_ID }} exec -T pipeline bash -c "
- cd /home/lme-user/LME/testing/v2/installers && \
- IP_ADDRESS=\$(cat pipe-${{ env.UNIQUE_ID }}.ip.txt) && \
- ssh lme-user@\$IP_ADDRESS 'cd /home/lme-user/LME/testing/tests && \
+ ssh lme-user@${{ env.AZURE_IP }} 'cd /home/lme-user/LME/testing/tests && \
  echo ELASTIC_PASSWORD=\"$ES_PASSWORD\" >> .env && \
  echo KIBANA_PASSWORD=\"$KIBANA_PASSWORD\" >> .env && \
  echo elastic=\"$ES_PASSWORD\" >> .env && \
  source venv/bin/activate && \
- pytest -v  selenium_tests/'
+ pytest -v selenium_tests/'
  "
  
  - name: Cleanup Azure resources
@@ -311,4 +307,4 @@ jobs:
  run: |
  cd testing/v2/development
  docker compose -p ${{ env.UNIQUE_ID }} down
- docker system prune -af
+ docker system prune -af

.github/workflows/linux_only.yml

@@ -16,6 +16,7 @@ jobs:
  ES_PASSWORD: ""
  KIBANA_PASSWORD: ""
  AZURE_IP: ""
+ IP_ADDRESS: ""
 
  steps:
  - name: Checkout repository
@@ -26,6 +27,9 @@ jobs:
  cd testing/v2/development
  echo "HOST_UID=$(id -u)" > .env
  echo "HOST_GID=$(id -g)" >> .env
+ PUBLIC_IP=$(curl -s https://api.ipify.org)
+ echo "IP_ADDRESS=$PUBLIC_IP" >> $GITHUB_ENV
+
  
  - name: Start pipeline container
  run: |
@@ -57,7 +61,7 @@ jobs:
  cd /home/lme-user/LME/testing/v2/installers && \
  python3 ./azure/build_azure_linux_network.py \
  -g pipe-${{ env.UNIQUE_ID }} \
- -s 0.0.0.0/0 \
+ -s ${{ env.IP_ADDRESS }}/32 \
  -vs Standard_E4d_v4 \
  -l westus \
  -ast 23:00 \

README.md

@@ -1,29 +1,65 @@
 
-![N|Solid](/docs/imgs/cisa.png)
-
 [![Downloads](https://img.shields.io/github/downloads/cisagov/lme/total.svg)]()
 
-# Logging Made Easy: Podmanized
 
-This will eventually be merged with the Readme file at [LME-README](https://github.com/cisagov/LME). 
+
+# Logging Made Easy: 
+
+CISA's Logging Made Easy has a self-install tutorial for organizations to gain a basic level of centralized security logging for Windows clients and provide functionality to detect attacks. LME is the integration of multiple open software platforms which come at no cost to users. LME helps users integrate software platforms together to produce an end-to-end logging capability. LME also provides some pre-made configuration files and scripts, although there is the option to do this on your own.
+
+Logging Made Easy can:
+
+- Show where administrative commands are being run on enrolled devices
+- See who is using which machine
+- In conjunction with threat reports, it is possible to query for the presence of an attacker in the form of Tactics, Techniques and Procedures (TTPs) 
+
+## Disclaimer: 
+
+LME is still in development, and version 2.1 will address scaling out the deployment.
+
+While LME offers SEIM like capabilities, it should be consider a small simple SIEM.
+
+The LME team simplified the process and created clear instruction on what to download and which configugrations to use, and created convinent scripts to auto configure when possible.
+
+LME is not able to comment on or troubleshoot individual installations. If you believe you have have found an issue with the LME code or documentation please submit a GitHub issue. If you have a question about your installation, please look through all open and closed issues to see if it has been addressed before. If not, then submit a [GitHub issue](https://github.com/cisagov/lme/issues) using the Bug Template, ensuring that you provide all the requested information.
+
+For general questions about LME and suggestions, please visit [GitHub Discussions](https://github.com/cisagov/lme/discussions) to add a discussion post.
+
+## Who is Logging Made Easy for?
+
+From single IT administrators with a handful of devices in their network to larger organizations.
+
+LME is suited for for:
+
+- Organizations without [SOC](https://en.wikipedia.org/wiki/Information_security_operations_center), SIEM or any monitoring in place at the moment.
+- Organizations that lack the budget, time or understanding to set up a logging system.
+- Organizations that that require gathering logs and monitoring IT
+- Organizations that understand LMEs limitiation
+
 
 ## Table of Contents:
+- [Pre-Requisites:](#architecture)
 - [Architecture:](#architecture)
 - [Installation:](#installation)
 - [Deploying Agents:](#deploying-agents)
 - [Password Encryption:](#password-encryption)
-- [Further Documentation:](#documentation)
+- [Further Documentation & Upgrading:](#documentation)
+
+## Pre-Requisites
+If you are unsure you meet the pre-requisites to installing LME, please read our [prerequisites documentation](/docs/markdown/prerequisites.md).
+The biggest Pre-requisite is setting up hardware for your ubuntu server with a minimum of `2 processors`, `16gb ram`, and `128gb` of dedicated storage for LME's Elasticsearch database.
 
 ## Architecture:
 Ubuntu 22.04 server running podman containers setup as podman quadlets controlled via systemd.
 
 ### Required Ports:
 Ports required are as follows:
  - Elasticsearch: *9200*
- - Kibana: 443
+ - Kibana: *443,5601*
  - Wazuh: *1514,1515,1516,55000,514*
  - Agent: *8220*
 
+**Kibana NOTE**: 5601 is the default port, and we've set kibana to listen on 443 as well
 
 ### Diagram: 
 
@@ -40,7 +76,7 @@ Podman is more secure (by default) against container escape attacks than Docker.
  - Elastic agents provide integrations, have more features than winlogbeat.
  - wazuh-manager: runs the wazuh manager so we can deploy and manage wazuh agents.
  - Wazuh (open source) gives EDR (Endpoint Detection Response) with security dashboards to cover the security of all of the machines.
- - lme-frontend: will host an api and gui that unifies the architecture behind one interface
+ - lme-frontend (*coming in a future release*): will host an api and gui that unifies the architecture behind one interface
 
 ### Agents: 
 Wazuh agents will enable EDR capabilities, while Elastic agents will enable logging capabilities.
@@ -49,10 +85,11 @@ Wazuh agents will enable EDR capabilities, while Elastic agents will enable logg
  - https://github.com/elastic/elastic-agent 
 
 ## Installation:
-
-If you are unsure you meet the pre-requisites to installing LME, please read our [prerequisites documentation](/docs/markdown/prerequisites.md)
 Please ensure you follow all the configuration steps required below.
 
+**Upgrading**:
+If you are a previous user of LME and wish to upgrade from 1.4 -> 2.0, please see our [upgrade documentation](/docs/markdown/maintenance/upgrading.md).
+
 
 ### Downloading LME:
 **All steps will assume you start in your cloned directory of LME on your ubuntu 22.04 server**
@@ -79,7 +116,7 @@ in `setup` find the configuration for certificate generation and password settin
 `instances.yml` defines the certificates that will get created. 
 The shellscripts initialize accounts and create certificates, and will run from their respective quadlet definitions `lme-setup-accts` and `lme-setup-certs` respectively.
 
-Quadlet configuration for containers is in: `/quadlet/`. These are mapped to the root's systemd unit files, but will execute as the `lmed` user.
+Quadlet configuration for containers is in: `/quadlet/`. These are mapped to the root's systemd unit files, but will execute as a non-privileged user.
 
 \***TO EDIT**:\*
 The only file that really needs to be touched is creating `/config/lme-environment.env`, which sets up the required environment variables
@@ -110,7 +147,7 @@ You can run this installer to run the total install in ansible.
 ```bash
 sudo apt update && sudo apt install -y ansible
 # cd ~/LME-PRIV/lme-2-arch # Or path to your clone of this repo
-ansible-playbook install_lme_local.yml
+ansible-playbook ./scripts/install_lme_local.yml
 ```
 This assumes that you have the repo in `~/LME/`. 
 
@@ -120,7 +157,6 @@ ansible-playbook ./scripts/install_lme_local.yml -e "clone_dir=/path/to/clone/di
 ```
 
 This also assumes your user can sudo without a password. If you need to input a password when you sudo, you can run it with the `-K` flag and it will prompt you for a password. 
-There is a step that will fail, this is expected, it is checking for podman secrets to see if they exist... on an intial install none will exist :) 
 
 #### Steps performed in automated install: 
 TODO finalize this with more words 
@@ -130,16 +166,21 @@ TODO finalize this with more words
 3. Setup Nix
 4. set service user passwords
 5. Install Quadlets
-6. Setup Containers for root
+6. Setup Containers for root: The contianers listed in `$clone_directory/config/containers.txt` will be pulled and tagged
 7. Start lme.service
 
 #### NOTES:
 
-1. `/opt/lme` will be owned by the lmed user, all lme services will run and execute as lmed, and this ensures least privilege in lmed's execution because lmed is a non-admin,unprivileged user.
+1. `/opt/lme` will be owned by root, all lme services will run and execute as unprivileged users. The active lme configuration is stored in `/opt/lme/config`.
 
-3. [this script](/scripts/set_sysctl_limits.sh) is executed via ansible AND will change unprivileged ports to start at 80, to allow kibana to listen on 443 from a user run container. If this is not desired, we will be publishing steps to setup firewall rules using ufw//iptables to manage the firewall on this host at a later time. 
-
-4. the master password will be stored at `/etc/lme/pass.sh` and owned by root, while service user passwords will be stored at `/etc/lme/vault/`
+2. Other relevant directories are listed here: 
+- `/root/.config/containers/containers.conf`: LME will setup a custom podman configuration for secrets management via [ansible vault](https://docs.ansible.com/ansible/latest/cli/ansible-vault.html).
+- `/etc/lme`: storage directory for the master password and user password vault
+- `/etc/lme/pass.sh`: the master password file
+- `/etc/containers/systemd`: directory where LME installs its quadlet service files
+- `/etc/systemd/system`: directory where lme.service is installed
+
+3. the master password will be stored at `/etc/lme/pass.sh` and owned by root, while service user passwords will be stored at `/etc/lme/vault/`
 
 
 ### Verification post install:
@@ -160,15 +201,13 @@ sudo -i journalctl -xu lme.service
 #try resetting failed: 
 sudo -i systemctl reset-failed lme*
 sudo -i systemctl restart lme.service
-```
 
-2. Check you can connect to elasticsearch
-```bash
-#substitute your password below:
-curl -k -u elastic:$(sudo -i ansible-vault view /etc/lme/vault/$(sudo -i podman secret ls | grep elastic | awk '{print $1}') | tr -d '\n') https://localhost:9200
+#also try inspecting container logs: 
+#CONTAINER_NAME=lme-elasticsearch
+sudo -i podman logs -f $CONTAINER_NAME
 ```
 
-3. Check conatiners are running:
+2. Check conatiners are running and healthy:
 ```bash
 sudo -i podman ps --format "{{.Names}} {{.Status}}"
 ``` 
@@ -180,11 +219,19 @@ lme-kibana Up 2 hours (healthy)
 lme-wazuh-manager Up About an hour
 lme-fleet-server Up 50 minutes
 ```
+We are working on getting health check commands for wazuh and fleet, currently they are not integrated
+
+3. Check you can connect to elasticsearch
+```bash
+#substitute your password below:
+curl -k -u elastic:$(sudo -i ansible-vault view /etc/lme/vault/$(sudo -i podman secret ls | grep elastic | awk '{print $1}') | tr -d '\n') https://localhost:9200
+```
 
 4. Check you can connect to kibana
+You can use an ssh proxy to forward a local port to the remote linux host
 ```bash
-#connect via ssh
-ssh -L 8080:localhost:443 [YOUR-LINUX-SERVER]
+#connect via ssh if you need to 
+ssh -L 8080:localhost:5601 [YOUR-LINUX-SERVER]
 #go to browser:
 #https://localhost:8080
 ```
@@ -246,7 +293,8 @@ systemctl start wazuh-agent
 From PowerShell with admin capabilities run the following command
 
 ```
-Invoke-WebRequest -Uri https://packages.wazuh.com/4.x/windows/wazuh-agent-4.7.5-1.msi -OutFile wazuh-agent-4.7.5-1.msi; Start-Process msiexec.exe -ArgumentList '/i wazuh-agent-4.7.5-1.msi /q WAZUH_MANAGER="IPADDRESS OF WAZUH HOST MACHINE"' -Wait -NoNewWindow
+Invoke-WebRequest -Uri https://packages.wazuh.com/4.x/windows/wazuh-agent-4.7.5-1.msi -OutFile wazuh-agent-4.7.5-1.msi;`
+Start-Process msiexec.exe -ArgumentList '/i wazuh-agent-4.7.5-1.msi /q WAZUH_MANAGER="IPADDRESS OF WAZUH HOST MACHINE"' -Wait -NoNewWindow`
 ```
 
 Start the service: 
@@ -265,12 +313,11 @@ NET START Wazuh
 ## Password Encryption:
 Password encryption is enabled using ansible-vault to store all lme user and lme service user passwords at rest.
 We do submit a hash of the password to Have I been pwned to check to see if it is compromised: [READ MORE HERE](https://haveibeenpwned.com/FAQs)
+
 ### where are passwords stored?:
 ```bash
 # Define user-specific paths
-USER_CONFIG_DIR="/root/.config/lme"
-USER_VAULT_DIR="/opt/lme/vault"
-USER_SECRETS_CONF="$USER_CONFIG_DIR/secrets.conf"
+USER_VAULT_DIR="/etc/lme/vault"
 PASSWORD_FILE="/etc/lme/pass.sh"
 ```
 
@@ -288,29 +335,36 @@ lme-user@ubuntu:~/LME-TEST$ sudo -i ${PWD}/scripts/password_management.sh -h
 ### grabbing passwords: 
 To view the appropriate service user password use ansible-vault, as root: 
 ```
+#script:
+$CLONE_DIRECTORY/scripts/extract_secrets.sh -p #to print
+
+#add them as variables to your current shell
+source $CLONE_DIRECTORY/scripts/extract_secrets.sh #without printing values
+source $CLONE_DIRECTORY/scripts/extract_secrets.sh -q #with no output
+
+## manually:
 #where wazuh_api is the service user whose password you want:
 sudo -i ansible-vault view /etc/lme/vault/$(sudo -i podman secret ls | grep wazuh_api | awk '{print $1}')
 ```
 
-
-
 # Documentation: 
 
 ### Logging Guidance
  - [LME in the CLOUD](/docs/markdown/logging-guidance/cloud.md)
- - [Log Retention](/docs/markdown/logging-guidance/retention.md) TODO update to be current
+ - [Log Retention](/docs/markdown/logging-guidance/retention.md) *TODO*: change link to new documentation
  - [Additional Log Types](/docs/markdown/logging-guidance/other-logging.md) 
 
-### Reference: TODO update these to current
- - [FAQ](/docs/markdown/reference/faq.md) 
- - [Troubleshooting](/docs/markdown/reference/troubleshooting.md)
+## Reference: 
+ - [FAQ](/docs/markdown/reference/faq.md) *TODO*
+ - [Troubleshooting](/docs/markdown/reference/troubleshooting.md) *TODO*
  - [Dashboard Descriptions](/docs/markdown/reference/dashboard-descriptions.md)
  - [Guide to Organizational Units](/docs/markdown/chapter1/guide_to_ous.md)
  - [Security Model](/docs/markdown/reference/security-model.md)
- - [DEV NOTES](/docs/markdown/reference/dev-notes)
 
-### Maintenance:
- - [Backups](/docs/markdown/maintenance/backups.md) 
- - [Upgrading](/docs/markdown/maintenance/upgrading.md) 
- - [Certificates](/docs/markdown/maintenance/certificates.md) 
-
+## Maintenance:
+ - [Backups](/docs/markdown/maintenance/backups.md) *TODO* change link to new documentation
+ - [Upgrading 1x -> 2x](/scripts/upgrade/README.md) 
+ - [Certificates](/docs/markdown/maintenance/certificates.md) *TODO* 
+
+## Agents: 
+*TODO* add in docs in new documentation

scripts/install_lme_local.yml → ansible/install_lme_local.yml

@@ -244,7 +244,6 @@
  set_fact:
  ansible_env: "{{ ansible_env | combine({'PATH': ansible_env.PATH ~ ':/nix/var/nix/profiles/default/bin'}) }}"
 
-
  - name: Update PATH in user's profile
  lineinfile:
  path: "~/.profile"
@@ -291,7 +290,9 @@
  args:
  executable: /bin/bash
  ignore_errors: true
-
+ #only fail on a real error
+ failed_when: result.rc != 0 and (result.rc == 1 and result.changed == false)
+
  - name: Set podman secret passwords
  shell: |
  source /root/.profile 
@@ -306,7 +307,8 @@
  - wazuh_api
  - wazuh
  become: yes
- when: result is failed
+ ## only run this when
+ when: result.rc == 1 
 
 - name: Install Quadlets
  hosts: localhost