Implement edge filtering, state tracking, clean out/ directory, and add Gitea CI workflow

2026-09-04 15:32:39 +02:00
parent e634b060df
commit 7052e68589
26 changed files with 1310 additions and 1449 deletions
@@ -14,6 +14,7 @@
 7. [Repository & Shippables Structure](#repository--shippables-structure)
 8. [Getting Started & Installation](#getting-started--installation)
 9. [Running Tests](#running-tests)
+10. [Automated Releases via Gitea Actions](#automated-releases-via-gitea-actions)

 ---

@@ -22,7 +23,7 @@
 ### 1. Edge Thinness & Zero State
 Site agents running on Windows and Linux act strictly as lightweight forwarders:
 - **No Local Database**: Clients maintain zero state and no local SQLite or cache files.
- **Source-Level Noise Stripping**: Conversational, informational, and debugging log noise (`INFO`, `DEBUG`, audit entries) is dropped directly at the source.
+- **Source-Level Filtering**: Agents upload candidate entries spanning from informational events up to errors (`INFO`, `WARNING`, `ERROR`, `CRITICAL`), while stripping verbose debugging noise (`DEBUG`, audit entries) and skipping any entries older than 24 hours.
 - **End-to-End Encryption**: Logs are encrypted using the server's OpenPGP public key before leaving the edge node.
 - **Secure TCP Sockets**: Ingestion occurs over low-overhead authenticated TCP sockets rather than bulky HTTP/HTTPS endpoints.

@@ -41,9 +42,9 @@ Instead of human engineers manually diving through noisy logs, **Hermes** ingest

 ```mermaid
 graph TB
-    subgraph Edge Nodes [Zero-State Edge Forwarders]
-        W[Win_Client.py / Win_Client.exe<br/>Windows Event Log Application]
-        L[Linux_Client.py / Linux_Client.bin<br/>systemd journalctl -p warning]
+    subgraph Edge Nodes [State-Tracking Edge Forwarders]
+        W[Win_Client.py / Win_Client.exe / Win_Client.pyz<br/>Windows Event Log Application]
+        L[Linux_Client.py / Linux_Client.bin<br/>systemd journalctl -p info]
    end

    subgraph Security Layer [Security & Framing]
@@ -171,36 +172,31 @@ The server automatically infers site attribution from domain qualifiers (e.g. `n

 ```
 LOGAR/
+├── .gitea/
+│   └── workflows/
+│       ├── ci.yml                  # Continuous Integration automated test suite (runs on every push)
+│       └── release.yml             # Automated standalone binary release workflow (runs on tag v*)
 ├── .gitignore                      # Ignore venv, caches, DBs, and private keys
 ├── requirements.txt                # Unified dependencies
 ├── README.md                       # Comprehensive documentation
 ├── Server.py                       # Central TCP server and Hermes API
 ├── Win_Client.py                   # Windows edge forwarder
 ├── Linux_Client.py                 # Linux edge forwarder
+├── server_config.sample.json       # Central server sample configuration
+├── package_dist.py                 # Multi-platform standalone binary packaging script
+├── upload_release.py               # Direct Gitea REST API release asset publisher
 ├── test_pipeline.py                # End-to-end integration test
-└── out/                            # Standalone shippable distributions
-    ├── server/
-    │   ├── Server.py               # Python source
-    │   ├── server_config.sample.json
-    │   ├── requirements.txt
-    │   ├── README.md
-    │   └── test/
-    │       └── test_server.py      # Server unit tests
+├── tests/                          # Unified unit test suites
+│   ├── test_server.py              # Server unit tests
+│   ├── test_win_client.py          # Windows client unit tests
+│   └── test_linux_client.py        # Linux client unit tests
+└── out/                            # Edge forwarder deployment packages
    ├── win_client/
-    │   ├── Win_Client.py           # Python source
-    │   ├── client_config.sample.json
-    │   ├── requirements.txt
-    │   ├── README.md
-    │   └── test/
-    │       └── test_win_client.py  # Windows client unit tests
+    │   ├── client_config.sample.json # Reference client configuration
+    │   └── README.md               # Windows service installation & configuration guide
    └── linux_client/
-        ├── build_bin.sh            # PyInstaller native ELF compiler script
-        ├── Linux_Client.py         # Python source
-        ├── client_config.sample.json
-        ├── requirements.txt
-        ├── README.md
-        └── test/
-            └── test_linux_client.py# Linux client unit tests
+        ├── client_config.sample.json # Reference client configuration
+        └── README.md               # Linux service installation & configuration guide
 ```

 ---
@@ -224,19 +220,48 @@ LOGAR/

 ### 2. Windows Client Deployment

-1. Copy `Win_Client.py` (and `requirements.txt`) plus `client_config.json` to the target machine.
-2. Run manually or schedule via Task Scheduler (every 3 hours):
+#### Option A: Precompiled Standalone Executable (Recommended)
+1. Download `Win_Client.exe` (or `Win_Client.pyz`) from the repository releases.
+2. Place `client_config.json` (exported from the server) in the same directory.
+3. Run manually or schedule via Task Scheduler (every 3 hours):
   ```powershell
-   python Win_Client.py --hours 6
+   .\Win_Client.exe --hours 24
+   ```
+
+#### Option B: Python Source Execution
+1. Copy `Win_Client.py`, `requirements.txt`, and `client_config.json` to the target machine.
+2. Install client dependencies:
+   ```powershell
+   python -m pip install -r requirements.txt
+   ```
+3. Run manually or schedule via Task Scheduler:
+   ```powershell
+   python Win_Client.py --hours 24
   ```

 ### 3. Linux Client Deployment

-1. Copy `Linux_Client.py` (and `requirements.txt`) plus `client_config.json` to `/opt/logar/`.
-2. (Optional) Run `build_bin.sh` to compile a standalone ELF binary if desired.
+#### Option A: Precompiled Standalone Binary (Recommended)
+1. Download `Linux_Client.bin` from the repository releases.
+2. Place `Linux_Client.bin` and `client_config.json` into `/opt/logar/` and make it executable:
+   ```bash
+   chmod +x /opt/logar/Linux_Client.bin
+   ```
 3. Run via cron or systemd timer:
   ```bash
-   0 */3 * * * python3 /opt/logar/Linux_Client.py --hours 6
+   0 */3 * * * /opt/logar/Linux_Client.bin --hours 24
+   ```
+
+#### Option B: Python Source Execution
+1. Copy `Linux_Client.py`, `requirements.txt`, and `client_config.json` to `/opt/logar/`.
+2. Install client dependencies:
+   ```bash
+   python3 -m pip install -r requirements.txt
+   ```
+3. (Optional) Run `out/linux_client/build_bin.sh` to compile a standalone ELF binary locally if desired.
+4. Run via cron or systemd timer:
+   ```bash
+   0 */3 * * * python3 /opt/logar/Linux_Client.py --hours 24
   ```

 ---
@@ -244,46 +269,69 @@ LOGAR/
 ## Running Tests

 ### 1. Component-Specific Unit Tests
-Each component in `out/` includes its own isolated test suite:
+The test suite is located in `tests/` and exercises all components:

 ```bash
-# Server tests (config generation, SQLite persistence, 4-run rule)
-python out/server/test/test_server.py
+# Run all unit tests
+python -m unittest discover -s tests

-# Windows client tests (config anonymity, machine ID, OpenPGP encryption)
-python out/win_client/test/test_win_client.py
-
-# Linux client tests (config anonymity, journalctl priority filter, OpenPGP)
-python out/linux_client/test/test_linux_client.py
+# Or run component tests individually:
+python -m unittest tests/test_server.py
+python -m unittest tests/test_win_client.py
+python -m unittest tests/test_linux_client.py
 ```

 ### 2. End-to-End Pipeline Integration Test
-Start the server in one shell and run the pipeline test:
-```bash
-python test_pipeline.py
-```
-This tests invalid token rejection, encrypted socket streaming, database persistence, status promotion upon the 4th run, and the Hermes API output.
+The pipeline test exercises invalid token rejection, encrypted socket streaming, database persistence, status promotion upon the 4th run, and the Hermes API report output.
+
+1. **Start the server** in Shell 1 (creates `server_config.json` on first run):
+   ```bash
+   python Server.py
+   ```
+2. **Export client configuration** in Shell 2 (required for testing):
+   ```bash
+   python Server.py --create-client-config --server-host 127.0.0.1 --server-port 9443 --client-out client_config.json
+   ```
+3. **Execute the integration test** in Shell 2:
+   ```bash
+   python test_pipeline.py
+   ```
+
+## Continuous Integration via Gitea Actions
+
+Continuous integration is automated via [`.gitea/workflows/ci.yml`](.gitea/workflows/ci.yml) and triggers automatically on **every push** and pull request:
+1. **Syntax Compilation**: Validates all Python scripts (`Server.py`, `Win_Client.py`, `Linux_Client.py`, `package_dist.py`, `upload_release.py`, `test_pipeline.py`, and test suites).
+2. **Component Unit Tests**: Discovers and runs all unit tests in `tests/` (`test_server.py`, `test_win_client.py`, `test_linux_client.py`).
+3. **End-to-End Pipeline Verification**: Automatically spins up the LOGAR server hub, generates test configs, runs `test_pipeline.py` (testing socket authentication, 4-run rule persistence, Hermes API report, and client integrations), and shuts down the test instance.

 ---

 ## Automated Releases via Gitea Actions

-Releases are automated via [`.gitea/workflows/release.yml`](.gitea/workflows/release.yml) using your Gitea action runner:
+Release builds are automated via [`.gitea/workflows/release.yml`](.gitea/workflows/release.yml) using your Gitea action runner:

 ### Publishing a Release
 Whenever you want to release a new version with compiled standalone binaries:
 ```bash
-git tag v1.0.0
-git push origin v1.0.0
+git tag v1.0.1
+git push origin v1.0.1
 ```

 ### What Gitea Actions Does Automatically:
 1. Gitea runner executes the workflow on tag push.
-2. Runs `package_dist.py` to compile native standalone binaries:
-   - `Linux_Client.bin` (standalone binary)
-   - `Server.bin` (standalone server binary)
+2. Installs Python, system build tools (`binutils`, `zip`), PyInstaller, and project dependencies via `apt-get` and `pip3`.
+3. Runs `package_dist.py` to compile standalone binaries:
+   - `Linux_Client.bin` (native ELF binary compiled with PyInstaller)
+   - `Server.bin` (native server ELF binary compiled with PyInstaller)
   - `Win_Client.pyz` (standalone executable zipapp)
-   - `SHA256SUMS.txt` (checksums)
-3. Publishes the Gitea release using `gitea-release-action` and attaches the compiled binary assets.
+   - `SHA256SUMS.txt` (SHA-256 cryptographic checksums)
+4. Publishes the Gitea release directly via Python (`python3 upload_release.py --skip-build`) using the Gitea REST API to attach the compiled binary assets (avoiding runner Node runtime limitations).

-*(Note: You can also use `upload_release.py` from your Windows machine to upload Windows `.exe` binaries directly if desired).*
+### Building & Publishing Windows Executables (`.exe`) Locally
+Because the Linux Gitea runner compiles ELF binaries, native Windows PE executables (`Win_Client.exe`, `Server.exe`) can be built and published directly from a Windows workstation:
+
+```powershell
+# Compiles Win_Client.exe, Server.exe, Linux_Client.bin, and uploads to Gitea
+python upload_release.py --tag v1.0.0 --token <YOUR_GITEA_TOKEN>
+```
+*(Environment variables `GITEA_TOKEN`, `GITEA_SERVER_URL`, `GITEA_REPOSITORY`, and `GITEA_REF_NAME` are also supported automatically).*