# LOGAR: Edge-Thin Log Analysis & Temporal Verification System **LOGAR** is an enterprise log aggregation, verification, and anomaly detection architecture designed for heterogeneous server fleets (Windows & Linux). It combines lightweight zero-state edge forwarders with a centralized cloud hub that applies OpenPGP encryption, authenticated TCP streaming, temporal persistence tracking across 12-hour evaluation windows, an automated 4-run rule to filter transient warnings, and immediate pass-through for critical errors before reporting verified anomalies to **Hermes**. --- ## Table of Contents 1. [Core Philosophy](#core-philosophy) 2. [Architecture & Data Flow](#architecture--data-flow) 3. [Security & Cryptographic Model](#security--cryptographic-model) 4. [Cloud-Side Temporal Persistence & 4-Run Rule](#cloud-side-temporal-persistence--4-run-rule) 5. [Agentic Hermes Integration](#agentic-hermes-integration) 6. [Dynamic Machine & Domain Identification](#dynamic-machine--domain-identification) 7. [Repository & Shippables Structure](#repository--shippables-structure) 8. [Getting Started & Installation](#getting-started--installation) 9. [Running Tests](#running-tests) 10. [Automated Releases via Gitea Actions](#automated-releases-via-gitea-actions) --- ## Core Philosophy ### 1. Edge Thinness & Zero State Site agents running on Windows and Linux act strictly as lightweight forwarders: - **No Local Database**: Clients maintain zero state and no local SQLite or cache files. - **Source-Level Filtering**: Agents upload candidate entries spanning from informational events up to errors (`INFO`, `WARNING`, `ERROR`, `CRITICAL`), while stripping verbose debugging noise (`DEBUG`, audit entries) and skipping any entries older than 24 hours. - **End-to-End Encryption**: Logs are encrypted using the server's OpenPGP public key before leaving the edge node. - **Secure TCP Sockets**: Ingestion occurs over low-overhead authenticated TCP sockets rather than bulky HTTP/HTTPS endpoints. ### 2. Cloud-Side Temporal Persistence & Severity Routing The central Python/TCP hub handles the heavy lifting: - State tracking is managed centrally in SQLite (`logar_state.db`). - Candidate issues are evaluated over a **12-hour temporal evaluation window**. - **Warning Persistence (4-Run Rule)**: `WARNING` level issues must persist across **at least 4 consecutive runs / cycles** within the 12-hour window to be confirmed as genuine anomalies, automatically filtering out transient blips. - **Immediate Error Pass**: Critical errors (`ERROR`, `CRITICAL`, `FATAL`) bypass the 4-run threshold and are promoted immediately to `VERIFIED` on their first occurrence. ### 3. Agentic Integration with Hermes Instead of human engineers manually diving through noisy logs, **Hermes** ingests pre-filtered anomalies directly from the cloud hub (`GET /api/hermes/report`), treating verified errors and 4-run validated warnings as actionable system artifacts to trigger precise team notifications. --- ## Architecture & Data Flow ```mermaid graph TB subgraph Edge Nodes [State-Tracking Edge Forwarders] W[Win_Client.py / Win_Client.exe / Win_Client.pyz
Windows Event Log Application] L[Linux_Client.py / Linux_Client.bin
systemd journalctl -p info] end subgraph Security Layer [Security & Framing] E[OpenPGP Payload Encryption
Server Public Key & Fingerprint] S[Length-Prefixed Framing
4-byte Big-Endian + Auth Envelope] end subgraph Cloud Hub [LOGAR Central Server Hub] TCP[Authenticated TCP Listener
Port 9443] DEC[OpenPGP Decryption
Server Private Key] DB[(SQLite Persistence
active_issues & ingest_runs)] RULE{12h Window &
4-Run Rule} end subgraph Agentic Reporting [Downstream Integration] API[FastAPI / Uvicorn Reporting
Port 8443] HERMES[Hermes Agent
GET /api/hermes/report] end W --> E L --> E E --> S S -->|TCP Stream| TCP TCP --> DEC DEC --> RULE RULE --> DB DB --> API API --> HERMES ``` --- ## Security & Cryptographic Model ### Pure-Python OpenPGP (RFC 4880) - **Zero OS Binary Dependency**: Utilizes `pgpy` and `cryptography` in pure Python. **No native GnuPG or `gpg` binary installation is required** on the server, Windows nodes, or Linux nodes. - **First-Run Automatic Key Generation**: On the first launch, if `server_config.json` is missing, `Server.py` automatically generates: - An OpenPGP RSA 2048 keypair with encryption-only usage flags. - An armored private key (`private_key`) and public key (`public_key`). - A SHA-256 public encryption fingerprint (`server_fingerprint`). - A cryptographically random authentication secret token (`auth_token`). - **Client Configuration Exporter**: ```bash python src/Server.py --create-client-config --server-host 127.0.0.1 --server-port 9443 --client-out client_config.json ``` Produces an anonymous client config containing only the server socket coordinates, authentication token, and the encryption-only public key & fingerprint. - **Socket Protocol Framing**: - `[4 bytes big-endian unsigned int]` : Total envelope length. - `[JSON Envelope]` : ```json { "auth_token": "", "timestamp": "2026-09-03T...", "encrypted_payload": "-----BEGIN PGP MESSAGE-----\n..." } ``` - Unauthorized clients or invalid authentication tokens are rejected immediately. --- ## Cloud-Side Temporal Persistence & 4-Run Rule Incoming candidate logs are tracked in SQLite table `active_issues`: - **Issue Fingerprint**: Formatted as `{site_name}:{server}:{signature}`. - **12-Hour Evaluation Window**: - When an issue is observed, the hub compares `(now - last_seen)`. - If more than 12 hours have passed since the issue was last recorded, the previous window is expired and the cycle resets to `run_count = 1`. - **4-Run Rule for Warnings**: - The 4-run persistence threshold specifically applies to `WARNING` (and `INFO`) events to eliminate transient operational noise. - For each distinct run batch, `run_count` increments. - Warnings with `run_count < 4` are marked as `TRANSIENT` and excluded from Hermes reports. - When `run_count >= 4` within the active 12-hour window, the warning transitions to `VERIFIED`. - **Immediate Verification for Errors**: - High-severity events (`ERROR`, `CRITICAL`, `FATAL`) **always pass immediately**. - On their very first ingestion (`run_count = 1`), errors are promoted directly to `VERIFIED` and surfaced to Hermes without waiting for 4 runs. --- ## Agentic Hermes Integration The server hub serves a REST reporting API (default port `8443`): ### `GET /api/hermes/report` Returns all **verified anomalies** (immediate critical errors and warnings verified after 4 consecutive runs within the 12-hour evaluation window): ```json [ { "fingerprint": "corp.internal:web-app-01.corp.internal:NginxWorkerCrash", "site": "corp.internal", "server": "web-app-01.corp.internal", "signature": "NginxWorkerCrash", "severity": "ERROR", "message": "Worker process 4120 terminated with signal 11", "os_type": "linux", "first_seen": "2026-09-03T09:00:00+00:00", "last_seen": "2026-09-03T21:00:00+00:00", "consecutive_runs": 4, "evaluation_window": "12h", "verified": true, "status": "VERIFIED" } ] ``` ### `GET /api/hermes/all` Diagnostic endpoint listing all active issues (both `TRANSIENT` candidate blips and `VERIFIED` anomalies). ### `GET /health` Returns hub health, encryption fingerprint, and listener ports. --- ## Dynamic Machine & Domain Identification Client configurations intentionally contain **no machine name or site name**. Both forwarders dynamically identify their host and domain at runtime via `get_machine_identifier()`: 1. **Fully Qualified Domain Name (FQDN)**: Checked via `socket.getfqdn()`. 2. **OS-Specific Domain Discovery**: - **Windows**: Checks Active Directory environment variable `USERDNSDOMAIN` / `USERDOMAIN`. - **Linux**: Parses `/etc/resolv.conf` `domain` and `search` directives. 3. **Reverse DNS Lookup**: Resolves canonical hostname via `socket.gethostbyaddr`. 4. **Fallback**: Local hostname `socket.gethostname()`. The server automatically infers site attribution from domain qualifiers (e.g. `node01.corp.internal` $\rightarrow$ site `corp.internal`). --- ## Repository & Shippables Structure ``` LOGAR/ ├── .gitea/ │ └── workflows/ │ ├── ci.yml # Continuous Integration automated test suite (runs on every push) │ ├── release-linux.yml # Linux release workflow (compiles Server.bin, Linux_Client.bin, checksums) │ └── release-windows.yml # Windows release workflow (compiles Server.exe, Win_Client.exe, checksums) ├── compilation/ # Build, packaging, and release automation tools │ ├── package_dist.py # Multi-platform standalone binary packaging script │ ├── requirements.txt # Unified project dependencies │ └── upload_release.py # Direct Gitea REST API release asset publisher ├── src/ # Core application source modules │ ├── __init__.py │ ├── Server.py # Central TCP server and Hermes API │ ├── Win_Client.py # Windows edge forwarder │ └── Linux_Client.py # Linux edge forwarder ├── tests/ # Automated test suites │ ├── test_linux_client.py # Linux client unit tests │ ├── test_pipeline.py # End-to-end integration test │ ├── test_server.py # Server unit tests │ └── test_win_client.py # Windows client unit tests ├── .gitignore # Ignore venv, caches, DBs, and private keys ├── README.md # Comprehensive documentation ├── RELEASE_NOTES.md # Release history and changelog ├── server_config.sample.json # Central server sample configuration └── out/ # Standalone deployment documentation & sample configs ├── linux_server/ │ ├── README.md # Linux systemd service installation & hub guide │ └── server_config.sample.json # Reference server configuration ├── win_server/ │ ├── README.md # Windows service (NSSM/Task Scheduler) installation guide │ └── server_config.sample.json # Reference server configuration ├── linux_client/ │ ├── README.md # Linux service & timer installation guide │ └── client_config.sample.json # Reference client configuration └── win_client/ ├── README.md # Windows service installation & configuration guide └── client_config.sample.json # Reference client configuration ``` --- ## Getting Started & Installation ### 1. Central Server Hub 1. **Install dependencies**: ```bash pip install -r compilation/requirements.txt ``` 2. **Start the server** (generates `server_config.json` and keypair on first run): ```bash python src/Server.py ``` 3. **Export a client configuration**: ```bash python src/Server.py --create-client-config --server-host --server-port 9443 --client-out client_config.json ``` ### 2. Windows Client Deployment #### Option A: Precompiled Standalone Executable (Recommended) 1. Download `Win_Client.exe` (or `Win_Client.pyz`) from the repository releases. 2. Place `client_config.json` (exported from the server) in the same directory. 3. Run manually or schedule via Task Scheduler (every 3 hours): ```powershell .\Win_Client.exe --hours 24 ``` #### Option B: Python Source Execution 1. Copy `Win_Client.py`, `requirements.txt`, and `client_config.json` to the target machine. 2. Install client dependencies: ```powershell python -m pip install -r requirements.txt ``` 3. Run manually or schedule via Task Scheduler: ```powershell python Win_Client.py --hours 24 ``` ### 3. Linux Client Deployment #### Option A: Precompiled Standalone Binary (Recommended) 1. Download `Linux_Client.bin` from the repository releases. 2. Place `Linux_Client.bin` and `client_config.json` into `/opt/logar/` and make it executable: ```bash chmod +x /opt/logar/Linux_Client.bin ``` 3. Run via cron or systemd timer: ```bash 0 */3 * * * /opt/logar/Linux_Client.bin --hours 24 ``` #### Option B: Python Source Execution 1. Copy `Linux_Client.py`, `requirements.txt`, and `client_config.json` to `/opt/logar/`. 2. Install client dependencies: ```bash python3 -m pip install -r requirements.txt ``` 3. (Optional) Run `out/linux_client/build_bin.sh` to compile a standalone ELF binary locally if desired. 4. Run via cron or systemd timer: ```bash 0 */3 * * * python3 /opt/logar/Linux_Client.py --hours 24 ``` --- ## Running Tests ### 1. Component-Specific Unit Tests The test suite is located in `tests/` and exercises all components: ```bash # Run all unit tests python -m unittest discover -s tests # Or run component tests individually: python -m unittest tests/test_server.py python -m unittest tests/test_win_client.py python -m unittest tests/test_linux_client.py ``` ### 2. End-to-End Pipeline Integration Test The pipeline test exercises invalid token rejection, encrypted socket streaming, database persistence, status promotion upon the 4th run, and the Hermes API report output. 1. **Start the server** in Shell 1 (creates `server_config.json` on first run): ```bash python src/Server.py ``` 2. **Export client configuration** in Shell 2 (required for testing): ```bash python src/Server.py --create-client-config --server-host 127.0.0.1 --server-port 9443 --client-out client_config.json ``` 3. **Execute the integration test** in Shell 2: ```bash python tests/test_pipeline.py ``` ## Continuous Integration via Gitea Actions Continuous integration is automated via [`.gitea/workflows/ci.yml`](.gitea/workflows/ci.yml) and triggers automatically on **every push** and pull request: 1. **Syntax Compilation**: Validates all Python scripts (`src/Server.py`, `src/Win_Client.py`, `src/Linux_Client.py`, `compilation/package_dist.py`, `compilation/upload_release.py`, `tests/test_pipeline.py`, and test suites). 2. **Component Unit Tests**: Discovers and runs all unit tests in `tests/` (`test_server.py`, `test_win_client.py`, `test_linux_client.py`). 3. **End-to-End Pipeline Verification**: Automatically spins up the LOGAR server hub, generates test configs, runs `tests/test_pipeline.py` (testing socket authentication, 4-run rule persistence, Hermes API report, and client integrations), and shuts down the test instance. --- ## Automated Releases via Gitea Actions Release builds are automated via two dedicated Gitea Actions workflows running concurrently on native platform runners: - [`.gitea/workflows/release-linux.yml`](.gitea/workflows/release-linux.yml) (`ubuntu-latest`) - [`.gitea/workflows/release-windows.yml`](.gitea/workflows/release-windows.yml) (`windows-latest`) ### Publishing a Release Whenever you want to release a new version with compiled standalone binaries: ```bash git tag v1.0.2 git push origin v1.0.2 ``` *(You can also trigger builds manually via the Gitea UI using the **Run workflow** button (`workflow_dispatch`) on either workflow).* ### Automated Multi-Platform Compilation: 1. **Linux Runner** (`release-linux.yml`): - Compiles native Linux ELF executables: `Linux_Client.bin` and `Server.bin`. - Generates dedicated SHA-256 checksum files: - `linux_client_sha256sum` (verification for `Linux_Client.bin`) - `linux_agent_sha256sum` (alias for client/agent integrations) - `linux_server_sha256sum` (verification for `Server.bin`) - `SHA256SUMS_linux.txt` (summary manifest) - Attaches all Linux assets to the Gitea release. 2. **Windows Runner** (`release-windows.yml`): - Compiles native Windows PE executables: `Win_Client.exe` and `Server.exe`. - Generates dedicated SHA-256 checksum files: - `win_client_sha256sum` (verification for `Win_Client.exe`) - `win_agent_sha256sum` (alias for client/agent integrations) - `win_server_sha256sum` (verification for `Server.exe`) - `SHA256SUMS_windows.txt` (summary manifest) - Attaches all Windows assets to the Gitea release. 3. **Concurrent Publishing & Conflict Handling**: `upload_release.py` includes automatic retry and conflict resolution so concurrent Windows and Linux runners attach their respective assets to the release without collision. ### Verifying Checksums - On Linux: ```bash sha256sum -c linux_client_sha256sum # or sha256sum -c linux_server_sha256sum ``` - On Windows (PowerShell): ```powershell Get-FileHash .\Win_Client.exe -Algorithm SHA256 Get-Content .\win_client_sha256sum ``` ### Local Packaging & Manual Upload You can also compile and package binaries locally anytime: ```bash # Windows py -3.12 compilation/package_dist.py --target windows # Linux python3 compilation/package_dist.py --target linux ``` To upload local builds directly to Gitea: ```powershell python compilation/upload_release.py --tag v1.0.2 --token ``` *(Environment variables `GITEA_TOKEN`, `GITEA_SERVER_URL`, `GITEA_REPOSITORY`, and `GITEA_REF_NAME` are also supported automatically).*