# LOGAR: Edge-Thin Log Analysis & Temporal Verification System **LOGAR** is an enterprise log aggregation, verification, and anomaly detection architecture designed for heterogeneous server fleets (Windows & Linux). It combines lightweight zero-state edge forwarders with a centralized cloud hub that applies mutual TLS 1.3 (**mTLS**) authentication, dynamic PKI licensing and quota management, temporal persistence tracking across 12-hour evaluation windows, and direct anomaly pass-through enabling **Hermes AI** to perform all deep reasoning, anomaly evaluation, and autonomous decision-making. --- ## Table of Contents 1. [Core Philosophy](#core-philosophy) 2. [Architecture & Data Flow](#architecture--data-flow) 3. [mTLS Security, Dynamic PKI & Licensing](#mtls-security-dynamic-pki--licensing) 4. [Cloud-Side Temporal Persistence & Direct Hermes Reasoning](#cloud-side-temporal-persistence--direct-hermes-reasoning) 5. [Agentic Hermes & Client Management API](#agentic-hermes--client-management-api) 6. [Dynamic Machine & Domain Identification](#dynamic-machine--domain-identification) 7. [Automated Service Installers (Linux & Windows)](#automated-service-installers-linux--windows) 8. [Repository & Shippables Structure](#repository--shippables-structure) 9. [Getting Started & Deployment](#getting-started--deployment) 10. [Running Tests](#running-tests) 11. [Automated Releases via Gitea Actions](#automated-releases-via-gitea-actions) --- ## Core Philosophy ### 1. Edge Thinness & Zero State Site agents running on Windows and Linux act strictly as lightweight forwarders: - **No Local Database**: Clients maintain zero local SQLite or heavy cache files. - **Source-Level Filtering**: Agents stream candidate entries from informational events up to errors (`INFO`, `WARNING`, `ERROR`, `CRITICAL`), stripping verbose debugging noise (`DEBUG`, trace entries) and skipping events older than 24 hours. - **Transport Security (mTLS 1.3)**: Logs are streamed directly over mutual TLS 1.3 sockets with hardware-bound / machine-unique client certificates. - **Zero Configuration Overhead**: Clients auto-bootstrap certificate enrollment on first run if configured with an enrollment secret. ### 2. Cloud-Side Temporal Persistence & Direct Hermes Reasoning The central Python hub handles state and temporal tracking: - State tracking is managed centrally in SQLite (`logar_state.db`). - Candidate issues are tracked over a **12-hour temporal evaluation window** with consecutive run counts and first/last-seen timestamps. - **Direct Hermes Intelligence**: All ingested actionable anomalies (`WARNING`, `ERROR`, `INFO`) are immediately promoted to `VERIFIED` and surfaced directly to Hermes. Hermes performs all multi-factor reasoning, pattern evaluation, and contextual thinking rather than relying on arbitrary hardcoded threshold rules. ### 3. Agentic Integration with Hermes Instead of engineers manually sifting through logs, **Hermes** ingests verified anomalies directly from the cloud hub (`GET /api/hermes/report`), treating errors and warnings with full temporal metadata (run counts, timestamps) as actionable system artifacts to trigger precise notifications and remediations. --- ## Architecture & Data Flow ```mermaid graph TB subgraph Edge Nodes [Lightweight Edge Forwarders] W[Win_Client.exe / Win_Client.py
Windows Event Log Ingestion] L[Linux_Client.bin / Linux_Client.py
systemd journalctl -p warning] end subgraph Enrollment [Dynamic PKI & Licensing] ENR[POST /api/client/enroll
License Quota & Secret Validation] CA[Internal Root CA
Signs RSA-2048 Client Cert] end subgraph Transport [mTLS 1.3 Security Layer] MTLS[Mutual TLS 1.3 Handshake
Port 9443 - Client Cert Required] AUTH[Extract Client CN & Fingerprint
Validate Active License in SQLite] end subgraph Hub [LOGAR Server Hub] INGEST[Length-Prefixed Frame Ingestion] DB[(SQLite Persistence
active_issues, clients, license_config)] RULE{12h Window &
Temporal Ingestion} end subgraph Downstream [Hermes Agent & Monitoring] API[FastAPI Reporting & Management
Port 8443] HERMES[Hermes Agent
GET /api/hermes/report] end W -->|Auto-Enrollment| ENR L -->|Auto-Enrollment| ENR ENR --> CA CA -->|ca.crt, client.crt, client.key| W CA -->|ca.crt, client.crt, client.key| L W -->|mTLS Stream| MTLS L -->|mTLS Stream| MTLS MTLS --> AUTH AUTH --> INGEST INGEST --> RULE RULE --> DB DB --> API API --> HERMES ``` --- ## mTLS Security, Dynamic PKI & Fleet Licensing ### 1. TLS 1.3 Mutual Authentication (mTLS) - **Port 9443**: Ingestion occurs exclusively over TLS 1.3 sockets with `ssl.CERT_REQUIRED`. - Both the hub and edge clients verify each other's certificates: - Client verifies server certificate against `ca.crt`. - Server verifies client certificate against the Subordinate/Root CA. - **Client CN Identification & Revocation**: In the TLS handshake, the server extracts the `commonName` attribute (`client_id`), validates that the client is marked `active` in the `clients` table, updates the `last_seen` timestamp, and drops unregistered or revoked certificates immediately. ### 2. Master License Authority (`logar.eibl.tech`), Multi-Tenancy & Subordinate CA - **Master Root Authority**: Standalone PKI repository & submodule ([`LOGAR-LICENSING/`](file:///u:/max/Desktop/LOGAR/LOGAR-LICENSING/README.md), upstream at [`https://gitea.eibl.tech/me0nline/LOGAR-LICENSING.git`](https://gitea.eibl.tech/me0nline/LOGAR-LICENSING.git)) deployed at `logar.eibl.tech`. - **Multi-Tenant Fleet Isolation & User Management**: - Supports multiple isolated **Tenants** (e.g. `DEFAULT`, `CORP_A`, `PRODUCTION_FLEET`) with live tenant filtering. - Built-in user management with PBKDF2 password hashing, RBAC (Administrators & Operators), and self-service password updates. - **Custom Enterprise X.509 OIDs**: - `OID_LOGAR_SEATS (1.3.6.1.4.1.59999.1.1)`: Embeds the exact licensed seat quota directly inside the signed Subordinate CA certificate. - `OID_LOGAR_LICENSE_ID (1.3.6.1.4.1.59999.1.2)`: Uniquely tracks enterprise license entitlement. - `OID_LOGAR_ISSUER_URL (1.3.6.1.4.1.59999.1.3)`: Embedded endpoint coordinate (`https://logar.eibl.tech`). - **Bootstrap `.lic` Packages & Instant WebSocket Auto-Enrollment**: - The License Authority (`logar.eibl.tech`) generates initial **Bootstrap `.lic` Packages** containing Master Root CA, dedicated communication credentials (`communication_cert` + `communication_key`), CSR templates, assigned tenant, and WebSocket coordinates bound directly to the authority server host. - **1-Click Import & Live Signing**: Importing the `.lic` file on the client server (`python src/Server.py --import-lic ` or via WebUI) connects directly via **WebSocket**, submits the server Sub-CA CSR, and immediately installs the officially signed Sub-CA license certificate. - **Self-Updating Client Servers & Live WebSocket Sync**: - Client servers update themselves against the master authority (`wss://logar.eibl.tech/ws/license`) over WebSocket (`python src/Server.py --sync-license` or via WebUI / automated watchdog). - Seat quotas can be increased with each renewal or updated on-demand in the License Authority WebUI. - The client server automatically receives the updated seat quota in its signed certificate, updates its SQLite database and runtime configuration, and reloads its in-flight `ssl.SSLContext` without dropping client connections or restarting services. - **Automated Container Pipeline & Ephemeral NPM Pass-Through**: - The license authority repository includes a dedicated CI/CD workflow ([`.gitea/workflows/build-container.yml`](file:///u:/max/Desktop/LOGAR/LOGAR-LICENSING/.gitea/workflows/build-container.yml)) publishing to `gitea.eibl.tech/me0nline/logar-licensing:latest` (authenticated via `giteatoken` secret). - Works seamlessly with ephemeral Nginx Proxy Manager (NPM) by enabling the standard **Websockets Support** toggle for `logar.eibl.tech`. - Admin WebUI is protected by authenticated login with user and tenant administration. ### 3. Integrated Fleet Licensing & WebUI Dashboard - **WebUI URL**: Accessible at `http://:8443/webui` (or `/`). - **Visual Seat Quota Meter**: Real-time progress bar displaying active forwarders vs. allocated seat limit. - **Licensing Actions**: - 1-click **Export .lic Request** file download. - Interactive **View & Copy CA CSR** modal. - Drag-and-drop or paste **Import License (.lic / ca.crt)** modal. - **Sync & Renew via mTLS** button for immediate upstream synchronization and seat expansion. - **Seat Management & Revocation Table**: - Lists all connected Windows and Linux edge forwarders. - **Revoke Seat**: Instantly drops and blocks the client at the TLS socket handshake layer. - **Reactivate Seat**: Restores the client to active status if within seat quota limits. ### 4. Cloud-Side In-Flight Certificate Watchdog - **Continuous Hub PKI Watchdog**: - The server hub runs a continuous background watchdog coroutine (`cert_validity_watchdog`, running every 12 hours). - Automatically renews Subordinate CA certificates over mTLS with `logar.eibl.tech` and Server TLS certificates in-flight, dynamically reloading the active `ssl.SSLContext`. - **Client Proactive Check & Reactive Self-Healing**: - **Proactive Renewal**: Edge clients inspect `client.crt` before every run cycle. If expiring within 30 days, they contact `/api/client/enroll` to renew. - **Reactive Self-Healing**: If the server CA rotates or a handshake error occurs, edge clients catch the exception, re-bootstrap enrollment against the hub, and re-establish streaming cleanly. --- ## Cloud-Side Temporal Persistence & Direct Hermes Reasoning Incoming candidate logs are tracked in SQLite table `active_issues`: - **Issue Fingerprint**: Formatted as `{site_name}:{server}:{signature}`. - **12-Hour Evaluation Window**: - The hub compares `(now - last_seen)`. - If more than 12 hours have elapsed since the issue was last recorded, the previous window expires and the cycle resets to `run_count = 1`. - **Direct Verification & Temporal Telemetry**: - Ingested anomalies are immediately promoted to `VERIFIED` and made available to Hermes with complete temporal metadata (first seen, last seen, consecutive run counts). - Hermes performs all contextual analysis, pattern recognition, and alerting logic. --- ## Agentic Hermes & Client Management API The server hub exposes a management and reporting REST API (default port `8443`): ### `POST /api/client/enroll` Client enrollment endpoint: - **Request**: ```json { "client_id": "web-worker-01.corp.internal", "hostname": "web-worker-01", "os": "linux", "enrollment_secret": "" } ``` - **Response**: ```json { "ca_cert": "-----BEGIN CERTIFICATE-----\n...", "client_cert": "-----BEGIN CERTIFICATE-----\n...", "client_key": "-----BEGIN RSA PRIVATE KEY-----\n..." } ``` ### `GET /api/clients` Returns seat quota status and registered client telemetry: ```json { "active_seats": 2, "max_seats": 10, "clients": [ { "client_id": "web-worker-01.corp.internal", "hostname": "web-worker-01", "os_type": "linux", "cert_fingerprint": "7D5B660B...", "status": "active", "first_seen": "2026-09-04 18:00:00", "last_seen": "2026-09-04 19:15:00" } ] } ``` ### `GET /api/hermes/report` Returns all **verified anomalies** within the active 12-hour evaluation window with full temporal metadata for direct Hermes reasoning: ```json [ { "fingerprint": "corp.internal:web-worker-01.corp.internal:PostgresPoolExhausted", "site": "corp.internal", "server": "web-worker-01.corp.internal", "signature": "PostgresPoolExhausted", "severity": "WARNING", "message": "Connection pool saturated (>95%)", "os_type": "linux", "first_seen": "2026-09-04T07:00:00+00:00", "last_seen": "2026-09-04T19:00:00+00:00", "consecutive_runs": 4, "evaluation_window": "12h", "verified": true, "status": "VERIFIED" } ] ``` ### `GET /api/hermes/all` Diagnostic endpoint listing all tracked anomalies in SQLite. ### `GET /health` Returns hub health, encryption fingerprint, listener ports, and mTLS status. --- ## Dynamic Machine & Domain Identification Client configurations intentionally contain **no hardcoded machine name or site name**. Both forwarders dynamically identify their host and domain at runtime via `get_machine_identifier()`: 1. **Fully Qualified Domain Name (FQDN)**: Checked via `socket.getfqdn()`. 2. **OS-Specific Domain Discovery**: - **Windows**: Checks Active Directory environment variables (`USERDNSDOMAIN`, `USERDOMAIN`). - **Linux**: Parses `/etc/resolv.conf` `domain` and `search` directives. 3. **Reverse DNS Lookup**: Resolves canonical hostname via `socket.gethostbyaddr`. 4. **Fallback**: Local hostname `socket.gethostname()`. The server automatically infers site attribution from domain qualifiers (e.g. `node01.corp.internal` $\rightarrow$ site `corp.internal`). --- ## Automated Service Installers (Linux & Windows) LOGAR provides production-grade installation scripts and installer builders for automated service deployment: ### 1. Linux Service Installers - **Client Installer (`compilation/install_linux_client.sh`)**: - Non-interactive script deploying to `/opt/logar-client`. - Automatically queries `/etc/machine-id` and enrolls with the hub via `curl`. - Installs and enables `logar-client.service` systemd unit. ```bash sudo ./compilation/install_linux_client.sh "http://hub.example.com:8443" "" ``` - **Server Installer (`compilation/install_linux_server.sh`)**: - Deploys server to `/opt/logar-server`. - Configures logging and installs `logar-server.service` with `LimitNOFILE=65536`. ```bash sudo ./compilation/install_linux_server.sh ``` ### 2. Windows Inno Setup Installers - Built using **Inno Setup 6** and bundled with **NSSM** (`compilation/nssm.exe`): - **Client Setup (`compilation/installer_client.iss`)**: Compiles `LOGAR-Client-Setup.exe`. Installs `Win_Client.exe` into `{autopf}\LOGAR`, sets up `LOGAR_Client` service via NSSM with stdout/stderr redirection to `{commonappdata}\LOGAR\client.log`, and starts the service. Clean uninstallation stops and removes the service. - **Server Setup (`compilation/installer_server.iss`)**: Compiles `LOGAR-Server-Setup.exe`. Installs `Server.exe` and sets up `LOGAR_Server` Windows service via NSSM. --- ## Repository & Shippables Structure ``` LOGAR/ ├── .gitea/ │ └── workflows/ │ ├── ci.yml # CI pipeline: syntax, 25 unit tests & mTLS pipeline test │ └── release.yml # Consolidated release workflow (Linux binaries & Windows installers) ├── compilation/ # Packaging, installers, and release automation │ ├── install_linux_client.sh # Automated Linux client systemd installation script │ ├── install_linux_server.sh # Automated Linux server systemd installation script │ ├── installer_client.iss # Inno Setup Windows Client installer script │ ├── installer_server.iss # Inno Setup Windows Server installer script │ ├── nssm.exe # Official 64-bit NSSM service manager binary │ ├── package_dist.py # Standalone binary compiler & packager │ ├── requirements.txt # Unified project dependencies │ └── upload_release.py # Gitea REST API release asset publisher ├── src/ # Core application source modules │ ├── __init__.py │ ├── Server.py # Central mTLS server, temporal engine, and Hermes REST API │ ├── server_enrollment.py # Dynamic PKI, Root CA, and client certificate generator │ ├── Win_Client.py # Windows edge forwarder with auto-enrollment │ └── Linux_Client.py # Linux edge forwarder with auto-enrollment ├── tests/ # Automated test suites │ ├── test_linux_client.py # Linux client unit tests & mTLS certificate validation │ ├── test_pipeline.py # End-to-end mTLS integration & direct anomaly verification test │ ├── test_server.py # Server unit tests, PKI generation, and seat quota tests │ └── test_win_client.py # Windows client unit tests & mTLS certificate validation ├── .gitignore # Ignores venv, caches, DBs, and private keys ├── README.md # Architecture and usage documentation ├── RELEASE_NOTES.md # Release history and changelog ├── server_config.sample.json # Reference server configuration └── out/ # Component guides and sample configs ├── linux_server/ │ ├── README.md │ └── server_config.sample.json ├── win_server/ │ ├── README.md │ └── server_config.sample.json ├── linux_client/ │ ├── README.md │ └── client_config.sample.json └── win_client/ ├── README.md └── client_config.sample.json ``` --- ## Getting Started & Deployment ### 1. Central Server Hub 1. **Install dependencies**: ```bash pip install -r compilation/requirements.txt ``` 2. **Start the server** (generates `server_config.json`, Root CA, and server certs on first run): ```bash python src/Server.py ``` 3. **Export a client configuration**: ```bash python src/Server.py --create-client-config --server-host --server-port 9443 --client-out client_config.json ``` ### 2. Windows Client Deployment 1. Download `LOGAR-Client-Setup.exe` from releases and run it, or place `Win_Client.exe` and `client_config.json` in `C:\Program Files\LOGAR`. 2. On first run with `client_config.json`, `Win_Client.exe` automatically enrolls with the hub, receives its mTLS certificates, and establishes secure streaming. ### 3. Linux Client Deployment 1. Run the automated installer: ```bash sudo ./compilation/install_linux_client.sh "http://:8443" "" ``` 2. The installer enrolls the client, configures `/etc/logar/certs`, and activates `logar-client.service`. --- ## Running Tests ### 1. Component Unit Tests ```bash python -m unittest discover -s tests -v ``` Runs all **25 unit tests**, covering: - Dynamic Root CA generation and server TLS certificate issuance. - Dynamic client certificate issuance with CN and authority key extensions. - Enrollment secret authentication, seat limits, and certificate revocation. - Proactive certificate validity checks, Root CA auto-renewal, and in-flight server SSLContext reload. - Windows & Linux event log collection, deduplication, and mTLS certificate verification. - 12-hour evaluation window and direct Hermes anomaly verification. ### 2. End-to-End Pipeline Integration Test ```bash # 1. Initialize test configuration python src/Server.py --create-client-config --server-host 127.0.0.1 --server-port 9443 --client-out client_config.json # 2. Launch server in background python src/Server.py & # 3. Run integration test python tests/test_pipeline.py ``` Tests client enrollment, secret rejection, mTLS TLS 1.3 socket handshake, direct warning promotion, immediate error promotion, and Hermes report output. --- ## Releases & Binary Distribution Releases can be built and published through three complementary channels: ### 1. Tag Push Automation (Gitea Actions) Pushing a release tag automatically triggers the build workflows: ```bash git tag v2.0.1 git push origin v2.0.1 ``` The consolidated workflow **`release.yml`** defines two parallel jobs: - **`release-linux`** (`ubuntu-latest`): Compiles standalone native ELF binaries (`Linux_Client.bin`, `Server.bin`) and checksums. - **`release-windows`** (`windows-latest`): Compiles Windows executables (`Win_Client.exe`, `Server.exe`), builds Inno Setup installers, and publishes checksums (requires self-hosted Windows Act Runner). ### 2. Manual Workflow Dispatch (Gitea UI) Workflows can be manually triggered on demand from the Gitea web interface: 1. Navigate to **Actions** $\rightarrow$ **Release Binaries & Installers** (`release.yml`). 2. Click **Run workflow**, set the release tag (defaults to `v2.0.1`), and run. ### 3. Native Local Windows Build & Direct Release Upload For environments without a registered Windows CI runner, Windows executables can be built and published directly to Gitea releases: ```powershell # 1. Build Windows binaries locally python compilation/package_dist.py --target windows # 2. Upload assets and release notes directly to the Gitea release python compilation/upload_release.py --tag v2.0.1 --token --skip-build ```