405 lines
20 KiB
Markdown
405 lines
20 KiB
Markdown
# LOGAR: Edge-Thin Log Analysis & Temporal Verification System
|
|
|
|
**LOGAR** is an enterprise log aggregation, verification, and anomaly detection architecture designed for heterogeneous server fleets (Windows & Linux). It combines lightweight zero-state edge forwarders with a centralized cloud hub that applies mutual TLS 1.3 (**mTLS**) authentication, dynamic PKI licensing and quota management, temporal persistence tracking across 12-hour evaluation windows, an automated 4-run rule to filter transient warnings, and immediate pass-through for critical errors before reporting verified anomalies to **Hermes**.
|
|
|
|
---
|
|
|
|
## Table of Contents
|
|
1. [Core Philosophy](#core-philosophy)
|
|
2. [Architecture & Data Flow](#architecture--data-flow)
|
|
3. [mTLS Security, Dynamic PKI & Licensing](#mtls-security-dynamic-pki--licensing)
|
|
4. [Cloud-Side Temporal Persistence & 4-Run Rule](#cloud-side-temporal-persistence--4-run-rule)
|
|
5. [Agentic Hermes & Client Management API](#agentic-hermes--client-management-api)
|
|
6. [Dynamic Machine & Domain Identification](#dynamic-machine--domain-identification)
|
|
7. [Automated Service Installers (Linux & Windows)](#automated-service-installers-linux--windows)
|
|
8. [Repository & Shippables Structure](#repository--shippables-structure)
|
|
9. [Getting Started & Deployment](#getting-started--deployment)
|
|
10. [Running Tests](#running-tests)
|
|
11. [Automated Releases via Gitea Actions](#automated-releases-via-gitea-actions)
|
|
|
|
---
|
|
|
|
## Core Philosophy
|
|
|
|
### 1. Edge Thinness & Zero State
|
|
Site agents running on Windows and Linux act strictly as lightweight forwarders:
|
|
- **No Local Database**: Clients maintain zero local SQLite or heavy cache files.
|
|
- **Source-Level Filtering**: Agents stream candidate entries from informational events up to errors (`INFO`, `WARNING`, `ERROR`, `CRITICAL`), stripping verbose debugging noise (`DEBUG`, trace entries) and skipping events older than 24 hours.
|
|
- **Transport Security (mTLS 1.3)**: Logs are streamed directly over mutual TLS 1.3 sockets with hardware-bound / machine-unique client certificates.
|
|
- **Zero Configuration Overhead**: Clients auto-bootstrap certificate enrollment on first run if configured with an enrollment secret.
|
|
|
|
### 2. Cloud-Side Temporal Persistence & Severity Routing
|
|
The central Python hub handles state and verification:
|
|
- State tracking is managed centrally in SQLite (`logar_state.db`).
|
|
- Candidate issues are evaluated over a **12-hour temporal evaluation window**.
|
|
- **Warning Persistence (4-Run Rule)**: `WARNING` issues must persist across **at least 4 consecutive runs / cycles** within the 12-hour window to be confirmed as genuine anomalies, automatically filtering transient infrastructure blips.
|
|
- **Immediate Error Pass**: Critical errors (`ERROR`, `CRITICAL`, `FATAL`) bypass the 4-run threshold and are promoted immediately to `VERIFIED` on their first occurrence.
|
|
|
|
### 3. Agentic Integration with Hermes
|
|
Instead of engineers manually sifting through logs, **Hermes** ingests pre-filtered anomalies directly from the cloud hub (`GET /api/hermes/report`), treating verified errors and 4-run validated warnings as actionable system artifacts to trigger precise notifications and remediations.
|
|
|
|
---
|
|
|
|
## Architecture & Data Flow
|
|
|
|
```mermaid
|
|
graph TB
|
|
subgraph Edge Nodes [Lightweight Edge Forwarders]
|
|
W[Win_Client.exe / Win_Client.py<br/>Windows Event Log Ingestion]
|
|
L[Linux_Client.bin / Linux_Client.py<br/>systemd journalctl -p warning]
|
|
end
|
|
|
|
subgraph Enrollment [Dynamic PKI & Licensing]
|
|
ENR[POST /api/client/enroll<br/>License Quota & Secret Validation]
|
|
CA[Internal Root CA<br/>Signs RSA-2048 Client Cert]
|
|
end
|
|
|
|
subgraph Transport [mTLS 1.3 Security Layer]
|
|
MTLS[Mutual TLS 1.3 Handshake<br/>Port 9443 - Client Cert Required]
|
|
AUTH[Extract Client CN & Fingerprint<br/>Validate Active License in SQLite]
|
|
end
|
|
|
|
subgraph Hub [LOGAR Server Hub]
|
|
INGEST[Length-Prefixed Frame Ingestion]
|
|
DB[(SQLite Persistence<br/>active_issues, clients, license_config)]
|
|
RULE{12h Window &<br/>4-Run Rule}
|
|
end
|
|
|
|
subgraph Downstream [Hermes Agent & Monitoring]
|
|
API[FastAPI Reporting & Management<br/>Port 8443]
|
|
HERMES[Hermes Agent<br/>GET /api/hermes/report]
|
|
end
|
|
|
|
W -->|Auto-Enrollment| ENR
|
|
L -->|Auto-Enrollment| ENR
|
|
ENR --> CA
|
|
CA -->|ca.crt, client.crt, client.key| W
|
|
CA -->|ca.crt, client.crt, client.key| L
|
|
|
|
W -->|mTLS Stream| MTLS
|
|
L -->|mTLS Stream| MTLS
|
|
MTLS --> AUTH
|
|
AUTH --> INGEST
|
|
INGEST --> RULE
|
|
RULE --> DB
|
|
DB --> API
|
|
API --> HERMES
|
|
```
|
|
|
|
---
|
|
|
|
## mTLS Security, Dynamic PKI & Fleet Licensing
|
|
|
|
### 1. TLS 1.3 Mutual Authentication (mTLS)
|
|
- **Port 9443**: Ingestion occurs exclusively over TLS 1.3 sockets with `ssl.CERT_REQUIRED`.
|
|
- Both the hub and edge clients verify each other's certificates:
|
|
- Client verifies server certificate against `ca.crt`.
|
|
- Server verifies client certificate against the Subordinate/Root CA.
|
|
- **Client CN Identification & Revocation**: In the TLS handshake, the server extracts the `commonName` attribute (`client_id`), validates that the client is marked `active` in the `clients` table, updates the `last_seen` timestamp, and drops unregistered or revoked certificates immediately.
|
|
|
|
### 2. Master License Authority (`logar.eibl.tech`) & Seat-Bearing Subordinate CA
|
|
- **Master Root Authority**: Standalone PKI service ([`src/License_Authority.py`](file:///u:/max/Desktop/LOGAR/src/License_Authority.py)) maintained at `logar.eibl.tech`.
|
|
- **Custom Enterprise X.509 OIDs**:
|
|
- `OID_LOGAR_SEATS (1.3.6.1.4.1.59999.1.1)`: Embeds the exact licensed seat quota directly inside the signed Subordinate CA certificate.
|
|
- `OID_LOGAR_LICENSE_ID (1.3.6.1.4.1.59999.1.2)`: Uniquely tracks enterprise license entitlement.
|
|
- `OID_LOGAR_ISSUER_URL (1.3.6.1.4.1.59999.1.3)`: Embedded endpoint coordinate (`https://logar.eibl.tech`).
|
|
- **Exportable `.lic` Request Files**:
|
|
- The LOGAR server generates its Subordinate CA private key (`certs/ca.key`), Certificate Signing Request (`certs/ca.csr`), and packages them into an exportable `logar_server.lic` file containing the CSR, server fingerprint, and license authority coordinate.
|
|
- Exportable directly from the WebUI or via CLI (`python src/Server.py --export-lic server.lic`).
|
|
- **One-Time Pass & Zero-Touch mTLS Renewals**:
|
|
- The signed CA certificate only needs to be imported **once** into the LOGAR Server (via WebUI or CLI `--import-ca`).
|
|
- Once imported, the server hub's background watchdog monitors `ca.crt` expiration (< 30 days).
|
|
- Renewals are performed **automatically over mTLS**: the LOGAR Server authenticates directly against `https://logar.eibl.tech/api/license/renew` using its existing active certificate and key, receives the renewed Subordinate CA certificate preserving its seat quota, and reloads its in-flight `ssl.SSLContext` without dropping client connections or restarting services.
|
|
|
|
### 3. Integrated Fleet Licensing & WebUI Dashboard
|
|
- **WebUI URL**: Accessible at `http://<server_host>:8443/webui` (or `/`).
|
|
- **Visual Seat Quota Meter**: Real-time progress bar displaying active forwarders vs. allocated seat limit.
|
|
- **Licensing Actions**:
|
|
- 1-click **Export .lic Request** file download.
|
|
- Interactive **View & Copy CA CSR** modal.
|
|
- Drag-and-drop or paste **Import Signed CA Certificate** modal.
|
|
- **Renew via mTLS Now** button for immediate upstream renewal.
|
|
- **Seat Management & Revocation Table**:
|
|
- Lists all connected Windows and Linux edge forwarders.
|
|
- **Revoke Seat**: Instantly drops and blocks the client at the TLS socket handshake layer.
|
|
- **Reactivate Seat**: Restores the client to active status if within seat quota limits.
|
|
|
|
### 4. Cloud-Side In-Flight Certificate Watchdog
|
|
- **Continuous Hub PKI Watchdog**:
|
|
- The server hub runs a continuous background watchdog coroutine (`cert_validity_watchdog`, running every 12 hours).
|
|
- Automatically renews Subordinate CA certificates over mTLS with `logar.eibl.tech` and Server TLS certificates in-flight, dynamically reloading the active `ssl.SSLContext`.
|
|
- **Client Proactive Check & Reactive Self-Healing**:
|
|
- **Proactive Renewal**: Edge clients inspect `client.crt` before every run cycle. If expiring within 30 days, they contact `/api/client/enroll` to renew.
|
|
- **Reactive Self-Healing**: If the server CA rotates or a handshake error occurs, edge clients catch the exception, re-bootstrap enrollment against the hub, and re-establish streaming cleanly.
|
|
|
|
---
|
|
|
|
## Cloud-Side Temporal Persistence & 4-Run Rule
|
|
|
|
Incoming candidate logs are tracked in SQLite table `active_issues`:
|
|
- **Issue Fingerprint**: Formatted as `{site_name}:{server}:{signature}`.
|
|
- **12-Hour Evaluation Window**:
|
|
- The hub compares `(now - last_seen)`.
|
|
- If more than 12 hours have elapsed since the issue was last recorded, the previous window expires and the cycle resets to `run_count = 1`.
|
|
- **4-Run Rule for Warnings**:
|
|
- The 4-run persistence threshold applies to `WARNING` (and `INFO`) events to eliminate transient operational noise.
|
|
- Each distinct run batch increments `run_count`.
|
|
- Warnings with `run_count < 4` are marked as `TRANSIENT` and excluded from Hermes reports.
|
|
- When `run_count >= 4` within the active 12-hour window, the warning transitions to `VERIFIED`.
|
|
- **Immediate Verification for Errors**:
|
|
- High-severity events (`ERROR`, `CRITICAL`, `FATAL`) **always pass immediately**.
|
|
- On their very first ingestion (`run_count = 1`), errors are promoted directly to `VERIFIED` and surfaced to Hermes without waiting for 4 runs.
|
|
|
|
---
|
|
|
|
## Agentic Hermes & Client Management API
|
|
|
|
The server hub exposes a management and reporting REST API (default port `8443`):
|
|
|
|
### `POST /api/client/enroll`
|
|
Client enrollment endpoint:
|
|
- **Request**:
|
|
```json
|
|
{
|
|
"client_id": "web-worker-01.corp.internal",
|
|
"hostname": "web-worker-01",
|
|
"os": "linux",
|
|
"enrollment_secret": "<SECRET>"
|
|
}
|
|
```
|
|
- **Response**:
|
|
```json
|
|
{
|
|
"ca_cert": "-----BEGIN CERTIFICATE-----\n...",
|
|
"client_cert": "-----BEGIN CERTIFICATE-----\n...",
|
|
"client_key": "-----BEGIN RSA PRIVATE KEY-----\n..."
|
|
}
|
|
```
|
|
|
|
### `GET /api/clients`
|
|
Returns seat quota status and registered client telemetry:
|
|
```json
|
|
{
|
|
"active_seats": 2,
|
|
"max_seats": 10,
|
|
"clients": [
|
|
{
|
|
"client_id": "web-worker-01.corp.internal",
|
|
"hostname": "web-worker-01",
|
|
"os_type": "linux",
|
|
"cert_fingerprint": "7D5B660B...",
|
|
"status": "active",
|
|
"first_seen": "2026-09-04 18:00:00",
|
|
"last_seen": "2026-09-04 19:15:00"
|
|
}
|
|
]
|
|
}
|
|
```
|
|
|
|
### `GET /api/hermes/report`
|
|
Returns all **verified anomalies** (immediate critical errors and warnings verified after 4 consecutive runs within the 12-hour window):
|
|
```json
|
|
[
|
|
{
|
|
"fingerprint": "corp.internal:web-worker-01.corp.internal:PostgresPoolExhausted",
|
|
"site": "corp.internal",
|
|
"server": "web-worker-01.corp.internal",
|
|
"signature": "PostgresPoolExhausted",
|
|
"severity": "WARNING",
|
|
"message": "Connection pool saturated (>95%) across 4 runs",
|
|
"os_type": "linux",
|
|
"first_seen": "2026-09-04T07:00:00+00:00",
|
|
"last_seen": "2026-09-04T19:00:00+00:00",
|
|
"consecutive_runs": 4,
|
|
"evaluation_window": "12h",
|
|
"verified": true,
|
|
"status": "VERIFIED"
|
|
}
|
|
]
|
|
```
|
|
|
|
### `GET /api/hermes/all`
|
|
Diagnostic endpoint listing all candidate issues (`TRANSIENT` and `VERIFIED`).
|
|
|
|
### `GET /health`
|
|
Returns hub health, encryption fingerprint, listener ports, and mTLS status.
|
|
|
|
---
|
|
|
|
## Dynamic Machine & Domain Identification
|
|
|
|
Client configurations intentionally contain **no hardcoded machine name or site name**. Both forwarders dynamically identify their host and domain at runtime via `get_machine_identifier()`:
|
|
1. **Fully Qualified Domain Name (FQDN)**: Checked via `socket.getfqdn()`.
|
|
2. **OS-Specific Domain Discovery**:
|
|
- **Windows**: Checks Active Directory environment variables (`USERDNSDOMAIN`, `USERDOMAIN`).
|
|
- **Linux**: Parses `/etc/resolv.conf` `domain` and `search` directives.
|
|
3. **Reverse DNS Lookup**: Resolves canonical hostname via `socket.gethostbyaddr`.
|
|
4. **Fallback**: Local hostname `socket.gethostname()`.
|
|
|
|
The server automatically infers site attribution from domain qualifiers (e.g. `node01.corp.internal` $\rightarrow$ site `corp.internal`).
|
|
|
|
---
|
|
|
|
## Automated Service Installers (Linux & Windows)
|
|
|
|
LOGAR provides production-grade installation scripts and installer builders for automated service deployment:
|
|
|
|
### 1. Linux Service Installers
|
|
- **Client Installer (`compilation/install_linux_client.sh`)**:
|
|
- Non-interactive script deploying to `/opt/logar-client`.
|
|
- Automatically queries `/etc/machine-id` and enrolls with the hub via `curl`.
|
|
- Installs and enables `logar-client.service` systemd unit.
|
|
```bash
|
|
sudo ./compilation/install_linux_client.sh "http://hub.example.com:8443" "<ENROLLMENT_SECRET>"
|
|
```
|
|
- **Server Installer (`compilation/install_linux_server.sh`)**:
|
|
- Deploys server to `/opt/logar-server`.
|
|
- Configures logging and installs `logar-server.service` with `LimitNOFILE=65536`.
|
|
```bash
|
|
sudo ./compilation/install_linux_server.sh
|
|
```
|
|
|
|
### 2. Windows Inno Setup Installers
|
|
- Built using **Inno Setup 6** and bundled with **NSSM** (`compilation/nssm.exe`):
|
|
- **Client Setup (`compilation/installer_client.iss`)**: Compiles `LOGAR-Client-Setup.exe`. Installs `Win_Client.exe` into `{autopf}\LOGAR`, sets up `LOGAR_Client` service via NSSM with stdout/stderr redirection to `{commonappdata}\LOGAR\client.log`, and starts the service. Clean uninstallation stops and removes the service.
|
|
- **Server Setup (`compilation/installer_server.iss`)**: Compiles `LOGAR-Server-Setup.exe`. Installs `Server.exe` and sets up `LOGAR_Server` Windows service via NSSM.
|
|
|
|
---
|
|
|
|
## Repository & Shippables Structure
|
|
|
|
```
|
|
LOGAR/
|
|
├── .gitea/
|
|
│ └── workflows/
|
|
│ ├── ci.yml # CI pipeline: syntax, 25 unit tests & mTLS pipeline test
|
|
│ └── release.yml # Consolidated release workflow (Linux binaries & Windows installers)
|
|
├── compilation/ # Packaging, installers, and release automation
|
|
│ ├── install_linux_client.sh # Automated Linux client systemd installation script
|
|
│ ├── install_linux_server.sh # Automated Linux server systemd installation script
|
|
│ ├── installer_client.iss # Inno Setup Windows Client installer script
|
|
│ ├── installer_server.iss # Inno Setup Windows Server installer script
|
|
│ ├── nssm.exe # Official 64-bit NSSM service manager binary
|
|
│ ├── package_dist.py # Standalone binary compiler & packager
|
|
│ ├── requirements.txt # Unified project dependencies
|
|
│ └── upload_release.py # Gitea REST API release asset publisher
|
|
├── src/ # Core application source modules
|
|
│ ├── __init__.py
|
|
│ ├── Server.py # Central mTLS server, temporal engine, and Hermes REST API
|
|
│ ├── server_enrollment.py # Dynamic PKI, Root CA, and client certificate generator
|
|
│ ├── Win_Client.py # Windows edge forwarder with auto-enrollment
|
|
│ └── Linux_Client.py # Linux edge forwarder with auto-enrollment
|
|
├── tests/ # Automated test suites
|
|
│ ├── test_linux_client.py # Linux client unit tests & mTLS certificate validation
|
|
│ ├── test_pipeline.py # End-to-end mTLS integration & 4-run verification test
|
|
│ ├── test_server.py # Server unit tests, PKI generation, and seat quota tests
|
|
│ └── test_win_client.py # Windows client unit tests & mTLS certificate validation
|
|
├── .gitignore # Ignores venv, caches, DBs, and private keys
|
|
├── README.md # Architecture and usage documentation
|
|
├── RELEASE_NOTES.md # Release history and changelog
|
|
├── server_config.sample.json # Reference server configuration
|
|
└── out/ # Component guides and sample configs
|
|
├── linux_server/
|
|
│ ├── README.md
|
|
│ └── server_config.sample.json
|
|
├── win_server/
|
|
│ ├── README.md
|
|
│ └── server_config.sample.json
|
|
├── linux_client/
|
|
│ ├── README.md
|
|
│ └── client_config.sample.json
|
|
└── win_client/
|
|
├── README.md
|
|
└── client_config.sample.json
|
|
```
|
|
|
|
---
|
|
|
|
## Getting Started & Deployment
|
|
|
|
### 1. Central Server Hub
|
|
|
|
1. **Install dependencies**:
|
|
```bash
|
|
pip install -r compilation/requirements.txt
|
|
```
|
|
2. **Start the server** (generates `server_config.json`, Root CA, and server certs on first run):
|
|
```bash
|
|
python src/Server.py
|
|
```
|
|
3. **Export a client configuration**:
|
|
```bash
|
|
python src/Server.py --create-client-config --server-host <SERVER_IP> --server-port 9443 --client-out client_config.json
|
|
```
|
|
|
|
### 2. Windows Client Deployment
|
|
1. Download `LOGAR-Client-Setup.exe` from releases and run it, or place `Win_Client.exe` and `client_config.json` in `C:\Program Files\LOGAR`.
|
|
2. On first run with `client_config.json`, `Win_Client.exe` automatically enrolls with the hub, receives its mTLS certificates, and establishes secure streaming.
|
|
|
|
### 3. Linux Client Deployment
|
|
1. Run the automated installer:
|
|
```bash
|
|
sudo ./compilation/install_linux_client.sh "http://<HUB_HOST>:8443" "<ENROLLMENT_SECRET>"
|
|
```
|
|
2. The installer enrolls the client, configures `/etc/logar/certs`, and activates `logar-client.service`.
|
|
|
|
---
|
|
|
|
## Running Tests
|
|
|
|
### 1. Component Unit Tests
|
|
```bash
|
|
python -m unittest discover -s tests -v
|
|
```
|
|
Runs all **25 unit tests**, covering:
|
|
- Dynamic Root CA generation and server TLS certificate issuance.
|
|
- Dynamic client certificate issuance with CN and authority key extensions.
|
|
- Enrollment secret authentication, seat limits, and certificate revocation.
|
|
- Proactive certificate validity checks, Root CA auto-renewal, and in-flight server SSLContext reload.
|
|
- Windows & Linux event log collection, deduplication, and mTLS certificate verification.
|
|
- 12-hour evaluation window and 4-run rule progression.
|
|
|
|
### 2. End-to-End Pipeline Integration Test
|
|
```bash
|
|
# 1. Initialize test configuration
|
|
python src/Server.py --create-client-config --server-host 127.0.0.1 --server-port 9443 --client-out client_config.json
|
|
|
|
# 2. Launch server in background
|
|
python src/Server.py &
|
|
|
|
# 3. Run integration test
|
|
python tests/test_pipeline.py
|
|
```
|
|
Tests client enrollment, secret rejection, mTLS TLS 1.3 socket handshake, warning 4-run rule promotion, immediate error promotion, and Hermes report output.
|
|
|
|
---
|
|
|
|
## Releases & Binary Distribution
|
|
|
|
Releases can be built and published through three complementary channels:
|
|
|
|
### 1. Tag Push Automation (Gitea Actions)
|
|
Pushing a release tag automatically triggers the build workflows:
|
|
```bash
|
|
git tag v2.0.1
|
|
git push origin v2.0.1
|
|
```
|
|
The consolidated workflow **`release.yml`** defines two parallel jobs:
|
|
- **`release-linux`** (`ubuntu-latest`): Compiles standalone native ELF binaries (`Linux_Client.bin`, `Server.bin`) and checksums.
|
|
- **`release-windows`** (`windows-latest`): Compiles Windows executables (`Win_Client.exe`, `Server.exe`), builds Inno Setup installers, and publishes checksums (requires self-hosted Windows Act Runner).
|
|
|
|
### 2. Manual Workflow Dispatch (Gitea UI)
|
|
Workflows can be manually triggered on demand from the Gitea web interface:
|
|
1. Navigate to **Actions** $\rightarrow$ **Release Binaries & Installers** (`release.yml`).
|
|
2. Click **Run workflow**, set the release tag (defaults to `v2.0.1`), and run.
|
|
|
|
### 3. Native Local Windows Build & Direct Release Upload
|
|
For environments without a registered Windows CI runner, Windows executables can be built and published directly to Gitea releases:
|
|
```powershell
|
|
# 1. Build Windows binaries locally
|
|
python compilation/package_dist.py --target windows
|
|
|
|
# 2. Upload assets and release notes directly to the Gitea release
|
|
python compilation/upload_release.py --tag v2.0.1 --token <GITEA_TOKEN> --skip-build
|
|
```
|