Compare commits
6
Commits
95c33c1bf3
...
85bae117b6
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
85bae117b6 | ||
|
|
c8b309d0af | ||
|
|
3b2c3e0fd7 | ||
|
|
b7da54feb2 | ||
|
|
2cd7a990b5 | ||
|
|
2980ad2268 |
@@ -1,6 +1,6 @@
|
|||||||
# LOGAR: Edge-Thin Log Analysis & Temporal Verification System
|
# LOGAR: Edge-Thin Log Analysis & Temporal Verification System
|
||||||
|
|
||||||
**LOGAR** is an enterprise log aggregation, verification, and anomaly detection architecture designed for heterogeneous server fleets (Windows & Linux). It combines lightweight zero-state edge forwarders with a centralized cloud hub that applies mutual TLS 1.3 (**mTLS**) authentication, dynamic PKI licensing and quota management, temporal persistence tracking across 12-hour evaluation windows, an automated 4-run rule to filter transient warnings, and immediate pass-through for critical errors before reporting verified anomalies to **Hermes**.
|
**LOGAR** is an enterprise log aggregation, verification, and anomaly detection architecture designed for heterogeneous server fleets (Windows & Linux). It combines lightweight zero-state edge forwarders with a centralized cloud hub that applies mutual TLS 1.3 (**mTLS**) authentication, dynamic PKI licensing and quota management, temporal persistence tracking across 12-hour evaluation windows, and direct anomaly pass-through enabling **Hermes AI** to perform all deep reasoning, anomaly evaluation, and autonomous decision-making.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -8,7 +8,7 @@
|
|||||||
1. [Core Philosophy](#core-philosophy)
|
1. [Core Philosophy](#core-philosophy)
|
||||||
2. [Architecture & Data Flow](#architecture--data-flow)
|
2. [Architecture & Data Flow](#architecture--data-flow)
|
||||||
3. [mTLS Security, Dynamic PKI & Licensing](#mtls-security-dynamic-pki--licensing)
|
3. [mTLS Security, Dynamic PKI & Licensing](#mtls-security-dynamic-pki--licensing)
|
||||||
4. [Cloud-Side Temporal Persistence & 4-Run Rule](#cloud-side-temporal-persistence--4-run-rule)
|
4. [Cloud-Side Temporal Persistence & Direct Hermes Reasoning](#cloud-side-temporal-persistence--direct-hermes-reasoning)
|
||||||
5. [Agentic Hermes & Client Management API](#agentic-hermes--client-management-api)
|
5. [Agentic Hermes & Client Management API](#agentic-hermes--client-management-api)
|
||||||
6. [Dynamic Machine & Domain Identification](#dynamic-machine--domain-identification)
|
6. [Dynamic Machine & Domain Identification](#dynamic-machine--domain-identification)
|
||||||
7. [Automated Service Installers (Linux & Windows)](#automated-service-installers-linux--windows)
|
7. [Automated Service Installers (Linux & Windows)](#automated-service-installers-linux--windows)
|
||||||
@@ -28,15 +28,14 @@ Site agents running on Windows and Linux act strictly as lightweight forwarders:
|
|||||||
- **Transport Security (mTLS 1.3)**: Logs are streamed directly over mutual TLS 1.3 sockets with hardware-bound / machine-unique client certificates.
|
- **Transport Security (mTLS 1.3)**: Logs are streamed directly over mutual TLS 1.3 sockets with hardware-bound / machine-unique client certificates.
|
||||||
- **Zero Configuration Overhead**: Clients auto-bootstrap certificate enrollment on first run if configured with an enrollment secret.
|
- **Zero Configuration Overhead**: Clients auto-bootstrap certificate enrollment on first run if configured with an enrollment secret.
|
||||||
|
|
||||||
### 2. Cloud-Side Temporal Persistence & Severity Routing
|
### 2. Cloud-Side Temporal Persistence & Direct Hermes Reasoning
|
||||||
The central Python hub handles state and verification:
|
The central Python hub handles state and temporal tracking:
|
||||||
- State tracking is managed centrally in SQLite (`logar_state.db`).
|
- State tracking is managed centrally in SQLite (`logar_state.db`).
|
||||||
- Candidate issues are evaluated over a **12-hour temporal evaluation window**.
|
- Candidate issues are tracked over a **12-hour temporal evaluation window** with consecutive run counts and first/last-seen timestamps.
|
||||||
- **Warning Persistence (4-Run Rule)**: `WARNING` issues must persist across **at least 4 consecutive runs / cycles** within the 12-hour window to be confirmed as genuine anomalies, automatically filtering transient infrastructure blips.
|
- **Direct Hermes Intelligence**: All ingested actionable anomalies (`WARNING`, `ERROR`, `INFO`) are immediately promoted to `VERIFIED` and surfaced directly to Hermes. Hermes performs all multi-factor reasoning, pattern evaluation, and contextual thinking rather than relying on arbitrary hardcoded threshold rules.
|
||||||
- **Immediate Error Pass**: Critical errors (`ERROR`, `CRITICAL`, `FATAL`) bypass the 4-run threshold and are promoted immediately to `VERIFIED` on their first occurrence.
|
|
||||||
|
|
||||||
### 3. Agentic Integration with Hermes
|
### 3. Agentic Integration with Hermes
|
||||||
Instead of engineers manually sifting through logs, **Hermes** ingests pre-filtered anomalies directly from the cloud hub (`GET /api/hermes/report`), treating verified errors and 4-run validated warnings as actionable system artifacts to trigger precise notifications and remediations.
|
Instead of engineers manually sifting through logs, **Hermes** ingests verified anomalies directly from the cloud hub (`GET /api/hermes/report`), treating errors and warnings with full temporal metadata (run counts, timestamps) as actionable system artifacts to trigger precise notifications and remediations.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -62,7 +61,7 @@ graph TB
|
|||||||
subgraph Hub [LOGAR Server Hub]
|
subgraph Hub [LOGAR Server Hub]
|
||||||
INGEST[Length-Prefixed Frame Ingestion]
|
INGEST[Length-Prefixed Frame Ingestion]
|
||||||
DB[(SQLite Persistence<br/>active_issues, clients, license_config)]
|
DB[(SQLite Persistence<br/>active_issues, clients, license_config)]
|
||||||
RULE{12h Window &<br/>4-Run Rule}
|
RULE{12h Window &<br/>Temporal Ingestion}
|
||||||
end
|
end
|
||||||
|
|
||||||
subgraph Downstream [Hermes Agent & Monitoring]
|
subgraph Downstream [Hermes Agent & Monitoring]
|
||||||
@@ -139,21 +138,16 @@ graph TB
|
|||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Cloud-Side Temporal Persistence & 4-Run Rule
|
## Cloud-Side Temporal Persistence & Direct Hermes Reasoning
|
||||||
|
|
||||||
Incoming candidate logs are tracked in SQLite table `active_issues`:
|
Incoming candidate logs are tracked in SQLite table `active_issues`:
|
||||||
- **Issue Fingerprint**: Formatted as `{site_name}:{server}:{signature}`.
|
- **Issue Fingerprint**: Formatted as `{site_name}:{server}:{signature}`.
|
||||||
- **12-Hour Evaluation Window**:
|
- **12-Hour Evaluation Window**:
|
||||||
- The hub compares `(now - last_seen)`.
|
- The hub compares `(now - last_seen)`.
|
||||||
- If more than 12 hours have elapsed since the issue was last recorded, the previous window expires and the cycle resets to `run_count = 1`.
|
- If more than 12 hours have elapsed since the issue was last recorded, the previous window expires and the cycle resets to `run_count = 1`.
|
||||||
- **4-Run Rule for Warnings**:
|
- **Direct Verification & Temporal Telemetry**:
|
||||||
- The 4-run persistence threshold applies to `WARNING` (and `INFO`) events to eliminate transient operational noise.
|
- Ingested anomalies are immediately promoted to `VERIFIED` and made available to Hermes with complete temporal metadata (first seen, last seen, consecutive run counts).
|
||||||
- Each distinct run batch increments `run_count`.
|
- Hermes performs all contextual analysis, pattern recognition, and alerting logic.
|
||||||
- Warnings with `run_count < 4` are marked as `TRANSIENT` and excluded from Hermes reports.
|
|
||||||
- When `run_count >= 4` within the active 12-hour window, the warning transitions to `VERIFIED`.
|
|
||||||
- **Immediate Verification for Errors**:
|
|
||||||
- High-severity events (`ERROR`, `CRITICAL`, `FATAL`) **always pass immediately**.
|
|
||||||
- On their very first ingestion (`run_count = 1`), errors are promoted directly to `VERIFIED` and surfaced to Hermes without waiting for 4 runs.
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -202,7 +196,7 @@ Returns seat quota status and registered client telemetry:
|
|||||||
```
|
```
|
||||||
|
|
||||||
### `GET /api/hermes/report`
|
### `GET /api/hermes/report`
|
||||||
Returns all **verified anomalies** (immediate critical errors and warnings verified after 4 consecutive runs within the 12-hour window):
|
Returns all **verified anomalies** within the active 12-hour evaluation window with full temporal metadata for direct Hermes reasoning:
|
||||||
```json
|
```json
|
||||||
[
|
[
|
||||||
{
|
{
|
||||||
@@ -211,7 +205,7 @@ Returns all **verified anomalies** (immediate critical errors and warnings verif
|
|||||||
"server": "web-worker-01.corp.internal",
|
"server": "web-worker-01.corp.internal",
|
||||||
"signature": "PostgresPoolExhausted",
|
"signature": "PostgresPoolExhausted",
|
||||||
"severity": "WARNING",
|
"severity": "WARNING",
|
||||||
"message": "Connection pool saturated (>95%) across 4 runs",
|
"message": "Connection pool saturated (>95%)",
|
||||||
"os_type": "linux",
|
"os_type": "linux",
|
||||||
"first_seen": "2026-09-04T07:00:00+00:00",
|
"first_seen": "2026-09-04T07:00:00+00:00",
|
||||||
"last_seen": "2026-09-04T19:00:00+00:00",
|
"last_seen": "2026-09-04T19:00:00+00:00",
|
||||||
@@ -224,7 +218,7 @@ Returns all **verified anomalies** (immediate critical errors and warnings verif
|
|||||||
```
|
```
|
||||||
|
|
||||||
### `GET /api/hermes/all`
|
### `GET /api/hermes/all`
|
||||||
Diagnostic endpoint listing all candidate issues (`TRANSIENT` and `VERIFIED`).
|
Diagnostic endpoint listing all tracked anomalies in SQLite.
|
||||||
|
|
||||||
### `GET /health`
|
### `GET /health`
|
||||||
Returns hub health, encryption fingerprint, listener ports, and mTLS status.
|
Returns hub health, encryption fingerprint, listener ports, and mTLS status.
|
||||||
@@ -296,7 +290,7 @@ LOGAR/
|
|||||||
│ └── Linux_Client.py # Linux edge forwarder with auto-enrollment
|
│ └── Linux_Client.py # Linux edge forwarder with auto-enrollment
|
||||||
├── tests/ # Automated test suites
|
├── tests/ # Automated test suites
|
||||||
│ ├── test_linux_client.py # Linux client unit tests & mTLS certificate validation
|
│ ├── test_linux_client.py # Linux client unit tests & mTLS certificate validation
|
||||||
│ ├── test_pipeline.py # End-to-end mTLS integration & 4-run verification test
|
│ ├── test_pipeline.py # End-to-end mTLS integration & direct anomaly verification test
|
||||||
│ ├── test_server.py # Server unit tests, PKI generation, and seat quota tests
|
│ ├── test_server.py # Server unit tests, PKI generation, and seat quota tests
|
||||||
│ └── test_win_client.py # Windows client unit tests & mTLS certificate validation
|
│ └── test_win_client.py # Windows client unit tests & mTLS certificate validation
|
||||||
├── .gitignore # Ignores venv, caches, DBs, and private keys
|
├── .gitignore # Ignores venv, caches, DBs, and private keys
|
||||||
@@ -362,7 +356,7 @@ Runs all **25 unit tests**, covering:
|
|||||||
- Enrollment secret authentication, seat limits, and certificate revocation.
|
- Enrollment secret authentication, seat limits, and certificate revocation.
|
||||||
- Proactive certificate validity checks, Root CA auto-renewal, and in-flight server SSLContext reload.
|
- Proactive certificate validity checks, Root CA auto-renewal, and in-flight server SSLContext reload.
|
||||||
- Windows & Linux event log collection, deduplication, and mTLS certificate verification.
|
- Windows & Linux event log collection, deduplication, and mTLS certificate verification.
|
||||||
- 12-hour evaluation window and 4-run rule progression.
|
- 12-hour evaluation window and direct Hermes anomaly verification.
|
||||||
|
|
||||||
### 2. End-to-End Pipeline Integration Test
|
### 2. End-to-End Pipeline Integration Test
|
||||||
```bash
|
```bash
|
||||||
@@ -375,7 +369,7 @@ python src/Server.py &
|
|||||||
# 3. Run integration test
|
# 3. Run integration test
|
||||||
python tests/test_pipeline.py
|
python tests/test_pipeline.py
|
||||||
```
|
```
|
||||||
Tests client enrollment, secret rejection, mTLS TLS 1.3 socket handshake, warning 4-run rule promotion, immediate error promotion, and Hermes report output.
|
Tests client enrollment, secret rejection, mTLS TLS 1.3 socket handshake, direct warning promotion, immediate error promotion, and Hermes report output.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|||||||
@@ -14,7 +14,7 @@ Standalone compiled binary and automated systemd service distribution for Linux
|
|||||||
- **Master License Authority (`logar.eibl.tech`) & Seat-Bearing CA**: Integrates with the upstream authority. Subordinate CA certificates embed exact seat quotas via custom enterprise X.509 OID `1.3.6.1.4.1.59999.1.1`.
|
- **Master License Authority (`logar.eibl.tech`) & Seat-Bearing CA**: Integrates with the upstream authority. Subordinate CA certificates embed exact seat quotas via custom enterprise X.509 OID `1.3.6.1.4.1.59999.1.1`.
|
||||||
- **Zero-Touch mTLS Renewals**: Once initial signed CA cert is imported, the server hub automatically executes in-flight mTLS renewals with `https://logar.eibl.tech/api/license/renew` when nearing expiration (< 30 days).
|
- **Zero-Touch mTLS Renewals**: Once initial signed CA cert is imported, the server hub automatically executes in-flight mTLS renewals with `https://logar.eibl.tech/api/license/renew` when nearing expiration (< 30 days).
|
||||||
- **Fleet Management & WebUI Dashboard (Port 8443)**: Modern browser WebUI accessible at `http://localhost:8443/webui` to monitor seat quotas, export `.lic` request files, copy CA CSRs, activate signed CA certs, and revoke/reactivate forwarder seats with immediate socket enforcement.
|
- **Fleet Management & WebUI Dashboard (Port 8443)**: Modern browser WebUI accessible at `http://localhost:8443/webui` to monitor seat quotas, export `.lic` request files, copy CA CSRs, activate signed CA certs, and revoke/reactivate forwarder seats with immediate socket enforcement.
|
||||||
- **Warning Persistence & Immediate Error Routing**: High-severity `ERROR`, `CRITICAL`, and `FATAL` events are promoted to `VERIFIED` immediately on their first occurrence. Operational `WARNING` and `INFO` events require persistence across at least 4 distinct client transmission cycles within a rolling 12-hour evaluation window.
|
- **Direct Hermes Intelligence & Temporal Telemetry**: All ingested actionable anomalies (`WARNING`, `ERROR`, `INFO`) are immediately promoted to `VERIFIED` and surfaced directly to Hermes with rich temporal metadata (run counts, timestamps) across a rolling 12-hour evaluation window, allowing Hermes AI to perform all contextual reasoning and decisions.
|
||||||
- **Embedded Hermes Reporting API (Port 8443)**: Integrated REST API exposing `/api/hermes/report`, `/api/clients`, and `/api/client/enroll`.
|
- **Embedded Hermes Reporting API (Port 8443)**: Integrated REST API exposing `/api/hermes/report`, `/api/clients`, and `/api/client/enroll`.
|
||||||
- **State Database**: Stores issue lifecycle records, client telemetry, and licensing quotas in a local SQLite database (`logar_state.db`).
|
- **State Database**: Stores issue lifecycle records, client telemetry, and licensing quotas in a local SQLite database (`logar_state.db`).
|
||||||
|
|
||||||
@@ -58,7 +58,7 @@ Output:
|
|||||||
Transport Security: mTLS (TLS 1.3)
|
Transport Security: mTLS (TLS 1.3)
|
||||||
License Quota: 10 Active Seats
|
License Quota: 10 Active Seats
|
||||||
Server Encryption Fingerprint: 375388960531264EA0648EC0D2C4E4ABC6F22AC2
|
Server Encryption Fingerprint: 375388960531264EA0648EC0D2C4E4ABC6F22AC2
|
||||||
Evaluation Window: 12 hours | 4-Run Rule: Warnings | Immediate Pass: Errors
|
Evaluation Window: 12 hours | Direct Hermes Reasoning Enabled
|
||||||
============================================================
|
============================================================
|
||||||
[*] LOGAR mTLS TLSv1.3 Socket Server listening on 0.0.0.0:9443
|
[*] LOGAR mTLS TLSv1.3 Socket Server listening on 0.0.0.0:9443
|
||||||
[*] Hermes Reporting API available at http://0.0.0.0:8443/api/hermes/report
|
[*] Hermes Reporting API available at http://0.0.0.0:8443/api/hermes/report
|
||||||
@@ -81,7 +81,7 @@ The generated `server_config.json` contains:
|
|||||||
"tls_enabled": true,
|
"tls_enabled": true,
|
||||||
"db_path": "logar_state.db",
|
"db_path": "logar_state.db",
|
||||||
"evaluation_window_hours": 12,
|
"evaluation_window_hours": 12,
|
||||||
"min_persistence_runs": 4,
|
"min_persistence_runs": 1,
|
||||||
"server_fingerprint": "375388960531264EA0648EC0D2C4E4ABC6F22AC2",
|
"server_fingerprint": "375388960531264EA0648EC0D2C4E4ABC6F22AC2",
|
||||||
"public_key": "-----BEGIN PGP PUBLIC KEY BLOCK-----\n...",
|
"public_key": "-----BEGIN PGP PUBLIC KEY BLOCK-----\n...",
|
||||||
"private_key": "-----BEGIN PGP PRIVATE KEY BLOCK-----\n..."
|
"private_key": "-----BEGIN PGP PRIVATE KEY BLOCK-----\n..."
|
||||||
|
|||||||
@@ -14,7 +14,7 @@ Standalone compiled executable and installer distribution for Windows Server env
|
|||||||
- **Master License Authority (`logar.eibl.tech`) & Seat-Bearing CA**: Integrates with the upstream authority. Subordinate CA certificates embed exact seat quotas via custom enterprise X.509 OID `1.3.6.1.4.1.59999.1.1`.
|
- **Master License Authority (`logar.eibl.tech`) & Seat-Bearing CA**: Integrates with the upstream authority. Subordinate CA certificates embed exact seat quotas via custom enterprise X.509 OID `1.3.6.1.4.1.59999.1.1`.
|
||||||
- **Zero-Touch mTLS Renewals**: Once initial signed CA cert is imported, the server hub automatically executes in-flight mTLS renewals with `https://logar.eibl.tech/api/license/renew` when nearing expiration (< 30 days).
|
- **Zero-Touch mTLS Renewals**: Once initial signed CA cert is imported, the server hub automatically executes in-flight mTLS renewals with `https://logar.eibl.tech/api/license/renew` when nearing expiration (< 30 days).
|
||||||
- **Fleet Management & WebUI Dashboard (Port 8443)**: Modern browser WebUI accessible at `http://localhost:8443/webui` to monitor seat quotas, export `.lic` request files, copy CA CSRs, activate signed CA certs, and revoke/reactivate forwarder seats with immediate socket enforcement.
|
- **Fleet Management & WebUI Dashboard (Port 8443)**: Modern browser WebUI accessible at `http://localhost:8443/webui` to monitor seat quotas, export `.lic` request files, copy CA CSRs, activate signed CA certs, and revoke/reactivate forwarder seats with immediate socket enforcement.
|
||||||
- **Warning Persistence & Immediate Error Routing**: High-severity `ERROR`, `CRITICAL`, and `FATAL` events are promoted to `VERIFIED` immediately on their first occurrence. Operational `WARNING` and `INFO` events require persistence across at least 4 distinct transmission cycles within a rolling 12-hour evaluation window.
|
- **Direct Hermes Intelligence & Temporal Telemetry**: All ingested actionable anomalies (`WARNING`, `ERROR`, `INFO`) are immediately promoted to `VERIFIED` and surfaced directly to Hermes with rich temporal metadata (run counts, timestamps) across a rolling 12-hour evaluation window, allowing Hermes AI to perform all contextual reasoning and decisions.
|
||||||
- **Embedded Hermes Reporting API (Port 8443)**: Integrated REST API exposing `/api/hermes/report`, `/api/clients`, and `/api/client/enroll`.
|
- **Embedded Hermes Reporting API (Port 8443)**: Integrated REST API exposing `/api/hermes/report`, `/api/clients`, and `/api/client/enroll`.
|
||||||
- **State Database**: Stores issue lifecycle records, client telemetry, and licensing quotas in a local SQLite database (`logar_state.db`).
|
- **State Database**: Stores issue lifecycle records, client telemetry, and licensing quotas in a local SQLite database (`logar_state.db`).
|
||||||
|
|
||||||
@@ -58,7 +58,7 @@ Output:
|
|||||||
Transport Security: mTLS (TLS 1.3)
|
Transport Security: mTLS (TLS 1.3)
|
||||||
License Quota: 10 Active Seats
|
License Quota: 10 Active Seats
|
||||||
Server Encryption Fingerprint: 375388960531264EA0648EC0D2C4E4ABC6F22AC2
|
Server Encryption Fingerprint: 375388960531264EA0648EC0D2C4E4ABC6F22AC2
|
||||||
Evaluation Window: 12 hours | 4-Run Rule: Warnings | Immediate Pass: Errors
|
Evaluation Window: 12 hours | Direct Hermes Reasoning Enabled
|
||||||
============================================================
|
============================================================
|
||||||
[*] LOGAR mTLS TLSv1.3 Socket Server listening on 0.0.0.0:9443
|
[*] LOGAR mTLS TLSv1.3 Socket Server listening on 0.0.0.0:9443
|
||||||
[*] Hermes Reporting API available at http://0.0.0.0:8443/api/hermes/report
|
[*] Hermes Reporting API available at http://0.0.0.0:8443/api/hermes/report
|
||||||
@@ -81,7 +81,7 @@ The generated `server_config.json` contains:
|
|||||||
"tls_enabled": true,
|
"tls_enabled": true,
|
||||||
"db_path": "logar_state.db",
|
"db_path": "logar_state.db",
|
||||||
"evaluation_window_hours": 12,
|
"evaluation_window_hours": 12,
|
||||||
"min_persistence_runs": 4,
|
"min_persistence_runs": 1,
|
||||||
"server_fingerprint": "375388960531264EA0648EC0D2C4E4ABC6F22AC2",
|
"server_fingerprint": "375388960531264EA0648EC0D2C4E4ABC6F22AC2",
|
||||||
"public_key": "-----BEGIN PGP PUBLIC KEY BLOCK-----\n...",
|
"public_key": "-----BEGIN PGP PUBLIC KEY BLOCK-----\n...",
|
||||||
"private_key": "-----BEGIN PGP PRIVATE KEY BLOCK-----\n..."
|
"private_key": "-----BEGIN PGP PRIVATE KEY BLOCK-----\n..."
|
||||||
|
|||||||
+11
-19
@@ -37,7 +37,7 @@ except ImportError:
|
|||||||
CONFIG_FILE_NAME = "server_config.json"
|
CONFIG_FILE_NAME = "server_config.json"
|
||||||
DEFAULT_DB_FILE = "logar_state.db"
|
DEFAULT_DB_FILE = "logar_state.db"
|
||||||
EVALUATION_WINDOW_HOURS = 12
|
EVALUATION_WINDOW_HOURS = 12
|
||||||
RUN_THRESHOLD = 4
|
RUN_THRESHOLD = 1
|
||||||
|
|
||||||
app = FastAPI(title="LOGAR Cloud Ingestion & Hermes Hub", version="2.0.1")
|
app = FastAPI(title="LOGAR Cloud Ingestion & Hermes Hub", version="2.0.1")
|
||||||
|
|
||||||
@@ -260,8 +260,8 @@ def export_license_request_lic(config: Dict[str, Any]) -> Tuple[Dict[str, Any],
|
|||||||
|
|
||||||
def process_ingested_logs(payload: Dict[str, Any], db_path: str, window_hours: int, min_runs: int) -> Dict[str, Any]:
|
def process_ingested_logs(payload: Dict[str, Any], db_path: str, window_hours: int, min_runs: int) -> Dict[str, Any]:
|
||||||
"""
|
"""
|
||||||
Evaluates candidate issues against the 12-hour evaluation window and 4-run rule.
|
Evaluates ingested log anomalies and updates temporal run tracking.
|
||||||
Zero-state clients send raw candidate entries; this engine handles temporal state.
|
All anomalies are promoted directly to VERIFIED for Hermes AI reasoning and decision-making.
|
||||||
"""
|
"""
|
||||||
client_server = payload.get("server", "unknown-host")
|
client_server = payload.get("server", "unknown-host")
|
||||||
site_name = payload.get("site_name") or (client_server.split(".", 1)[1] if "." in client_server else "default")
|
site_name = payload.get("site_name") or (client_server.split(".", 1)[1] if "." in client_server else "default")
|
||||||
@@ -288,9 +288,6 @@ def process_ingested_logs(payload: Dict[str, Any], db_path: str, window_hours: i
|
|||||||
if severity in ["DEBUG", "TRACE"]:
|
if severity in ["DEBUG", "TRACE"]:
|
||||||
continue
|
continue
|
||||||
|
|
||||||
# Errors are always passed immediately; the 4-run rule only concerns warnings
|
|
||||||
is_error = severity in ["ERROR", "CRITICAL", "FATAL"]
|
|
||||||
|
|
||||||
signature = log.get("signature", "unknown")
|
signature = log.get("signature", "unknown")
|
||||||
server = log.get("server", client_server)
|
server = log.get("server", client_server)
|
||||||
message = log.get("message", "")
|
message = log.get("message", "")
|
||||||
@@ -312,21 +309,17 @@ def process_ingested_logs(payload: Dict[str, Any], db_path: str, window_hours: i
|
|||||||
|
|
||||||
# 12-hour evaluation window expiry check
|
# 12-hour evaluation window expiry check
|
||||||
if (now - last_seen_dt) > timedelta(hours=window_hours):
|
if (now - last_seen_dt) > timedelta(hours=window_hours):
|
||||||
# Window elapsed: reset to new cycle
|
|
||||||
new_runs = 1
|
new_runs = 1
|
||||||
new_first_seen = now_iso
|
new_first_seen = now_iso
|
||||||
new_status = "VERIFIED" if is_error else "TRANSIENT"
|
|
||||||
else:
|
else:
|
||||||
# Same run guard: only increment count once per distinct run batch
|
|
||||||
if last_run_id != run_id:
|
if last_run_id != run_id:
|
||||||
new_runs = run_count + 1
|
new_runs = run_count + 1
|
||||||
else:
|
else:
|
||||||
new_runs = run_count
|
new_runs = run_count
|
||||||
new_first_seen = first_seen_str
|
new_first_seen = first_seen_str
|
||||||
# 4-run rule applies to warnings; errors are always passed immediately as VERIFIED
|
|
||||||
new_status = "VERIFIED" if (is_error or new_runs >= min_runs) else "TRANSIENT"
|
|
||||||
|
|
||||||
if new_status == "VERIFIED" and current_status != "VERIFIED":
|
new_status = "VERIFIED"
|
||||||
|
if current_status != "VERIFIED":
|
||||||
promoted_to_verified += 1
|
promoted_to_verified += 1
|
||||||
|
|
||||||
cursor.execute("""
|
cursor.execute("""
|
||||||
@@ -335,8 +328,7 @@ def process_ingested_logs(payload: Dict[str, Any], db_path: str, window_hours: i
|
|||||||
WHERE fingerprint = ?
|
WHERE fingerprint = ?
|
||||||
""", (new_runs, now_iso, new_first_seen, new_status, run_id, message, severity, fp))
|
""", (new_runs, now_iso, new_first_seen, new_status, run_id, message, severity, fp))
|
||||||
else:
|
else:
|
||||||
initial_status = "VERIFIED" if (is_error or 1 >= min_runs) else "TRANSIENT"
|
initial_status = "VERIFIED"
|
||||||
if initial_status == "VERIFIED":
|
|
||||||
promoted_to_verified += 1
|
promoted_to_verified += 1
|
||||||
cursor.execute("""
|
cursor.execute("""
|
||||||
INSERT INTO active_issues
|
INSERT INTO active_issues
|
||||||
@@ -548,7 +540,7 @@ async def handle_socket_client(reader: asyncio.StreamReader, writer: asyncio.Str
|
|||||||
if client_id and "server" not in log_payload:
|
if client_id and "server" not in log_payload:
|
||||||
log_payload["server"] = client_id
|
log_payload["server"] = client_id
|
||||||
|
|
||||||
# Ingest and apply 12h window / 4-run rule
|
# Ingest and evaluate anomalies with temporal run tracking
|
||||||
res = process_ingested_logs(
|
res = process_ingested_logs(
|
||||||
log_payload,
|
log_payload,
|
||||||
db_path=SERVER_STATE["config"]["db_path"],
|
db_path=SERVER_STATE["config"]["db_path"],
|
||||||
@@ -694,8 +686,8 @@ def list_clients():
|
|||||||
def get_verified_anomalies_for_hermes():
|
def get_verified_anomalies_for_hermes():
|
||||||
"""
|
"""
|
||||||
Ingestion endpoint for Hermes agentic workflows.
|
Ingestion endpoint for Hermes agentic workflows.
|
||||||
Returns only verified anomalies that have satisfied the 4-run persistence rule
|
Returns verified anomalies within the active 12-hour evaluation window
|
||||||
within the active 12-hour evaluation window. Transient blips (< 4 runs) are excluded.
|
for Hermes AI direct reasoning and autonomous decision-making.
|
||||||
"""
|
"""
|
||||||
db_path = SERVER_STATE["config"]["db_path"]
|
db_path = SERVER_STATE["config"]["db_path"]
|
||||||
conn = sqlite3.connect(db_path)
|
conn = sqlite3.connect(db_path)
|
||||||
@@ -1284,7 +1276,7 @@ def serve_webui():
|
|||||||
<span class="logo-icon">🛰️</span>
|
<span class="logo-icon">🛰️</span>
|
||||||
<div>
|
<div>
|
||||||
<h1>LOGAR Fleet Hub & License Management</h1>
|
<h1>LOGAR Fleet Hub & License Management</h1>
|
||||||
<div class="subtitle">Cloud Hub: <strong>__SERVER_NAME__</strong> | Evaluator: 12h Rolling Window & 4-Run Rule</div>
|
<div class="subtitle">Cloud Hub: <strong>__SERVER_NAME__</strong> | Evaluator: 12h Rolling Window & Direct Hermes Intelligence</div>
|
||||||
</div>
|
</div>
|
||||||
</div>
|
</div>
|
||||||
<div class="system-tags">
|
<div class="system-tags">
|
||||||
@@ -1380,7 +1372,7 @@ def serve_webui():
|
|||||||
|
|
||||||
<div class="card">
|
<div class="card">
|
||||||
<div class="card-header">
|
<div class="card-header">
|
||||||
<span class="card-title" style="font-size: 1rem;">Hermes Verified Anomalies (4-Run Rule & Errors)</span>
|
<span class="card-title" style="font-size: 1rem;">Hermes Verified Anomalies</span>
|
||||||
<span class="badge badge-active">Active Window: 12h</span>
|
<span class="badge badge-active">Active Window: 12h</span>
|
||||||
</div>
|
</div>
|
||||||
<table>
|
<table>
|
||||||
|
|||||||
@@ -146,7 +146,7 @@ def run_tests():
|
|||||||
resp_bytes.extend(chunk)
|
resp_bytes.extend(chunk)
|
||||||
return json.loads(resp_bytes.decode("utf-8"))
|
return json.loads(resp_bytes.decode("utf-8"))
|
||||||
|
|
||||||
print("\n=== [3] Testing Temporal Persistence & 4-Run Rule for Warnings over mTLS ===")
|
print("\n=== [3] Testing Temporal Persistence & Direct Anomaly Verification over mTLS ===")
|
||||||
test_signature = "TestServiceDegraded"
|
test_signature = "TestServiceDegraded"
|
||||||
candidate_log = [{
|
candidate_log = [{
|
||||||
"server": client_id,
|
"server": client_id,
|
||||||
@@ -161,10 +161,10 @@ def run_tests():
|
|||||||
assert resp.get("status") == "success", f"Run {run_num} failed: {resp}"
|
assert resp.get("status") == "success", f"Run {run_num} failed: {resp}"
|
||||||
promoted = resp.get("promoted_verified", 0)
|
promoted = resp.get("promoted_verified", 0)
|
||||||
print(f"[Run {run_num}/4] Ingested successfully via mTLS. Promoted to verified: {promoted}")
|
print(f"[Run {run_num}/4] Ingested successfully via mTLS. Promoted to verified: {promoted}")
|
||||||
if run_num < 4:
|
if run_num == 1:
|
||||||
assert promoted == 0, f"Expected 0 promoted on run {run_num} for warning, got {promoted}"
|
assert promoted == 1, f"Expected 1 promoted on run 1 for warning, got {promoted}"
|
||||||
else:
|
else:
|
||||||
assert promoted == 1, f"Expected 1 promoted on run 4 for warning, got {promoted}"
|
assert promoted == 0, f"Expected 0 newly promoted on run {run_num} (already verified), got {promoted}"
|
||||||
|
|
||||||
# Inspect SQLite database directly
|
# Inspect SQLite database directly
|
||||||
conn = sqlite3.connect(server_conf.get("db_path", "logar_state.db"))
|
conn = sqlite3.connect(server_conf.get("db_path", "logar_state.db"))
|
||||||
@@ -178,7 +178,7 @@ def run_tests():
|
|||||||
print(f"[DB Verification] Issue '{test_signature}' -> run_count: {run_count}, status: {status}")
|
print(f"[DB Verification] Issue '{test_signature}' -> run_count: {run_count}, status: {status}")
|
||||||
assert run_count >= 4, f"Expected run_count >= 4, got {run_count}"
|
assert run_count >= 4, f"Expected run_count >= 4, got {run_count}"
|
||||||
assert status == "VERIFIED", f"Expected status 'VERIFIED', got {status}"
|
assert status == "VERIFIED", f"Expected status 'VERIFIED', got {status}"
|
||||||
print("[OK] 4-Run Rule verified: Warning promoted to VERIFIED anomaly on 4th run over mTLS!")
|
print("[OK] Direct verification confirmed: Warning immediately verified for Hermes with run count tracking over mTLS!")
|
||||||
|
|
||||||
print("\n=== [4] Testing Immediate Pass for Errors over mTLS ===")
|
print("\n=== [4] Testing Immediate Pass for Errors over mTLS ===")
|
||||||
error_signature = "TestServiceCrashImmediate"
|
error_signature = "TestServiceCrashImmediate"
|
||||||
|
|||||||
+17
-16
@@ -75,7 +75,7 @@ class TestServerComponent(unittest.TestCase):
|
|||||||
if os.path.exists(client_out):
|
if os.path.exists(client_out):
|
||||||
os.remove(client_out)
|
os.remove(client_out)
|
||||||
|
|
||||||
def test_4_run_rule_and_12h_window(self):
|
def test_direct_hermes_anomaly_and_12h_window(self):
|
||||||
Server.init_db(self.test_db)
|
Server.init_db(self.test_db)
|
||||||
log_entry = {
|
log_entry = {
|
||||||
"server": "app-worker-01.corp.local",
|
"server": "app-worker-01.corp.local",
|
||||||
@@ -89,23 +89,24 @@ class TestServerComponent(unittest.TestCase):
|
|||||||
"logs": [log_entry]
|
"logs": [log_entry]
|
||||||
}
|
}
|
||||||
|
|
||||||
# Runs 1 to 3: WARNING should remain TRANSIENT
|
# Run 1: WARNING is immediately VERIFIED for Hermes AI reasoning
|
||||||
for run_idx in range(1, 4):
|
res1 = Server.process_ingested_logs(payload, self.test_db, window_hours=12, min_runs=1)
|
||||||
res = Server.process_ingested_logs(payload, self.test_db, window_hours=12, min_runs=4)
|
self.assertEqual(res1["status"], "success")
|
||||||
self.assertEqual(res["status"], "success")
|
self.assertEqual(res1["promoted_verified"], 1)
|
||||||
self.assertEqual(res["promoted_verified"], 0)
|
|
||||||
|
|
||||||
conn = sqlite3.connect(self.test_db)
|
conn = sqlite3.connect(self.test_db)
|
||||||
c = conn.cursor()
|
c = conn.cursor()
|
||||||
c.execute("SELECT run_count, status FROM active_issues WHERE signature = ?", ("PostgresConnWarning",))
|
c.execute("SELECT run_count, status FROM active_issues WHERE signature = ?", ("PostgresConnWarning",))
|
||||||
row = c.fetchone()
|
row = c.fetchone()
|
||||||
conn.close()
|
conn.close()
|
||||||
self.assertEqual(row[0], 3)
|
self.assertEqual(row[0], 1)
|
||||||
self.assertEqual(row[1], "TRANSIENT")
|
self.assertEqual(row[1], "VERIFIED")
|
||||||
|
|
||||||
# Run 4: promotes WARNING to VERIFIED!
|
# Runs 2 to 4: Increments consecutive run count while remaining VERIFIED
|
||||||
res4 = Server.process_ingested_logs(payload, self.test_db, window_hours=12, min_runs=4)
|
for run_idx in range(2, 5):
|
||||||
self.assertEqual(res4["promoted_verified"], 1)
|
res = Server.process_ingested_logs(payload, self.test_db, window_hours=12, min_runs=1)
|
||||||
|
self.assertEqual(res["status"], "success")
|
||||||
|
self.assertEqual(res["promoted_verified"], 0)
|
||||||
|
|
||||||
conn = sqlite3.connect(self.test_db)
|
conn = sqlite3.connect(self.test_db)
|
||||||
c = conn.cursor()
|
c = conn.cursor()
|
||||||
@@ -130,7 +131,7 @@ class TestServerComponent(unittest.TestCase):
|
|||||||
}
|
}
|
||||||
|
|
||||||
# Run 1: ERROR must immediately promote to VERIFIED
|
# Run 1: ERROR must immediately promote to VERIFIED
|
||||||
res = Server.process_ingested_logs(payload, self.test_db, window_hours=12, min_runs=4)
|
res = Server.process_ingested_logs(payload, self.test_db, window_hours=12, min_runs=1)
|
||||||
self.assertEqual(res["status"], "success")
|
self.assertEqual(res["status"], "success")
|
||||||
self.assertEqual(res["promoted_verified"], 1)
|
self.assertEqual(res["promoted_verified"], 1)
|
||||||
|
|
||||||
@@ -156,7 +157,7 @@ class TestServerComponent(unittest.TestCase):
|
|||||||
{"server": "app-worker-01", "signature": "SigTrace", "severity": "TRACE", "message": "Trace msg", "os_type": "linux"}
|
{"server": "app-worker-01", "signature": "SigTrace", "severity": "TRACE", "message": "Trace msg", "os_type": "linux"}
|
||||||
]
|
]
|
||||||
}
|
}
|
||||||
res = Server.process_ingested_logs(payload, self.test_db, window_hours=12, min_runs=4)
|
res = Server.process_ingested_logs(payload, self.test_db, window_hours=12, min_runs=1)
|
||||||
self.assertEqual(res["status"], "success")
|
self.assertEqual(res["status"], "success")
|
||||||
|
|
||||||
conn = sqlite3.connect(self.test_db)
|
conn = sqlite3.connect(self.test_db)
|
||||||
@@ -171,10 +172,10 @@ class TestServerComponent(unittest.TestCase):
|
|||||||
self.assertNotIn("SigDebug", rows)
|
self.assertNotIn("SigDebug", rows)
|
||||||
self.assertNotIn("SigTrace", rows)
|
self.assertNotIn("SigTrace", rows)
|
||||||
|
|
||||||
# SigErr is immediately VERIFIED; SigWarn and SigInfo are TRANSIENT on run 1
|
# All actionable issues (ERROR, WARNING, INFO) are promoted to VERIFIED immediately for Hermes reasoning
|
||||||
self.assertEqual(rows["SigErr"], "VERIFIED")
|
self.assertEqual(rows["SigErr"], "VERIFIED")
|
||||||
self.assertEqual(rows["SigWarn"], "TRANSIENT")
|
self.assertEqual(rows["SigWarn"], "VERIFIED")
|
||||||
self.assertEqual(rows["SigInfo"], "TRANSIENT")
|
self.assertEqual(rows["SigInfo"], "VERIFIED")
|
||||||
|
|
||||||
def test_license_schema_and_pki_generation(self):
|
def test_license_schema_and_pki_generation(self):
|
||||||
secret = "test-secret-12345"
|
secret = "test-secret-12345"
|
||||||
|
|||||||
Reference in New Issue
Block a user