Update Windows server deployment guide for warning persistence and immediate error pass

README.md

@@ -1,6 +1,6 @@
 # LOGAR: Edge-Thin Log Analysis & Temporal Verification System
 
-**LOGAR** is an enterprise log aggregation, verification, and anomaly detection architecture designed for heterogeneous server fleets (Windows & Linux). It combines lightweight zero-state edge forwarders with a centralized cloud hub that applies OpenPGP encryption, authenticated TCP streaming, temporal persistence tracking across 12-hour evaluation windows, and an automated 4-run rule to filter out transient infrastructure blips before reporting verified anomalies to **Hermes**.
+**LOGAR** is an enterprise log aggregation, verification, and anomaly detection architecture designed for heterogeneous server fleets (Windows & Linux). It combines lightweight zero-state edge forwarders with a centralized cloud hub that applies OpenPGP encryption, authenticated TCP streaming, temporal persistence tracking across 12-hour evaluation windows, an automated 4-run rule to filter transient warnings, and immediate pass-through for critical errors before reporting verified anomalies to **Hermes**.
 
 ---
 
@@ -27,14 +27,15 @@ Site agents running on Windows and Linux act strictly as lightweight forwarders:
 - **End-to-End Encryption**: Logs are encrypted using the server's OpenPGP public key before leaving the edge node.
 - **Secure TCP Sockets**: Ingestion occurs over low-overhead authenticated TCP sockets rather than bulky HTTP/HTTPS endpoints.
 
-### 2. Cloud-Side Temporal Persistence
+### 2. Cloud-Side Temporal Persistence & Severity Routing
 The central Python/TCP hub handles the heavy lifting:
 - State tracking is managed centrally in SQLite (`logar_state.db`).
 - Candidate issues are evaluated over a **12-hour temporal evaluation window**.
-- An issue must persist across **at least 4 consecutive runs / cycles** to be confirmed as a genuine system anomaly. Transient blips and sporadic spikes are filtered out automatically.
+- **Warning Persistence (4-Run Rule)**: `WARNING` level issues must persist across **at least 4 consecutive runs / cycles** within the 12-hour window to be confirmed as genuine anomalies, automatically filtering out transient blips.
+- **Immediate Error Pass**: Critical errors (`ERROR`, `CRITICAL`, `FATAL`) bypass the 4-run threshold and are promoted immediately to `VERIFIED` on their first occurrence.
 
 ### 3. Agentic Integration with Hermes
-Instead of human engineers manually diving through noisy logs, **Hermes** ingests pre-filtered, 4-run validated anomalies directly from the cloud hub (`GET /api/hermes/report`), treating them as verified system artifacts to trigger precise team notifications.
+Instead of human engineers manually diving through noisy logs, **Hermes** ingests pre-filtered anomalies directly from the cloud hub (`GET /api/hermes/report`), treating verified errors and 4-run validated warnings as actionable system artifacts to trigger precise team notifications.
 
 ---
 
@@ -106,16 +107,20 @@ graph TB
 ---
 
 ## Cloud-Side Temporal Persistence & 4-Run Rule
-
+ 
 Incoming candidate logs are tracked in SQLite table `active_issues`:
 - **Issue Fingerprint**: Formatted as `{site_name}:{server}:{signature}`.
 - **12-Hour Evaluation Window**:
   - When an issue is observed, the hub compares `(now - last_seen)`.
-  - If more than 12 hours have passed since the issue was last recorded, the previous window is expired and the cycle resets to `run_count = 1` with status `TRANSIENT`.
+  - If more than 12 hours have passed since the issue was last recorded, the previous window is expired and the cycle resets to `run_count = 1`.
-- **4-Run Rule**:
+- **4-Run Rule for Warnings**:
+  - The 4-run persistence threshold specifically applies to `WARNING` (and `INFO`) events to eliminate transient operational noise.
   - For each distinct run batch, `run_count` increments.
-  - Issues with `run_count < 4` are marked as `TRANSIENT` and ignored by downstream reporting.
+  - Warnings with `run_count < 4` are marked as `TRANSIENT` and excluded from Hermes reports.
-  - When `run_count >= 4` within the active 12-hour window, the status transitions to `VERIFIED`.
+  - When `run_count >= 4` within the active 12-hour window, the warning transitions to `VERIFIED`.
+- **Immediate Verification for Errors**:
+  - High-severity events (`ERROR`, `CRITICAL`, `FATAL`) **always pass immediately**.
+  - On their very first ingestion (`run_count = 1`), errors are promoted directly to `VERIFIED` and surfaced to Hermes without waiting for 4 runs.
 
 ---
 
@@ -124,7 +129,7 @@ Incoming candidate logs are tracked in SQLite table `active_issues`:
 The server hub serves a REST reporting API (default port `8443`):
 
 ### `GET /api/hermes/report`
-Returns exclusively **verified anomalies** that have satisfied the 4-run rule within the active 12-hour evaluation window:
+Returns all **verified anomalies** (immediate critical errors and warnings verified after 4 consecutive runs within the 12-hour evaluation window):
 
 ```json
 [

RELEASE_NOTES.md

@@ -6,5 +6,6 @@
 - **Removed Client Filter Logic**: Removed restrictive source-level noise filtering on edge forwarders. Clients now collect and stream all candidate events from `INFO` up to `ERROR` over the lookback window instead of discarding them at the source.
 - **State Tracking & Deduplication**: Added persistent client state tracking (`client_state.json`) with journalctl cursors and Windows Event Log record numbers to guarantee that previously transmitted events are never resent.
 - **24-Hour Lookback Window**: Forwarders now scan and upload events from the last 24 hours (default `--hours 24`), skipping older entries.
+- **Warning Persistence & Immediate Error Routing**: Restructured temporal verification on the central hub so the 4-run persistence rule across the 12-hour evaluation window strictly governs `WARNING` and `INFO` events to suppress transient blips. High-severity `ERROR`, `CRITICAL`, and `FATAL` events are now promoted to `VERIFIED` immediately on their first occurrence and reported to Hermes without waiting for consecutive runs.
 - **Lightweight Distribution Structure**: Cleaned `out/` to strictly contain deployment documentation and sample configurations.
 - **Automated Gitea CI/CD**: Integrated push testing workflow (`ci.yml`) and multi-platform release asset packaging.

out/linux_server/README.md

@@ -11,7 +11,7 @@ Standalone compiled executable binary distribution for Linux server environments
 ### Key Architecture & Capabilities
 - **Pre-compiled & Dependency-Free**: Ships as a standalone native Linux ELF binary (`Server.bin`). No Python runtime, pip dependencies, or GnuPG binaries are required on the host system.
 - **Authenticated TCP Ingestion Socket (Port 9443)**: Accepts framed OpenPGP encrypted log batches streamed by edge forwarders (`Linux_Client.bin` and `Win_Client.exe`).
-- **4-Run Temporal Persistence Rule**: Ingested candidate error signatures are evaluated against an episodic threshold. An anomaly must occur across at least 4 distinct client transmission cycles within a sliding 12-hour evaluation window before promotion from transient noise to a `VERIFIED` anomaly.
+- **Warning Persistence & Immediate Error Routing**: High-severity `ERROR`, `CRITICAL`, and `FATAL` events are promoted to `VERIFIED` immediately on their first occurrence. Operational `WARNING` and `INFO` events are evaluated against an episodic threshold, requiring persistence across at least 4 distinct client transmission cycles within a sliding 12-hour evaluation window before promotion from transient noise to `VERIFIED`.
 - **Embedded Hermes Reporting API (Port 8443)**: Integrated REST API exposing `/api/hermes/report` for external scrapers, SIEM collectors, and alerting dashboards.
 - **Pure-Python OpenPGP Cryptography**: Zero dependency on external `gpg` binaries. Automatically generates RSA-2048 encryption keys and SHA-256 fingerprints on first launch.
 - **State Database**: Tracks anomaly lifecycles, run counters, and machine telemetry in a local SQLite state database (`logar_state.db`).
@@ -68,8 +68,8 @@ The generated `server_config.json` contains:
 | `hermes_port` | `8443` | HTTP port for the Hermes reporting endpoint |
 | `auth_token` | *(auto-generated)* | Pre-shared secret required in edge client envelopes |
 | `db_path` | `"logar_state.db"` | Path to persistent SQLite issue database |
-| `evaluation_window_hours` | `12` | Sliding temporal window for 4-run rule persistence |
+| `evaluation_window_hours` | `12` | Sliding temporal window for warning persistence |
-| `min_persistence_runs` | `4` | Number of distinct runs required to promote to `VERIFIED` |
+| `min_persistence_runs` | `4` | Number of distinct runs required to promote warnings to `VERIFIED` |
 
 ---
 

out/win_server/README.md

@@ -11,7 +11,7 @@ Standalone compiled executable distribution for Windows Server environments (`Se
 ### Key Architecture & Capabilities
 - **Pre-compiled & Dependency-Free**: Ships as a standalone Windows executable (`Server.exe`). No Python installation, pip packages, or GnuPG binaries are required on Windows Server.
 - **Authenticated TCP Ingestion Socket (Port 9443)**: Ingests framed OpenPGP encrypted log batches streamed from edge forwarder nodes (`Win_Client.exe` and `Linux_Client.bin`).
-- **4-Run Temporal Persistence Rule**: Filters transient noise by requiring an issue signature to recur across at least 4 episodic transmission cycles within a rolling 12-hour evaluation window before promotion to `VERIFIED`.
+- **Warning Persistence & Immediate Error Routing**: High-severity `ERROR`, `CRITICAL`, and `FATAL` events are promoted to `VERIFIED` immediately on their first occurrence. Operational `WARNING` and `INFO` events are evaluated against an episodic threshold, requiring persistence across at least 4 distinct client transmission cycles within a rolling 12-hour evaluation window before promotion to `VERIFIED`.
 - **Embedded Hermes Reporting API (Port 8443)**: Integrated REST API exposing `/api/hermes/report` for external dashboards, monitoring agents, and scrapers.
 - **Pure-Python OpenPGP Cryptography**: Automatically generates RSA-2048 encryption keys and a SHA-256 fingerprint on first launch without external dependencies.
 - **State Database**: Stores issue lifecycle records, run counters, and machine telemetry in a local SQLite database (`logar_state.db`).
@@ -68,8 +68,8 @@ The generated `server_config.json` contains:
 | `hermes_port` | `8443` | HTTP port for the Hermes reporting endpoint |
 | `auth_token` | *(auto-generated)* | Pre-shared authentication secret required in client envelopes |
 | `db_path` | `"logar_state.db"` | Path to persistent SQLite issue database |
-| `evaluation_window_hours` | `12` | Rolling evaluation window in hours for 4-run rule |
+| `evaluation_window_hours` | `12` | Rolling evaluation window in hours for warning persistence |
-| `min_persistence_runs` | `4` | Consecutive runs required to promote an issue to `VERIFIED` |
+| `min_persistence_runs` | `4` | Consecutive runs required to promote warning issues to `VERIFIED` |
 
 ---
 

src/Server.py

@@ -183,6 +183,9 @@ def process_ingested_logs(payload: Dict[str, Any], db_path: str, window_hours: i
         if severity in ["DEBUG", "TRACE"]:
             continue
 
+        # Errors are always passed immediately; the 4-run rule only concerns warnings
+        is_error = severity in ["ERROR", "CRITICAL", "FATAL"]
+
         signature = log.get("signature", "unknown")
         server = log.get("server", client_server)
         message = log.get("message", "")
@@ -207,7 +210,7 @@ def process_ingested_logs(payload: Dict[str, Any], db_path: str, window_hours: i
                 # Window elapsed: reset to new cycle
                 new_runs = 1
                 new_first_seen = now_iso
-                new_status = "TRANSIENT"
+                new_status = "VERIFIED" if is_error else "TRANSIENT"
             else:
                 # Same run guard: only increment count once per distinct run batch
                 if last_run_id != run_id:
@@ -215,8 +218,8 @@ def process_ingested_logs(payload: Dict[str, Any], db_path: str, window_hours: i
                 else:
                     new_runs = run_count
                 new_first_seen = first_seen_str
-                # 4-run rule enforcement
+                # 4-run rule applies to warnings; errors are always passed immediately as VERIFIED
-                new_status = "VERIFIED" if new_runs >= min_runs else "TRANSIENT"
+                new_status = "VERIFIED" if (is_error or new_runs >= min_runs) else "TRANSIENT"
 
             if new_status == "VERIFIED" and current_status != "VERIFIED":
                 promoted_to_verified += 1
@@ -227,7 +230,9 @@ def process_ingested_logs(payload: Dict[str, Any], db_path: str, window_hours: i
                 WHERE fingerprint = ?
             """, (new_runs, now_iso, new_first_seen, new_status, run_id, message, severity, fp))
         else:
-            initial_status = "VERIFIED" if 1 >= min_runs else "TRANSIENT"
+            initial_status = "VERIFIED" if (is_error or 1 >= min_runs) else "TRANSIENT"
+            if initial_status == "VERIFIED":
+                promoted_to_verified += 1
             cursor.execute("""
                 INSERT INTO active_issues 
                 (fingerprint, site_name, server, signature, severity, message, os_type, first_seen, last_seen, run_count, status, last_run_id)
@@ -328,8 +333,8 @@ def get_hermes_report():
     cursor.execute("""
         SELECT fingerprint, site_name, server, signature, severity, message, os_type, first_seen, last_seen, run_count, status
         FROM active_issues
-        WHERE status = 'VERIFIED' AND run_count >= ?
+        WHERE status = 'VERIFIED'
-    """, (min_runs,))
+    """)
     rows = cursor.fetchall()
     conn.close()
 
@@ -452,7 +457,7 @@ def main():
     print("=" * 60)
     print(f"  LOGAR Server Hub: {config['server_name']}")
     print(f"  Encryption Fingerprint: {config['server_fingerprint']}")
-    print(f"  Evaluation Window: {config['evaluation_window_hours']} hours | Rule: {config['min_persistence_runs']}+ consecutive runs")
+    print(f"  Evaluation Window: {config['evaluation_window_hours']} hours | 4-Run Rule: Warnings | Immediate Pass: Errors")
     print("=" * 60)
 
     try:

tests/test_pipeline.py

@@ -80,20 +80,25 @@ def run_tests():
     assert bad_resp.get("status") == "error", f"Expected error, got: {bad_resp}"
     print(f"[OK] Bad auth rejected correctly: {bad_resp['message']}")
 
-    test_signature = "TestServiceCrash"
+    test_signature = "TestServiceDegraded"
     candidate_log = [{
         "server": "test-edge-node",
         "os_type": "linux",
         "signature": test_signature,
-        "severity": "ERROR",
+        "severity": "WARNING",
-        "message": "Out of memory killer triggered"
+        "message": "Resource usage high warning"
     }]
 
-    print("\n=== [3] Testing Temporal Persistence & 4-Run Rule ===")
+    print("\n=== [3] Testing Temporal Persistence & 4-Run Rule for Warnings ===")
     for run_num in range(1, 5):
         resp = send_socket_batch(candidate_log)
         assert resp.get("status") == "success", f"Run {run_num} failed: {resp}"
-        print(f"[Run {run_num}/4] Ingested successfully. Promoted to verified: {resp.get('promoted_verified')}")
+        promoted = resp.get("promoted_verified", 0)
+        print(f"[Run {run_num}/4] Ingested successfully. Promoted to verified: {promoted}")
+        if run_num < 4:
+            assert promoted == 0, f"Expected 0 promoted on run {run_num} for warning, got {promoted}"
+        else:
+            assert promoted == 1, f"Expected 1 promoted on run 4 for warning, got {promoted}"
 
     # Inspect SQLite database directly
     conn = sqlite3.connect(server_conf.get("db_path", "logar_state.db"))
@@ -107,7 +112,33 @@ def run_tests():
     print(f"[DB Verification] Issue '{test_signature}' -> run_count: {run_count}, status: {status}")
     assert run_count >= 4, f"Expected run_count >= 4, got {run_count}"
     assert status == "VERIFIED", f"Expected status 'VERIFIED', got {status}"
-    print("[OK] 4-Run Rule verified: Transient issue promoted to VERIFIED anomaly!")
+    print("[OK] 4-Run Rule verified: Warning promoted to VERIFIED anomaly on 4th run!")
+
+    print("\n=== [3b] Testing Immediate Pass for Errors ===")
+    error_signature = "TestServiceCrashImmediate"
+    error_log = [{
+        "server": "test-edge-node",
+        "os_type": "linux",
+        "signature": error_signature,
+        "severity": "ERROR",
+        "message": "Fatal process crash occurred"
+    }]
+    err_resp = send_socket_batch(error_log)
+    assert err_resp.get("status") == "success", f"Error run failed: {err_resp}"
+    print(f"[Run 1/1] Error ingested successfully. Promoted to verified: {err_resp.get('promoted_verified')}")
+    assert err_resp.get("promoted_verified") == 1, f"Expected error to be promoted to verified immediately on run 1, got {err_resp.get('promoted_verified')}"
+
+    conn = sqlite3.connect(server_conf.get("db_path", "logar_state.db"))
+    cursor = conn.cursor()
+    cursor.execute("SELECT run_count, status FROM active_issues WHERE signature = ?", (error_signature,))
+    err_row = cursor.fetchone()
+    conn.close()
+    assert err_row is not None, "Error issue not found in SQLite"
+    err_run_count, err_status = err_row
+    print(f"[DB Verification] Issue '{error_signature}' -> run_count: {err_run_count}, status: {err_status}")
+    assert err_run_count == 1, f"Expected run_count == 1, got {err_run_count}"
+    assert err_status == "VERIFIED", f"Expected status 'VERIFIED', got {err_status}"
+    print("[OK] Immediate pass verified: Error promoted to VERIFIED anomaly immediately!")
 
     print("\n=== [4] Testing Hermes Reporting Endpoint (/api/hermes/report) ===")
     req = urllib.request.Request(f"http://{HERMES_HOST}:{HERMES_PORT}/api/hermes/report")
@@ -116,15 +147,21 @@ def run_tests():
         hermes_data = json.loads(response.read().decode("utf-8"))
     
     print(f"[Hermes API] Returned {len(hermes_data)} verified anomalies:")
-    found_issue = False
+    found_warning = False
+    found_error = False
     for issue in hermes_data:
         print(f"  - Fingerprint: {issue['fingerprint']} | Consecutive Runs: {issue['consecutive_runs']} | Status: {issue['status']}")
         if issue["signature"] == test_signature:
-            found_issue = True
+            found_warning = True
             assert issue["verified"] is True
             assert issue["consecutive_runs"] >= 4
+        if issue["signature"] == error_signature:
+            found_error = True
+            assert issue["verified"] is True
+            assert issue["consecutive_runs"] == 1
 
-    assert found_issue, f"Test issue {test_signature} should be in Hermes report"
+    assert found_warning, f"Warning issue {test_signature} should be in Hermes report"
+    assert found_error, f"Error issue {error_signature} should be in Hermes report"
     print("[OK] Hermes reporting validated!")
 
     print("\n=== [5] Testing Windows Client Script Integration ===")

tests/test_server.py

@@ -79,9 +79,9 @@ class TestServerComponent(unittest.TestCase):
         Server.init_db(self.test_db)
         log_entry = {
             "server": "app-worker-01.corp.local",
-            "signature": "PostgresConnTimeout",
+            "signature": "PostgresConnWarning",
-            "severity": "ERROR",
+            "severity": "WARNING",
-            "message": "Connection to database pool timed out after 30s",
+            "message": "Connection to database pool near capacity: 85%",
             "os_type": "linux"
         }
         payload = {
@@ -89,7 +89,7 @@ class TestServerComponent(unittest.TestCase):
             "logs": [log_entry]
         }
 
-        # Runs 1 to 3: should remain TRANSIENT
+        # Runs 1 to 3: WARNING should remain TRANSIENT
         for run_idx in range(1, 4):
             res = Server.process_ingested_logs(payload, self.test_db, window_hours=12, min_runs=4)
             self.assertEqual(res["status"], "success")
@@ -97,24 +97,53 @@ class TestServerComponent(unittest.TestCase):
 
         conn = sqlite3.connect(self.test_db)
         c = conn.cursor()
-        c.execute("SELECT run_count, status FROM active_issues WHERE signature = ?", ("PostgresConnTimeout",))
+        c.execute("SELECT run_count, status FROM active_issues WHERE signature = ?", ("PostgresConnWarning",))
         row = c.fetchone()
         conn.close()
         self.assertEqual(row[0], 3)
         self.assertEqual(row[1], "TRANSIENT")
 
-        # Run 4: promotes to VERIFIED!
+        # Run 4: promotes WARNING to VERIFIED!
         res4 = Server.process_ingested_logs(payload, self.test_db, window_hours=12, min_runs=4)
         self.assertEqual(res4["promoted_verified"], 1)
 
         conn = sqlite3.connect(self.test_db)
         c = conn.cursor()
-        c.execute("SELECT run_count, status FROM active_issues WHERE signature = ?", ("PostgresConnTimeout",))
+        c.execute("SELECT run_count, status FROM active_issues WHERE signature = ?", ("PostgresConnWarning",))
         row = c.fetchone()
         conn.close()
         self.assertEqual(row[0], 4)
         self.assertEqual(row[1], "VERIFIED")
 
+    def test_error_immediate_pass(self):
+        Server.init_db(self.test_db)
+        log_entry = {
+            "server": "app-worker-01.corp.local",
+            "signature": "KernelPanicCritical",
+            "severity": "ERROR",
+            "message": "Kernel panic - not syncing: Fatal hardware error",
+            "os_type": "linux"
+        }
+        payload = {
+            "server": "app-worker-01.corp.local",
+            "logs": [log_entry]
+        }
+
+        # Run 1: ERROR must immediately promote to VERIFIED
+        res = Server.process_ingested_logs(payload, self.test_db, window_hours=12, min_runs=4)
+        self.assertEqual(res["status"], "success")
+        self.assertEqual(res["promoted_verified"], 1)
+
+        conn = sqlite3.connect(self.test_db)
+        c = conn.cursor()
+        c.execute("SELECT run_count, status, severity FROM active_issues WHERE signature = ?", ("KernelPanicCritical",))
+        row = c.fetchone()
+        conn.close()
+        self.assertIsNotNone(row)
+        self.assertEqual(row[0], 1)
+        self.assertEqual(row[1], "VERIFIED")
+        self.assertEqual(row[2], "ERROR")
+
     def test_server_severity_filtering(self):
         Server.init_db(self.test_db)
         payload = {
@@ -132,15 +161,20 @@ class TestServerComponent(unittest.TestCase):
 
         conn = sqlite3.connect(self.test_db)
         c = conn.cursor()
-        c.execute("SELECT signature FROM active_issues ORDER BY signature")
+        c.execute("SELECT signature, status FROM active_issues ORDER BY signature")
-        sigs = [r[0] for r in c.fetchall()]
+        rows = dict(c.fetchall())
         conn.close()
 
-        self.assertIn("SigInfo", sigs)
+        self.assertIn("SigInfo", rows)
-        self.assertIn("SigWarn", sigs)
+        self.assertIn("SigWarn", rows)
-        self.assertIn("SigErr", sigs)
+        self.assertIn("SigErr", rows)
-        self.assertNotIn("SigDebug", sigs)
+        self.assertNotIn("SigDebug", rows)
-        self.assertNotIn("SigTrace", sigs)
+        self.assertNotIn("SigTrace", rows)
+
+        # SigErr is immediately VERIFIED; SigWarn and SigInfo are TRANSIENT on run 1
+        self.assertEqual(rows["SigErr"], "VERIFIED")
+        self.assertEqual(rows["SigWarn"], "TRANSIENT")
+        self.assertEqual(rows["SigInfo"], "TRANSIENT")
 
 
 if __name__ == "__main__":