LOGAR/README.md

# LOGAR: Edge-Thin Log Analysis & Temporal Verification System

**LOGAR** is an enterprise log aggregation, verification, and anomaly detection architecture designed for heterogeneous server fleets (Windows & Linux). It combines lightweight zero-state edge forwarders with a centralized cloud hub that applies OpenPGP encryption, authenticated TCP streaming, temporal persistence tracking across 12-hour evaluation windows, and an automated 4-run rule to filter out transient infrastructure blips before reporting verified anomalies to **Hermes**.

---

## Table of Contents
1. [Core Philosophy](#core-philosophy)
2. [Architecture & Data Flow](#architecture--data-flow)
3. [Security & Cryptographic Model](#security--cryptographic-model)
4. [Cloud-Side Temporal Persistence & 4-Run Rule](#cloud-side-temporal-persistence--4-run-rule)
5. [Agentic Hermes Integration](#agentic-hermes-integration)
6. [Dynamic Machine & Domain Identification](#dynamic-machine--domain-identification)
7. [Repository & Shippables Structure](#repository--shippables-structure)
8. [Getting Started & Installation](#getting-started--installation)
9. [Running Tests](#running-tests)
10. [Automated Releases via Gitea Actions](#automated-releases-via-gitea-actions)

---

## Core Philosophy

### 1. Edge Thinness & Zero State
Site agents running on Windows and Linux act strictly as lightweight forwarders:
- **No Local Database**: Clients maintain zero state and no local SQLite or cache files.
- **Source-Level Filtering**: Agents upload candidate entries spanning from informational events up to errors (`INFO`, `WARNING`, `ERROR`, `CRITICAL`), while stripping verbose debugging noise (`DEBUG`, audit entries) and skipping any entries older than 24 hours.
- **End-to-End Encryption**: Logs are encrypted using the server's OpenPGP public key before leaving the edge node.
- **Secure TCP Sockets**: Ingestion occurs over low-overhead authenticated TCP sockets rather than bulky HTTP/HTTPS endpoints.

### 2. Cloud-Side Temporal Persistence
The central Python/TCP hub handles the heavy lifting:
- State tracking is managed centrally in SQLite (`logar_state.db`).
- Candidate issues are evaluated over a **12-hour temporal evaluation window**.
- An issue must persist across **at least 4 consecutive runs / cycles** to be confirmed as a genuine system anomaly. Transient blips and sporadic spikes are filtered out automatically.

### 3. Agentic Integration with Hermes
Instead of human engineers manually diving through noisy logs, **Hermes** ingests pre-filtered, 4-run validated anomalies directly from the cloud hub (`GET /api/hermes/report`), treating them as verified system artifacts to trigger precise team notifications.

---

## Architecture & Data Flow

```mermaid
graph TB
    subgraph Edge Nodes [State-Tracking Edge Forwarders]
        W[Win_Client.py / Win_Client.exe / Win_Client.pyz<br/>Windows Event Log Application]
        L[Linux_Client.py / Linux_Client.bin<br/>systemd journalctl -p info]
    end

    subgraph Security Layer [Security & Framing]
        E[OpenPGP Payload Encryption<br/>Server Public Key & Fingerprint]
        S[Length-Prefixed Framing<br/>4-byte Big-Endian + Auth Envelope]
    end

    subgraph Cloud Hub [LOGAR Central Server Hub]
        TCP[Authenticated TCP Listener<br/>Port 9443]
        DEC[OpenPGP Decryption<br/>Server Private Key]
        DB[(SQLite Persistence<br/>active_issues & ingest_runs)]
        RULE{12h Window &<br/>4-Run Rule}
    end

    subgraph Agentic Reporting [Downstream Integration]
        API[FastAPI / Uvicorn Reporting<br/>Port 8443]
        HERMES[Hermes Agent<br/>GET /api/hermes/report]
    end

    W --> E
    L --> E
    E --> S
    S -->|TCP Stream| TCP
    TCP --> DEC
    DEC --> RULE
    RULE --> DB
    DB --> API
    API --> HERMES
```

---

## Security & Cryptographic Model

### Pure-Python OpenPGP (RFC 4880)
- **Zero OS Binary Dependency**: Utilizes `pgpy` and `cryptography` in pure Python. **No native GnuPG or `gpg` binary installation is required** on the server, Windows nodes, or Linux nodes.
- **First-Run Automatic Key Generation**: On the first launch, if `server_config.json` is missing, `Server.py` automatically generates:
  - An OpenPGP RSA 2048 keypair with encryption-only usage flags.
  - An armored private key (`private_key`) and public key (`public_key`).
  - A SHA-256 public encryption fingerprint (`server_fingerprint`).
  - A cryptographically random authentication secret token (`auth_token`).
- **Client Configuration Exporter**:
  ```bash
  python src/Server.py --create-client-config --server-host 127.0.0.1 --server-port 9443 --client-out client_config.json
  ```
  Produces an anonymous client config containing only the server socket coordinates, authentication token, and the encryption-only public key & fingerprint.
- **Socket Protocol Framing**:
  - `[4 bytes big-endian unsigned int]` : Total envelope length.
  - `[JSON Envelope]` :
    ```json
    {
      "auth_token": "<SECRET_TOKEN>",
      "timestamp": "2026-09-03T...",
      "encrypted_payload": "-----BEGIN PGP MESSAGE-----\n..."
    }
    ```
  - Unauthorized clients or invalid authentication tokens are rejected immediately.

---

## Cloud-Side Temporal Persistence & 4-Run Rule

Incoming candidate logs are tracked in SQLite table `active_issues`:
- **Issue Fingerprint**: Formatted as `{site_name}:{server}:{signature}`.
- **12-Hour Evaluation Window**:
  - When an issue is observed, the hub compares `(now - last_seen)`.
  - If more than 12 hours have passed since the issue was last recorded, the previous window is expired and the cycle resets to `run_count = 1` with status `TRANSIENT`.
- **4-Run Rule**:
  - For each distinct run batch, `run_count` increments.
  - Issues with `run_count < 4` are marked as `TRANSIENT` and ignored by downstream reporting.
  - When `run_count >= 4` within the active 12-hour window, the status transitions to `VERIFIED`.

---

## Agentic Hermes Integration

The server hub serves a REST reporting API (default port `8443`):

### `GET /api/hermes/report`
Returns exclusively **verified anomalies** that have satisfied the 4-run rule within the active 12-hour evaluation window:

```json
[
  {
    "fingerprint": "corp.internal:web-app-01.corp.internal:NginxWorkerCrash",
    "site": "corp.internal",
    "server": "web-app-01.corp.internal",
    "signature": "NginxWorkerCrash",
    "severity": "ERROR",
    "message": "Worker process 4120 terminated with signal 11",
    "os_type": "linux",
    "first_seen": "2026-09-03T09:00:00+00:00",
    "last_seen": "2026-09-03T21:00:00+00:00",
    "consecutive_runs": 4,
    "evaluation_window": "12h",
    "verified": true,
    "status": "VERIFIED"
  }
]
```

### `GET /api/hermes/all`
Diagnostic endpoint listing all active issues (both `TRANSIENT` candidate blips and `VERIFIED` anomalies).

### `GET /health`
Returns hub health, encryption fingerprint, and listener ports.

---

## Dynamic Machine & Domain Identification

Client configurations intentionally contain **no machine name or site name**. Both forwarders dynamically identify their host and domain at runtime via `get_machine_identifier()`:
1. **Fully Qualified Domain Name (FQDN)**: Checked via `socket.getfqdn()`.
2. **OS-Specific Domain Discovery**:
   - **Windows**: Checks Active Directory environment variable `USERDNSDOMAIN` / `USERDOMAIN`.
   - **Linux**: Parses `/etc/resolv.conf` `domain` and `search` directives.
3. **Reverse DNS Lookup**: Resolves canonical hostname via `socket.gethostbyaddr`.
4. **Fallback**: Local hostname `socket.gethostname()`.

The server automatically infers site attribution from domain qualifiers (e.g. `node01.corp.internal` $\rightarrow$ site `corp.internal`).

---

## Repository & Shippables Structure

```
LOGAR/
├── .gitea/
│   └── workflows/
│       ├── ci.yml                  # Continuous Integration automated test suite (runs on every push)
│       ├── release-linux.yml       # Linux release workflow (compiles Server.bin, Linux_Client.bin, checksums)
│       └── release-windows.yml     # Windows release workflow (compiles Server.exe, Win_Client.exe, checksums)
├── .gitignore                      # Ignore venv, caches, DBs, and private keys
├── requirements.txt                # Unified dependencies
├── README.md                       # Comprehensive documentation
├── src/                            # Core application source modules
│   ├── __init__.py
│   ├── Server.py                   # Central TCP server and Hermes API
│   ├── Win_Client.py               # Windows edge forwarder
│   └── Linux_Client.py             # Linux edge forwarder
├── server_config.sample.json       # Central server sample configuration
├── package_dist.py                 # Multi-platform standalone binary packaging script
├── upload_release.py               # Direct Gitea REST API release asset publisher
├── test_pipeline.py                # End-to-end integration test
├── tests/                          # Unified unit test suites
│   ├── test_server.py              # Server unit tests
│   ├── test_win_client.py          # Windows client unit tests
│   └── test_linux_client.py        # Linux client unit tests
└── out/                            # Edge forwarder deployment packages
    ├── win_client/
    │   ├── client_config.sample.json # Reference client configuration
    │   └── README.md               # Windows service installation & configuration guide
    └── linux_client/
        ├── client_config.sample.json # Reference client configuration
        └── README.md               # Linux service installation & configuration guide
```

---

## Getting Started & Installation

### 1. Central Server Hub

1. **Install dependencies**:
   ```bash
   pip install -r requirements.txt
   ```
2. **Start the server** (generates `server_config.json` and keypair on first run):
   ```bash
   python src/Server.py
   ```
3. **Export a client configuration**:
   ```bash
   python src/Server.py --create-client-config --server-host <SERVER_IP> --server-port 9443 --client-out client_config.json
   ```

### 2. Windows Client Deployment

#### Option A: Precompiled Standalone Executable (Recommended)
1. Download `Win_Client.exe` (or `Win_Client.pyz`) from the repository releases.
2. Place `client_config.json` (exported from the server) in the same directory.
3. Run manually or schedule via Task Scheduler (every 3 hours):
   ```powershell
   .\Win_Client.exe --hours 24
   ```

#### Option B: Python Source Execution
1. Copy `Win_Client.py`, `requirements.txt`, and `client_config.json` to the target machine.
2. Install client dependencies:
   ```powershell
   python -m pip install -r requirements.txt
   ```
3. Run manually or schedule via Task Scheduler:
   ```powershell
   python Win_Client.py --hours 24
   ```

### 3. Linux Client Deployment

#### Option A: Precompiled Standalone Binary (Recommended)
1. Download `Linux_Client.bin` from the repository releases.
2. Place `Linux_Client.bin` and `client_config.json` into `/opt/logar/` and make it executable:
   ```bash
   chmod +x /opt/logar/Linux_Client.bin
   ```
3. Run via cron or systemd timer:
   ```bash
   0 */3 * * * /opt/logar/Linux_Client.bin --hours 24
   ```

#### Option B: Python Source Execution
1. Copy `Linux_Client.py`, `requirements.txt`, and `client_config.json` to `/opt/logar/`.
2. Install client dependencies:
   ```bash
   python3 -m pip install -r requirements.txt
   ```
3. (Optional) Run `out/linux_client/build_bin.sh` to compile a standalone ELF binary locally if desired.
4. Run via cron or systemd timer:
   ```bash
   0 */3 * * * python3 /opt/logar/Linux_Client.py --hours 24
   ```

---

## Running Tests

### 1. Component-Specific Unit Tests
The test suite is located in `tests/` and exercises all components:

```bash
# Run all unit tests
python -m unittest discover -s tests

# Or run component tests individually:
python -m unittest tests/test_server.py
python -m unittest tests/test_win_client.py
python -m unittest tests/test_linux_client.py
```

### 2. End-to-End Pipeline Integration Test
The pipeline test exercises invalid token rejection, encrypted socket streaming, database persistence, status promotion upon the 4th run, and the Hermes API report output.

1. **Start the server** in Shell 1 (creates `server_config.json` on first run):
   ```bash
   python src/Server.py
   ```
2. **Export client configuration** in Shell 2 (required for testing):
   ```bash
   python src/Server.py --create-client-config --server-host 127.0.0.1 --server-port 9443 --client-out client_config.json
   ```
3. **Execute the integration test** in Shell 2:
   ```bash
   python test_pipeline.py
   ```

## Continuous Integration via Gitea Actions

Continuous integration is automated via [`.gitea/workflows/ci.yml`](.gitea/workflows/ci.yml) and triggers automatically on **every push** and pull request:
1. **Syntax Compilation**: Validates all Python scripts (`src/Server.py`, `src/Win_Client.py`, `src/Linux_Client.py`, `package_dist.py`, `upload_release.py`, `test_pipeline.py`, and test suites).
2. **Component Unit Tests**: Discovers and runs all unit tests in `tests/` (`test_server.py`, `test_win_client.py`, `test_linux_client.py`).
3. **End-to-End Pipeline Verification**: Automatically spins up the LOGAR server hub, generates test configs, runs `test_pipeline.py` (testing socket authentication, 4-run rule persistence, Hermes API report, and client integrations), and shuts down the test instance.

---

## Automated Releases via Gitea Actions

Release builds are automated via two dedicated Gitea Actions workflows running concurrently on native platform runners:
- [`.gitea/workflows/release-linux.yml`](.gitea/workflows/release-linux.yml) (`ubuntu-latest`)
- [`.gitea/workflows/release-windows.yml`](.gitea/workflows/release-windows.yml) (`windows-latest`)

### Publishing a Release
Whenever you want to release a new version with compiled standalone binaries:
```bash
git tag v1.0.1
git push origin v1.0.1
```
*(You can also trigger builds manually via the Gitea UI using the **Run workflow** button (`workflow_dispatch`) on either workflow).*

### Automated Multi-Platform Compilation:
1. **Linux Runner** (`release-linux.yml`):
   - Compiles native Linux ELF executables: `Linux_Client.bin` and `Server.bin`.
   - Generates dedicated SHA-256 checksum files:
     - `linux_client_sha256sum` (verification for `Linux_Client.bin`)
     - `linux_agent_sha256sum` (alias for client/agent integrations)
     - `linux_server_sha256sum` (verification for `Server.bin`)
     - `SHA256SUMS_linux.txt` (summary manifest)
   - Attaches all Linux assets to the Gitea release.

2. **Windows Runner** (`release-windows.yml`):
   - Compiles native Windows PE executables: `Win_Client.exe` and `Server.exe`.
   - Generates dedicated SHA-256 checksum files:
     - `win_client_sha256sum` (verification for `Win_Client.exe`)
     - `win_agent_sha256sum` (alias for client/agent integrations)
     - `win_server_sha256sum` (verification for `Server.exe`)
     - `SHA256SUMS_windows.txt` (summary manifest)
   - Attaches all Windows assets to the Gitea release.

3. **Concurrent Publishing & Conflict Handling**:
   `upload_release.py` includes automatic retry and conflict resolution so concurrent Windows and Linux runners attach their respective assets to the release without collision.

### Verifying Checksums
- On Linux:
  ```bash
  sha256sum -c linux_client_sha256sum
  # or
  sha256sum -c linux_server_sha256sum
  ```
- On Windows (PowerShell):
  ```powershell
  Get-FileHash .\Win_Client.exe -Algorithm SHA256
  Get-Content .\win_client_sha256sum
  ```

### Local Packaging & Manual Upload
You can also compile and package binaries locally anytime:
```bash
# Windows
py -3.12 package_dist.py --target windows

# Linux
python3 package_dist.py --target linux
```
To upload local builds directly to Gitea:
```powershell
python upload_release.py --tag v1.0.1 --token <YOUR_GITEA_TOKEN>
```
*(Environment variables `GITEA_TOKEN`, `GITEA_SERVER_URL`, `GITEA_REPOSITORY`, and `GITEA_REF_NAME` are also supported automatically).*