Development Workflow

This guide outlines the standard process for creating and deploying new detection logic and parsing filters.

Rule Development Lifecycle

1. Identify the Threat: Research the specific attack pattern (e.g., MITRE ATT&CK technique).

2. Analyze the Logs: View sample logs in the UTMStack Log Explorer to identify unique identifiers for the threat.

3. Draft the YAML: Create a new rule file. Define metadata (id, name, impact).

4. Implement Conditions: Write the where expression and define any required correlation logic.

5. Local Validation: Test the YAML syntax using a YAML linter.

6. Deployment: Upload the rule to the rules directory of the EventProcessor.

7. Simulation: Generate "attack" logs and verify that an alert is generated and properly deduplicated.

Filter Development Lifecycle

1. Source Discovery: Identify a new log source (e.g., a new firewall brand).

2. Schema Mapping: Map the raw log fields to the Standard Event Schema.

3. Pipeline Construction:

   • Start with a json or kv step if the log is structured.

• Use grok for unstructured strings.

• Add cast steps for types and rename for standardization.

1. Enrichment: Add dynamic steps for geolocation or TI lookups.

2. Sanitization: Remove temporary or highly redundant fields in the log.* namespace to optimize storage.

   Note: The raw field is protected for auditing purposes and cannot be removed.

3. Testing: Use the Playground or deploy to a dev environment and check the "Log Explorer" to ensure fields appear correctly.

Debugging Tips

• Check Logs: Monitor the EventProcessor logs for "parsing error" or "rule evaluation failed" messages.

• Isolate: If a filter is failing, comment out steps one by one to find the culprit.

• Validate Schema: Ensure that the types you cast to match the expected types in the rules.

• Audit Trace: Remember that the raw field always preserves the original message, which is useful for verifying your parsing logic.