Establish layered integration and system testing strategy

[darthfork]

Summary

Unit tests are useful, but they do not give enough confidence for promgithub's HTTP wiring, webhook ingestion flow, Redis-backed behavior, async processing, or deployment setup. We need a layered test strategy that validates the service the way operators actually use it.

This issue tracks the test plan and serves as the parent for concrete testing work.

Goals

• Add confidence beyond unit tests.
• Validate real HTTP webhook-to-metrics behavior.
• Cover Redis-backed shared state and deduplication behavior.
• Add black-box deployment/system smoke coverage.
• Keep fast PR feedback while reserving slower suites for heavier gates.

Proposed test layers

1. Unit tests

• Pure logic, payload normalization, metric transition logic, middleware behavior.
• Fast and implementation-focused.

2. Integration tests

• Boot the real HTTP server in-process.
• Exercise /webhook, /health, and /metrics over HTTP.
• Assert scrape output rather than only internal metric objects.
• Add a Redis-backed integration mode for dedupe and shared state validation.

3. System tests

• Run promgithub as a built artifact/container with supporting services.
• Validate config wiring, startup, health, metrics exposure, and basic end-to-end behavior from outside the process.
• Add Helm/chart smoke coverage in a disposable cluster.

Child issues

• Add in-process HTTP integration tests for webhook-to-metrics flows.
• Add Redis-backed integration tests for deduplication and shared state.
• Add async/backpressure integration coverage.
• Add black-box container and deployment smoke tests.
• Add CI lane design for layered test execution.

Acceptance criteria

• Each supported webhook type is covered by at least one HTTP-to-scrape integration test.
• Redis-backed dedupe/state behavior is validated with a real Redis instance.
• At least one black-box system smoke path validates startup, health, metrics, and webhook ingestion.
• CI clearly separates fast PR tests from slower deeper validation suites.