Runtime monitoring produces an enormous stream of events. A production Kubernetes cluster with 20 active services can generate thousands of syscall events every second. The issue is not whether a runtime tool can see them all, but whether it can identify the events that matter. A tool that alerts on everything is not a security tool. It is a noise machine that teaches engineers to disregard alerts.
Every AppSec team we speak with knows some version of this story: a prior tool was deployed, its alert volume became impossible to triage, the tool was muted, and trust in runtime alerts disappeared. Restoring that trust requires proving that firing alerts deserve attention. Behavioral baselining makes that possible without ongoing manual rule maintenance.
The Sources of Runtime False Positives
A basic runtime monitoring strategy applies fixed rules to every event: alert when a shell process starts, alert when a new IP address is contacted, and alert when a file under /etc is read. The rules themselves are not incorrect. A shell started by a web server is suspicious. Yet a platform that runs health check scripts through a shell, or services that routinely invoke system utilities, will trigger those rules repeatedly on benign activity.
False positive rate depends less on how rules look in the abstract than on how closely they match the monitored service's actual behavior. A rule that suits a stateless API may fire continuously on a service running periodic maintenance jobs. A rule designed for a payments processor may be unsuitable for a data processing pipeline. Applying generic rules everywhere creates false positives in proportion to the gap between the service and the rule author's mental model.
Behavioral Baselining
Behavioral baselining models what a particular workload normally does, then alerts when activity departs from that model. Rather than asking, "is this a known-bad event," it asks, "does this event fall outside this workload's normal range?"
The baseline records multiple dimensions of workload behavior:
- Process lineage. Which processes spawn which children. An API service that never spawns shell children should not be spawning shell children during production traffic.
- Network behavior. Which external IP ranges and ports the workload normally contacts. An outbound connection to a new IP address in a range the service has never contacted before is anomalous, even if the connection itself uses a normal port.
- File system access patterns. Which files and directories the workload reads and writes. Access to
/etc/shadowfrom a web application process is outside any legitimate access pattern. - Syscall frequency distributions. A process that makes 10
execvecalls per hour during normal operation and then makes 200 in a two-minute window is anomalous, even if each individual call looks legitimate.
How Oligo Creates and Updates Baselines
During a configurable initial learning window, typically 24 to 72 hours of production traffic, the baseline observes the workload. Events are recorded but do not generate alerts during this period. The resulting behavioral model includes process trees, network destinations, file access patterns, and syscall frequency distributions.
In Kubernetes, baselines belong to workload identity rather than individual pod instances. That identity consists of the deployment name plus namespace. A new pod with the same identity immediately receives the existing baseline. This correctly supports horizontal pod autoscaling: new pods serving an existing service do not begin with an empty baseline that must be learned again from scratch.
Baseline changes occur incrementally. When legitimate new behavior emerges, such as a new dependency contacting an external API or a deployment adding a scheduled job, the baseline can change in one of two ways. Automatically, the behavior is treated as confirmed safe after N occurrences without an associated exploitation signal. Or manually, an engineer reviews and explicitly accepts it. The manual option suits high-sensitivity workloads where automatic expansion could weaken the security guarantee.
The Deployment Challenge
Kubernetes rolling deployments are the most difficult scenario for behavioral baselining. A new service version may legitimately behave differently: a new endpoint may reach an external service, a new dependency may open files in a new path, or a new background worker may start processes absent from the prior version. Without version awareness, every deployment creates a wave of alerts.
Oligo handles this by making the active deployment version part of the baseline. When a deployment event appears through the Kubernetes API, specifically when the pod's app.kubernetes.io/version label or container image digest changes, the baseline enters a short confirmation window for the new version. During that window, sensitivity is reduced for changes consistent with deployment activity, while remaining active for changes that do not fit the deployment, such as an unexpected IP address appearing mid-deployment that is absent from the new version's configuration.
This is not blanket suppression during deployments. That distinction is critical because attackers know when organizations release code. Deployment creates a period of elevated activity that can conceal malicious behavior if monitoring suppresses every alert.
Baselining Does Not Replace
Behavioral baselining does not replace explicit detection rules. We still maintain rules for known-malicious patterns: specific JNDI lookup signatures for Log4Shell, process names linked to crypto miners, and the pattern of an execve to a shell immediately after a network receive event. These rules fire regardless of baseline state because they are so specific to exploitation that no legitimate baseline would contain them.
Baselining cuts noise in ambiguous cases where a generic rule would match both legitimate and malicious activity. It does not reduce confidence in rules with no legitimate behavioral equivalent. Combining explicit rules for known patterns with baseline deviation for novel patterns keeps alert volume low enough for engineers to read the alerts that do fire.
An alert that appears once a week and is correct three times out of three gets read. One that fires 40 times per day gets disabled. Alert quality is a trust issue, and trust returns through one accurate alert at a time.