Green is good news: the configured checks have passed. What matters is what those checks cover. A running process and a successful login are different test results.
In this article
What green means here
Consider a web application. Its process is running and its port is reachable, but login fails because a required database connection is broken. If monitoring only checks the process and port, both checks can still correctly report OK. The login flow simply has not been tested.
This is a reason to define the right checks. Good monitoring shows which functions work and where a problem affects the service.
From a running process to a working function
Process and port checks are useful. They quickly identify basic failures. A service that people depend on also needs suitable functional checks:
- Does the application respond within a reasonable time?
- Does it return the expected content, rather than just any successful HTTP response?
- Can an important, clearly bounded workflow complete successfully?
How far a test should go depends on the service. It should create little load and avoid changing production data. If it needs an account, grant only the permissions required for that test.
An alert should help resolve the problem
A useful alert names the affected function and its impact. Metrics and logs then support diagnosis. Retry policies, timeouts and appropriate alerting help distinguish brief anomalies from persistent failures.
Test the check itself as well. Does it detect a controlled failure? Does it report a timeout? Does the alert clear after recovery?
Give green a clear meaning
A useful status should be explainable: what was checked, when was it checked, and what counts as success? Green can then stand for a specific, verified function.
A practical guide is available in the Public Repository: Designing Useful Monitoring Checks.
Further reading: Google SRE: Monitoring Distributed Systems.