L2 IT Support Engineer

L2 Observability / SRE Engineer (Prometheus, Loki, Grafana + OpenStack)

You will own Cloudoor's observability stack end to end and be the escalation point for incidents on our OpenStack cloud. You turn noisy signals into clear alerts, fast diagnosis and fewer repeat incidents.

What you'll do
• Design, write and tune PromQL queries, recording rules and alerting rules for OpenStack services, hypervisors and customer workloads.
• Own Alertmanager configuration: routing trees, receivers, grouping, silences and inhibition rules that cut alert noise without hiding real problems.
• Build Grafana dashboards that answer operational questions at a glance, from service-level overviews to per-component drill-downs.
• Run log pipelines in Loki and write LogQL queries to trace failures across Nova, Neutron, Cinder, Keystone and related services.
• Handle L2 escalations from the L1 team: diagnose OpenStack issues (failed instance builds, networking faults, volume attach errors, API failures) and resolve or hand off with a clear write-up.
• Lead incident response and write blameless post-mortems with concrete follow-up actions.
• Write runbooks and coach L1 engineers so more incidents are resolved at first line.
• Define SLIs and SLOs with the team and track error budgets.

What you'll bring (required)
• Strong PromQL: rates, aggregations, histograms and quantiles, recording rules, and alert design that avoids flapping.
• Hands-on Alertmanager experience, including routing and inhibition rules in production.
• Grafana dashboard design with templating and variables, built for the people who will read them.
• Loki and LogQL: label strategy, log queries and metric queries from logs.
• Solid Linux administration and networking: systemd, journald, TCP/IP, DNS, routing, VLANs, and tools such as tcpdump, ss and ip.
• Practical OpenStack troubleshooting across core services and their logs.
• 3+ years in an SRE, DevOps, cloud operations or infrastructure role.

Nice to have
• OVN / Open vSwitch: logical flows, tracing packets through br-int.
• Ceph: cluster health, OSD and placement group troubleshooting, performance tuning.
• Kubernetes operations, including the Prometheus Operator or kube-prometheus-stack.
• OpenTelemetry for traces and metrics.
• Automation with Ansible, Python or Go, and config managed in Git.

What success looks like after 6 months
• Every production alert is actionable, routed to the right owner and linked to a runbook.
• Core OpenStack services have SLO dashboards the whole team uses.
• The L1 team resolves a clearly larger share of incidents without escalation.