oc logs reads container stdout. It cannot tell you why the kubelet killed the pod, and that split is where most OpenShift log questions go wrong. Container output lives in the API server’s reach; the kubelet, CRI-O and NetworkManager write to the node’s systemd journal, which is a different command with a different permission model.
This guide covers OpenShift logs from the oc CLI end to end: pod and container logs with oc logs, node journals and /var/log files with oc adm node-logs, and a whole-cluster dump with oc adm must-gather. It also documents the flag combinations that return an empty result while exiting 0, which is the single most common reason a working command looks broken. Every command below was run on a live OpenShift 4.21.14 cluster in August 2026, on an RHCOS node with CRI-O as the container runtime.
Pick the right log source first
Four questions, three commands. Picking the wrong one is why people run oc logs against a node name and get a NotFound on a pod.
| What you need | Command | Source | Permission |
|---|---|---|---|
| Application stdout/stderr | oc logs | Container log file on the node, served by the kubelet | Namespace view |
| Kubelet, CRI-O, NetworkManager, OVS | oc adm node-logs -u <unit> | systemd journal on the node | Cluster, node admin |
| API audit trails, OVN logs, per-pod files | oc adm node-logs --path=<path> | Files under /var/log on the node | Cluster, node admin |
| Everything, for a support case | oc adm must-gather | Platform namespaces plus node services | cluster-admin |
Almost every command below takes a node name, so list the nodes and pick the one you are debugging:
oc get nodes -o wideThe single-node test cluster reports every role on one machine, which matters later when --role enters the picture:
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
crc Ready control-plane,master,worker 98d v1.34.6 192.168.126.11 <none> Red Hat Enterprise Linux CoreOS 9.6.20260504-0 (Plow) 5.14.0-570.112.1.el9_6.x86_64 cri-o://1.34.7-2.rhaos4.21.git7d1fe5c.el9Export the one you want. On a single-node cluster the first entry is the only entry, but on anything larger set it explicitly rather than trusting an index. Every later command uses ${NODE}:
export NODE=crc
echo "${NODE}"Create a throwaway workload to follow along
Every pod-level example below uses the same two pods: one with two containers, one that crashes on purpose. Building them takes a minute, so skip this only if you already have a misbehaving pod to aim at. The one section this does not cover is the DeploymentConfig example further down. That resource is still served on current releases, but creating one is outside what this walkthrough sets up.
oc new-project cfg-logs-demoPut both pods in one manifest:
vim logdemo.yamlThe writer counts, the ticker beats, and the crasher exits 1 after printing something that looks like a real failure:
apiVersion: v1
kind: Pod
metadata:
name: multi-log
labels:
app: logdemo
spec:
containers:
- name: writer
image: registry.access.redhat.com/ubi9/ubi-minimal
command: ["/bin/sh","-c","i=0; while true; do echo \"writer line $i\"; i=$((i+1)); sleep 5; done"]
- name: ticker
image: registry.access.redhat.com/ubi9/ubi-minimal
command: ["/bin/sh","-c","while true; do echo \"ticker heartbeat\"; sleep 7; done"]
---
apiVersion: v1
kind: Pod
metadata:
name: crasher
labels:
app: logdemo
spec:
containers:
- name: crasher
image: registry.access.redhat.com/ubi9/ubi-minimal
command: ["/bin/sh","-c","echo \"starting attempt\"; echo \"FATAL: cannot open /data/config.yaml\"; exit 1"]Apply it:
oc apply -f logdemo.yamlBoth pods are admitted immediately:
pod/multi-log created
pod/crasher createdThe controller examples later on need a Deployment as well, and the label deliberately differs from the pods above so the selector examples have something to exclude:
oc create deployment web --image=registry.access.redhat.com/ubi9/ubi-minimal -- /bin/sh -c "while true; do echo web-ready; sleep 10; done"After a minute the crasher has already failed a few times, which is exactly the state you want:
oc get podsTwo healthy containers in one pod, a deployment pod running, and a pod stuck in a restart loop:
NAME READY STATUS RESTARTS AGE
crasher 0/1 Error 2 45s
multi-log 2/2 Running 0 45s
web-77fd56b64-4ddxp 1/1 Running 0 45sTear the whole lot down with oc delete project cfg-logs-demo when you are finished.
Read pod and container logs with oc logs
The base form is the pod name, narrowed to one container with -c and to the last few lines with --tail:
oc logs multi-log -c writer --tail=3That returns the last three lines the container wrote:
writer line 6
writer line 7
writer line 8Multi-container pods behave in a way that catches people out. Omitting -c does not fail. The client picks the first container in the spec, tells you so, and prints that container’s log:
oc logs multi-logNotice the first line. It is a notice, not an error, and the exit status is 0:
Defaulted container "writer" out of: writer, ticker
writer line 0
writer line 1
writer line 2That notice goes to stderr, and the log lines go to stdout. Redirect the command to a file and the warning vanishes from the capture, leaving a file that silently contains one container out of two. This is worth internalising before you hand a log file to somebody else:
oc logs multi-log --tail=1 2>/dev/nullOnly the log line survives, with no hint that a second container exists:
writer line 12To read every container in the pod, ask for all of them and label the lines. Without --prefix the two streams are concatenated with nothing to tell them apart:
oc logs multi-log --all-containers --prefix --tail=2Each line now carries its origin in pod/container form:
[pod/multi-log/writer] writer line 7
[pod/multi-log/writer] writer line 8
[pod/multi-log/ticker] ticker heartbeat
[pod/multi-log/ticker] ticker heartbeatThe flags you will actually reach for day to day are -f to follow a live stream, --timestamps to prefix each line with an RFC 3339 timestamp when the application does not log one itself, --since for a relative window, and --limit-bytes as a blunt cap on a chatty container. That last one truncates on a byte boundary rather than a line boundary, so the final line of output arrives half-written.
oc logs multi-log -c ticker --timestamps --tail=2
oc logs multi-log -c writer --since=5m
oc logs -f multi-log -c tickerTimestamps come back with nanosecond precision, which is enough to line up two containers that are racing each other:
2026-08-20T00:09:41.458898662Z ticker heartbeat
2026-08-20T00:09:48.462541076Z ticker heartbeatThe web console exposes the same data on a pod’s Logs tab, and it defaults to the same first container the CLI picks. The dropdown next to the pause button is the console’s equivalent of -c, and the Current log dropdown on the right is where you switch to the previous container.
The line counter above the pane (“103 lines”) is a useful sanity check that you are looking at a live stream and not a stale render.
Error: “container nope is not valid for pod multi-log”
A typo in the container name fails loudly, which is the opposite of the defaulting behaviour above:
oc logs multi-log -c nopeThe client checks the name against the pod spec and refuses:
error: container nope is not valid for pod multi-logList the real container names straight from the pod spec rather than guessing them:
oc get pod multi-log -o jsonpath='{.spec.containers[*].name}'Init containers are not in that list, so a pod stuck in Init:0/1 needs -c <init-container-name> taken from .spec.initContainers instead.
Pull logs from a deployment, a label, or a DeploymentConfig
Pod names carry a random suffix that changes on every rollout, so scripting against them is a losing game. Point oc logs at the controller instead and it resolves a pod for you:
oc logs deployment/web --tail=2The output is the container log, not a description of the deployment:
web-ready
web-readyA label selector fans out across every matching pod in the namespace, and combines well with --prefix:
oc logs -l app=logdemo --tail=1 --prefixTwo pods answer, and the multi-container one still defaults to its first container, warning on stderr as before:
Defaulted container "writer" out of: writer, ticker
[pod/crasher/crasher] FATAL: cannot open /data/config.yaml
[pod/multi-log/writer] writer line 12DeploymentConfig is the OpenShift-specific case, and it is a trap. The resource is still served under apps.openshift.io/v1 on current releases, though the API now answers with a deprecation warning on stderr. What oc logs dc/<name> returns is the deployer pod’s log, meaning the record of the rollout, not the application’s output:
oc logs dc/legacyTwo deprecation warnings on stderr, then the deployer reporting that the rollout never became ready. Useful, and none of it is your application:
Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
Warning: apps.openshift.io/v1 DeploymentLog is deprecated in v4.14+, unavailable in v4.10000+
--> Scaling legacy-1 to 1
error: update acceptor rejected legacy-1: timed out waiting for the conditionNote the exit status on that one: it prints a line beginning error: and still exits 0, so a script checking $? reads it as a success. oc logs rc/<name> is not the answer either. It goes through the same deployment-log path, times out the same way, and does exit 1. For the application log behind a DeploymentConfig, list the pods the controller owns and read one of those directly. The same distinction applies to oc logs bc/<name>, which returns build output rather than runtime output.
Recover the logs of a crashed container
A pod in CrashLoopBackOff has no running container at all. Between attempts the container status is waiting, and the kubelet is counting down a backoff that doubles from 10 seconds to a ceiling of 5 minutes. While it waits, the last dead container is still on the node, so plain oc logs and oc logs --previous return the same bytes. --previous only becomes necessary once a newer container has started:
oc logs crasher --previousWhen the previous container is still on the node, the failure reason comes straight back:
starting attempt
FATAL: cannot open /data/config.yamlCross-check the exit code from the pod status, because a log that ends cleanly with a non-zero exit tells a different story than one that ends mid-write:
oc get pod crasher -o jsonpath='{.status.containerStatuses[0].lastState.terminated}'The terminated block names the container ID, the exit code and the timestamps:
{"containerID":"cri-o://3ef62cb74090632dd9ceff7052d4221ecc12630fceebcb6a9bf7c696ec48170a","exitCode":1,"finishedAt":"2026-08-20T00:10:02Z","reason":"Error","startedAt":"2026-08-20T00:10:02Z"}In the console the crashed pod looks like this, with the log stream marked as ended and the Current log dropdown offering the previous container.
Error: “unable to retrieve container logs for cri-o://…”
--previous is a race against garbage collection, and this is what losing the race looks like:
oc logs crasher --previousWhen the container has already been reaped, the message names an ID that no longer exists on the node:
unable to retrieve container logs for cri-o://a21f2040b7911cf7068cf8f3d087cdc0111e5f35d292b5ea1028acb8415c4e73The container ID in the message no longer exists. CRI-O reaped it along with its log file, and you can prove that from the node journal rather than guessing:
oc adm node-logs "${NODE}" -u crio --grep="RemoveContainer" --since=-30mCRI-O logs the removal with the namespace, pod and container it belonged to:
Aug 20 00:10:55.909587 crc crio[4472]: time="2026-08-20T00:10:55.909513812Z" level=info msg="Removed container 3ef62cb74090632dd9ceff7052d4221ecc12630fceebcb6a9bf7c696ec48170a: cfg-logs-demo/crasher/crasher" id=b0c5265b-66b5-4849-86f3-86519d5ff7b3 name=/runtime.v1.RuntimeService/RemoveContainerOnce that line appears the log is gone for good, so a pod that has been crash-looping overnight will rarely give up its first failure. Catching it earlier is the practical answer, and the backoff ladder is what gives you time. Counting the kubelet’s own backoff messages across the node’s current boot showed the doubling clearly: 10s, 20s, 40s, 1m20s, 2m40s, then 5m0s, with roughly nine in ten observations sitting at that ceiling. A pod that has been failing for a while therefore hands you a five-minute window between attempts. For anything you need after the fact, log shipping is the only real fix, which is what the cluster logging operator exists for.
Query node journals with oc adm node-logs
RHCOS runs almost nothing outside a container. The exceptions are the ones that matter when a node misbehaves: CRI-O, the kubelet, NetworkManager and the Open vSwitch stack are systemd units, and their output goes to the journal. oc adm node-logs reads that journal through the API server, so it needs no SSH access and no bastion host. Red Hat documents the command under gathering cluster data.
oc adm node-logs "${NODE}" -u kubelet --tail=20One detail derails journal searches constantly. The kubelet does not log under the identifier kubelet; on RHCOS the process tag is kubenswrapper. The unit filter -u kubelet is correct, but grepping the text for “kubelet” misses the overwhelming majority of the lines:
Aug 20 00:11:06.499614 crc kubenswrapper[4519]: E0820 00:11:06.498670 4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"crasher\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=crasher pod=crasher_cfg-logs-demo(cb43e341-0d53-44ef-831b-298771ae5304)\"" pod="cfg-logs-demo/crasher" podUID="cb43e341-0d53-44ef-831b-298771ae5304"Pass -u more than once to interleave two units, which is how you correlate a kubelet decision with the CRI-O action it triggered:
oc adm node-logs "${NODE}" -u crio -u kubelet --since=-3mSorting is on by default when you query journal units, so the two streams arrive in timestamp order rather than one after the other. Even a three-minute window on this single-node cluster returns both units’ lines interleaved rather than one block after the other, which is what makes correlating them worth doing at all. Which unit dominates depends entirely on what is happening at the time, so do not read anything into the split.
To hit every control plane node at once, swap the node name for a role. On a cluster where the same machine is both control plane and worker, the two role selectors return byte-identical output, because the selector is nothing more than a node label lookup:
oc adm node-logs --role master -u kubelet --tail=5
oc adm node-logs --role worker -u kubelet --tail=5
oc adm node-logs -l node-role.kubernetes.io/master= -u kubelet --tail=5There is no -f and no --follow on this command, so asking for one fails at argument parsing before any request reaches the cluster:
oc adm node-logs "${NODE}" -f --tail=2Argument parsing rejects the flag before any request is built:
error: unknown shorthand flag: 'f' in -f
See 'oc adm node-logs --help' for usage.For a live tail of a node service you need a shell on the node itself, which oc debug node/<name> gives you. The shell prompt on an OpenShift node walkthrough covers that route, and once you are there the usual journalctl -fu crio works. The same debug pod is how you get tcpdump and telnet onto a CoreOS node when the logs are not enough.
Error: “nodes … is forbidden: User … cannot get resource nodes”
Node logs are privileged. A developer with full rights inside a namespace has none at the cluster scope, and gets this. Reproducing it with --as needs impersonation rights of your own, so run these as cluster-admin:
oc adm node-logs "${NODE}" --tail=1 --as=developerImpersonating the user with --as reproduces exactly what they see, which is a faster check than logging in as them:
error: nodes "crc" is forbidden: User "developer" cannot get resource "nodes" in API group "" at the cluster scopeTwo permissions matter here, and neither is the one usually quoted. Watch the API server’s own audit log while the command runs and you see two requests from your user: a get on the node object, then a get on nodes/proxy against /api/v1/nodes/<node>/proxy/logs/journal. Count the audit lines and you get three, because the proxy request logs both a ResponseStarted and a ResponseComplete event. The subresource in play is proxy, and the denial above names nodes because the first of those two checks, the bare GET on the node object, is the one that fails. The user never reaches the proxy check. The ClusterRole that grants the pair is system:node-admin, singular:
oc get clusterrole system:node-admin -o jsonpath='{range .rules[*]}{.resources} {.verbs}{"\n"}{end}'The first and third rules are the ones your user needs. The middle one is the legacy proxy verb and is a red herring here:
["nodes"] ["get","list","watch"]
["nodes"] ["proxy"]
["nodes/log","nodes/metrics","nodes/proxy","nodes/spec","nodes/stats"] ["*"]Watch that name. oc adm node-logs --help states that "The system:node-admins role grants this permission by default", and no ClusterRole by that name exists:
oc get clusterrole system:node-adminsThe plural is the group, bound to the singular role, so following the help text verbatim gets you a dead end:
Error from server (NotFound): clusterroles.rbac.authorization.k8s.io "system:node-admins" not foundTwo built-in roles cover this, system:node-admin and system:kubelet-api-admin, and both carry the node object and the proxy subresource, which is why binding either one works. Bind by those exact names; a binding whose roleRef points at the nonexistent plural is created without complaint and grants nothing.
For a narrower role of your own, the grant is get on nodes plus get on nodes/proxy. The slash form matters here: RBAC matches a subresource request against resource/subresource, so the legacy proxy verb does not satisfy it. Add list on nodes too, or --role and -l will fail even though a named node works.
vim node-log-reader.yamlTwo rules, and note the verbs:
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: node-log-reader
rules:
- apiGroups: [""]
resources: ["nodes"]
verbs: ["get", "list"]
- apiGroups: [""]
resources: ["nodes/proxy"]
verbs: ["get"]Apply it and bind it to whoever needs node logs. Note the scope: this covers every oc adm node-logs form in this guide, but not oc debug node/<name>, which creates a privileged pod and needs its own permissions.
oc apply -f node-log-reader.yaml
oc adm policy add-cluster-role-to-user node-log-reader aliceNow the part that will waste your afternoon if nobody warns you. Check the grant on the user you just bound:
oc auth can-i get nodes/proxy --as=aliceIt answers yes on stdout, after a Warning: resource 'nodes' is not namespace scoped on stderr that only --all-namespaces silences. The yes looks like confirmation. It is not. In can-i the slash form means TYPE/NAME, not resource/subresource, so that command asked whether alice can get a node named proxy. The give-away is that any nonsense name answers the same way:
oc auth can-i get nodes/banana --as=aliceAlso yes, and there is no node called banana. Pass the subresource as a flag instead, which is the documented form and the only one that asks the real question:
oc auth can-i get nodes --subresource=proxy --as=aliceFor alice all three answer yes, so the difference is invisible. To see the trap you need a subject that holds cluster-wide get on nodes without the proxy subresource, and every cluster ships one. The machine-config daemon is a convenient example:
export MCD=system:serviceaccount:openshift-machine-config-operator:machine-config-daemon
oc auth can-i get nodes/proxy --as="${MCD}"
oc auth can-i get nodes --subresource=proxy --as="${MCD}"The two forms now disagree, and the second one is telling the truth:
yes
noConfirm which one to believe by running the real command as that subject:
oc adm node-logs "${NODE}" -u crio --tail=1 --as="${MCD}"Denied, on the subresource the slash form claimed was fine:
error: nodes "crc" is forbidden: User "system:serviceaccount:openshift-machine-config-operator:machine-config-daemon" cannot get resource "nodes/proxy" in API group "" at the cluster scopeFour roles against a throwaway service account, measured across both can-i forms and both node-logs forms, is the whole picture:
| ClusterRole grants | can-i get nodes/proxy | can-i get nodes --subresource=proxy | node-logs <node> | node-logs --role master |
|---|---|---|---|---|
proxy verb only | no | no | denied on nodes | denied on nodes |
get nodes + proxy verb | yes | no | denied on nodes/proxy | denied on nodes |
get nodes + get nodes/proxy | yes | yes | works | denied on nodes |
the role above (adds list) | yes | yes | works | works |
That is the trap in one line. The slash form says yes, the flag form says no, and the real command is denied. The daemon is not literally row two of the table above, since it holds list and not the legacy verb, but it fails at the same place: the missing nodes/proxy. Row three is why list belongs in the role: a named node works while --role does not. So check with --subresource. Then confirm with oc adm node-logs itself, which is the only check that cannot be fooled by a misread argument.
That leaves one loose end, and the command’s own help text walks you straight into it by suggesting oc adm policy who-can --all-namespaces get nodes/log. So where does nodes/log come from? It is real, but it belongs to a different actor. It is the permission the kubelet requires of the API server’s identity when the API server proxies your request onward, checked by the kubelet through a SubjectAccessReview rather than by the API server against you. Ask the policy engine who holds it and system:kube-apiserver is in the list for exactly that reason:
oc adm policy who-can --all-namespaces get nodes/logA warning arrives ahead of the answer, and it is worth knowing what it does not mean:
Warning: the server doesn't have a resource type 'nodes/log'That is an artifact of how who-can resolves its argument, not a statement about the subresource. It never splits on the slash, so it warns for every resource/subresource form you hand it, including pods/log, which is the one oc logs uses every day:
oc adm policy who-can --all-namespaces get pods/log 2>&1 1>/dev/null
oc adm policy who-can --all-namespaces get nodes/proxy 2>&1 1>/dev/nullBoth warn, and both resources plainly exist. Ignore the line; the user list underneath it is still computed correctly. If you do want to know which node subresources the API server registers, ask discovery instead, and note that nodes/log is not among them:
oc get --raw /api/v1 | jq -r '.resources[].name' | grep '^nodes'Three, and the log one is absent:
nodes
nodes/proxy
nodes/statusExpect a short list of humans and a long list of platform service accounts, and the next section shows exactly what that gate is protecting.
Read log files under /var/log on a node
Swap --path in for the journal and the same command becomes a file browser rooted at /var/log on the node. Directories list, files print, and the path is relative, so --path=/ and --path=containers are both valid. A trailing slash makes no difference: containers and containers/ return the same listing, whatever its length on your node. The one form to watch is --path=journal/ with the slash. Plain --path=journal is the flag’s default and renders text as usual, but the trailing-slash version hands you the journal directory as a gzip stream, close to a megabyte of it on this node, so redirect that one to a file rather than to your terminal.
oc adm node-logs "${NODE}" --path=/Twenty-eight entries come back on an RHCOS node, and the mix of directories and bare files is the first hint that this is a plain filesystem listing rather than a curated log index:
Half of that listing is misleading. glusterfs/, sssd/, chrony/, etcd/ and bootstrap-control-plane/ were all empty on the test node, and samba/ contained nothing but an old/ subdirectory. They are directories shipped by RHEL base packages, not evidence that Samba is running on your Kubernetes node. The directories that hold real data are the container ones and the API server ones.
There are five separate audit logs, and knowing which is which saves a lot of grepping:
| Path | What it records |
|---|---|
audit/audit.log | Linux auditd on the node itself, including SELinux denials |
kube-apiserver/audit.log | Every request to the Kubernetes API |
openshift-apiserver/audit.log | Requests to OpenShift-specific APIs (routes, builds, projects) |
oauth-apiserver/audit.log | Token and identity API activity |
oauth-server/audit.log | Login attempts against the OAuth server |
These are large and they are sensitive. One JSON object per line, carrying user, sourceIPs, userAgent, verb, requestURI and responseStatus fields, and growing at one to two thousand lines a minute across samples on a single-node cluster with nobody using it. That combination of volume and content is the real reason the command is gated behind node admin. The audit policy reference covers the profiles and how much each one records. Pipe the file through jq rather than reading it raw:
oc adm node-logs "${NODE}" --path=kube-apiserver/audit.log | jq -r 'select(.verb=="delete") | [.requestReceivedTimestamp, .user.username, .objectRef.resource] | @tsv'The kube-apiserver/ directory also holds termination.log, which is the first place to look when an API server pod restarted and you want to know why it went down. It also holds rotated audit files with timestamps in their names, several a day on a busy cluster, so an event from an hour ago may well have moved out of audit.log into a sibling. List the directory before you grep it.
Container logs are reachable here too. The kubelet writes them under pods/, one directory per pod named <namespace>_<pod>_<uid>, and the OVN network stack keeps its own files:
oc adm node-logs "${NODE}" --path=pods
oc adm node-logs "${NODE}" --path=ovn/acl-audit-log.log
oc adm node-logs "${NODE}" --path=ovn-kubernetes/ovn-k8s-cni-overlay.logThat acl-audit-log.log file is where NetworkPolicy denials land once you turn ACL logging on, which makes it the counterpart to the pod and container metrics view when traffic is disappearing and nothing in the application log explains it.
Filter node logs by unit, time, and pattern
The journal-mode flags map onto journalctl concepts without being journalctl, and the differences bite. Time windows accept a strict grammar:
oc adm node-logs "${NODE}" -u crio --since=-1h
oc adm node-logs "${NODE}" -u crio --since="2026-08-20 00:00:00" --until="2026-08-20 00:15:00"Output format is where the tool is quietly more capable than its own documentation. The help text advertises four values for -o. Feed it an invalid one and the validator names seven, plus the empty default, and prints the whole complaint twice. The doubling is specific to the API field validators; the client-side range check on --boot further down prints once:
oc adm node-logs "${NODE}" -u crio -o yaml --tail=1The rejection names the full set, in an order that shuffles between runs:
error: output: Unsupported value: "yaml": supported values: "cat", "", "short-precise", "json", "short", "short-unix", "short-iso", "short-iso-precise"
output: Unsupported value: "yaml": supported values: "cat", "", "short-precise", "json", "short", "short-unix", "short-iso", "short-iso-precise"short-iso, short-iso-precise and short-precise are all absent from --help and all work. The ISO variants are the ones you want when the log is going into a spreadsheet or another parser:
oc adm node-logs "${NODE}" -u crio -o short-iso-precise --tail=1An ISO 8601 timestamp with microseconds replaces the syslog-style date:
2026-08-20T00:03:14.865454+0000 crc crio[4472]: time="2026-08-20T00:03:14.865395212Z" level=info msg="Removed pod sandbox: ed1f97bfe3c009c02990c95d8cb5cc54e186dffe0b97bc05473962564f44c4ce" id=5909a9f0-a0ed-4918-b1e2-6cad82ecdcd4 name=/runtime.v1.RuntimeService/RemovePodSandbox-o json returns the full journal record instead of a rendered line, which is the version to use when you care about metadata the text form throws away. Fields include _SYSTEMD_UNIT, _PID, PRIORITY, _SELINUX_CONTEXT and _BOOT_ID.
Resist the obvious next step, which is filtering on PRIORITY to find errors. CRI-O and the kubelet write to stdout, so journald stamps every record they produce at the same severity:
oc adm node-logs "${NODE}" -u kubelet -o json --since=-1h | jq -r .PRIORITY | sort | uniq -cOne bucket:
1005 6Informational and fatal lines alike come back as priority 6, so a severity filter on this unit matches everything or nothing. The real severity is inside the message, in the klog level prefix that Kubernetes components emit: I for info, W for warning, E for error. Counting those over the same window finds what a priority filter cannot:
oc adm node-logs "${NODE}" -u kubelet --since=-1h | grep -cE '\]: E[0-9]{4} 'Two hundred and eighty-two error lines in that hour, on a single-node cluster whose only workload is the crashlooping pod from earlier, every one of them invisible to a PRIORITY filter. Your counts will be different; the single priority bucket is the part that holds:
282So grep the klog prefix for these units, and keep PRIORITY for units that genuinely use syslog severities.
Boot offsets work the way they do in journalctl, with 0 as the current boot and negative numbers walking backwards through the allowed range of -100 to 0. Asking for a boot the node never had is not an error, which is worth knowing before you script around it:
oc adm node-logs "${NODE}" --boot=-5 --tail=1
echo "exit=$?"The message goes to stdout and the exit status is 0, so it belongs to the same silent-success family as the flag traps below. The distinction that matters is valid-but-absent versus out-of-range. --help warns that “passing invalid boot offset will fail retrieving logs”, and an out-of-range value does fail loudly, but an offset inside the allowed range that the node simply never had exits 0:
Data from the specified boot (-5) is not available: No such boot ID in journal
exit=0An out-of-range offset behaves completely differently. That one is a real error on stderr with a non-zero status, so the two failure modes are not interchangeable:
oc adm node-logs "${NODE}" --boot=-101 --tail=1The range check fires client-side before anything is sent:
error: --boot accepts values [-100, 0]And --tail has a hard ceiling that it tells you about plainly:
oc adm node-logs "${NODE}" -u kubelet --tail=200000The limit is stated in the error itself:
error: tailLines: Invalid value: 200000: must be between 0 and 100000, inclusive
tailLines: Invalid value: 200000: must be between 0 and 100000, inclusiveError: “since: Invalid value … date must be a relative time”
journalctl happily accepts “2 minutes ago”. This command does not, and the error is generous enough to hand you the entire accepted grammar:
oc adm node-logs "${NODE}" -u kubelet --since="2 minutes ago"The parser refuses it and prints the grammar it does accept:
error: since: Invalid value: "2 minutes ago": date must be a relative time of the form '(+|-)[0-9]+(s|m|h|d)' or a date in 'YYYY-MM-DD HH:MM:SS' form
since: Invalid value: "2 minutes ago": date must be a relative time of the form '(+|-)[0-9]+(s|m|h|d)' or a date in 'YYYY-MM-DD HH:MM:SS' formSo -2m, -90s, -4h and -7d are all fine, a full timestamp is fine, and a bare 2m without the sign is not.
Why oc adm node-logs returned nothing
Four flag combinations here fail without telling you. Three return an empty or near-empty result and exit 0, and one of them is worse still: it returns a confident but wrong slice of the log. None of them prints a warning, and between them they account for most of the “the command does not work” reports.
A –role that does not exist is silent
Plenty of older guides tell you to run --role infra alongside master and worker. Infra nodes are a convention, not a default, and most clusters have no node carrying that label. Asking for logs from a role nobody has produces no output, no error and a zero exit status:
oc adm node-logs --role infra --path=/
echo "exit status: $?"The result reads exactly like a node with no logs:
exit status: 0A typo behaves identically, so --role wroker will never tell you it found nothing to talk to. Confirm the label exists before trusting an empty answer:
oc get nodes -l node-role.kubernetes.io/infra=–tail and –grep do not compose
This is the one that wastes the most time, because the command looks reasonable and the result looks like an answer. --grep pattern --tail=5 does not mean "the last five matching lines". Three counts settle it:
oc adm node-logs "${NODE}" -u crio --grep="RemoveContainer" | wc -l
oc adm node-logs "${NODE}" -u crio --grep="RemoveContainer" --tail=500 | wc -l
oc adm node-logs "${NODE}" -u crio --tail=500 | grep -c RemoveContainerThe pattern matches 192 lines on its own. Asking for the last 500 does not return 192 of them, and it does not return 500 either. It returns 64:
192
64
64Sixty-four is exactly what you get by piping --tail=500 into a local grep, which is the whole story: the filter runs inside the window rather than selecting it.
Use a small --tail and the same trap shows up in a starker form. If none of the last few entries happen to match, you get journalctl’s placeholder rather than a count, which is easy to misread as a clean result. Note that wc -l counts that placeholder as one line, so a naive count reports 1 where the honest answer is 0:
oc adm node-logs "${NODE}" -u crio --grep="RemoveContainer" --tail=5On a quiet node, most of the time:
-- No entries --Side by side, the two behaviours look like this:
Absolute counts depend on how long the node has been up and how busy it is, so your numbers will differ from these. The equality between the second and third command is the part that reproduces, and it reproduces most reliably at a large --tail, because a small window often contains no matches at all.
Adding --since changes the behaviour, but not into the behaviour you want:
oc adm node-logs "${NODE}" -u crio --grep="RemoveContainer" --since=-40m > /tmp/full.txt
oc adm node-logs "${NODE}" -u crio --grep="RemoveContainer" --since=-40m --tail=5 > /tmp/t5.txt
diff <(head -5 /tmp/full.txt) /tmp/t5.txt && echo "tail=5 returned the OLDEST five"Several dozen matches in the window, and --tail=5 does return five real matches this time. They are the wrong five. The diff against the first five lines is empty:
tail=5 returned the OLDEST fiveThe five entries ran from 00:46:33 to 00:55:57 while the window itself ended at 01:22:48, so everything from the last twenty-five minutes was thrown away without a word. That is the opposite of what a flag called --tail implies. It is also more dangerous than the first case, because the first case at least looks broken: an empty result makes you check your command. Here you get five plausible lines, correctly formatted, with no hint that every newer match was dropped. If you were checking whether a problem is still happening, this answers no when the truth is yes.
The conclusion is blunt. Do not combine --tail with --grep on this command in either form. Bound the window server-side and take the tail locally, where the ordering is yours:
oc adm node-logs "${NODE}" -u crio --grep="RemoveContainer" --since=-40m | tail -5One caveat on how far to carry this. The node under test runs systemd 252, and the ordering here comes from the journal seek that --since and --tail are translated into, so it is a property of that systemd rather than of oc. The seek order changed upstream in systemd 254, which means a node on a newer base can behave differently again. Neither combination is worth relying on. Filter on the server, tail on your laptop, and the version underneath stops mattering.
–grep is case-sensitive by default
The default is case-sensitive matching, which is the opposite of what most people assume from a debugging flag. The kubelet logs a line reading "Finished parsing log file" a hundred or more times an hour, and a lowercase pattern finds none of them:
oc adm node-logs "${NODE}" -u kubelet --grep="finished parsing" --since=-2hNothing matched, and the wording invites you to conclude nothing happened:
-- No entries --Turn case sensitivity off and the same pattern over the same window returns hundreds of lines. The exact count moves with the node; nothing versus something is the point:
oc adm node-logs "${NODE}" -u kubelet --grep="finished parsing" --case-sensitive=false --since=-2hThe first of them:
Aug 19 23:55:36.786634 crc kubenswrapper[4519]: I0819 23:55:36.786528 4519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-apiserver_kube-apiserver-crc_9d47f775ac944f35baf67b2de3b24351/kube-apiserver-check-endpoints/0.log"Note the --since in both commands. Swap it for --tail and neither version returns anything, for the reason described just above, which makes the case-sensitivity bug impossible to diagnose while --tail is in the way.
A misspelled unit name reports success
A unit name that does not exist does not say so. It produces a message that reads like an internal assertion, with the bogus unit name repeated back at you, and exits 0:
oc adm node-logs "${NODE}" -u nosuchunit.service --tail=2The reply opens with a blank line and then reads like an internal assertion rather than a missing unit:
options present and query resolved to log files for [nosuchunit.service nosuchunit.service]
try without specifying optionsThat is what a typo in -u looks like. A wrong node name, by contrast, fails cleanly with error: nodes "no-such-node" not found, so the node argument is the one place the command validates properly.
Collect the whole cluster with oc adm must-gather
When Red Hat support asks for data, this is the command. It schedules a pod that walks the cluster and writes everything into a directory under your current working directory, so run it somewhere with room:
mkdir -p ~/must-gather && cd ~/must-gather
oc adm must-gatherOn the single-node test cluster that finished in 1 minute 8 seconds and produced 72 MB across 2,746 files. Scale that by node count and namespace count to estimate a real cluster. The gather pod also competes for CPU and memory while it runs, on a cluster that is by definition already unhealthy if you are reaching for this command. Start it, then go and look at something else while it works.
The output lands in a must-gather.local.<random-digits> directory in your current directory, and inside that sits a second directory whose name encodes the gather image digest. That nesting looks alarming and is normal, though --dest-dir is worth using if you would rather choose the location. One level down you get namespaces/, nodes/, host_service_logs/, cluster-scoped-resources/, etcd_info/, network_logs/, monitoring/ and static-pods/, plus an event-filter.html file that is a genuinely useful offline event browser.
Here is the part that surprises people, and it is worth checking before you promise a support engineer that the archive contains your problem. must-gather is a platform dump, not an application dump. The run above collected 59 namespaces: 54 matching openshift-*, one named plainly openshift, and the four system namespaces default, kube-system, kube-node-lease and hostpath-provisioner. The application namespace holding the crashing pod was not among them.
ls must-gather.local.*/*/namespaces/ | grep -v '^openshift'Only the four system namespaces answer:
default
hostpath-provisioner
kube-node-lease
kube-systemFor the namespaces it does collect, the layout is worth learning because it solves the reaped-container problem from earlier. Each container gets current.log and previous.log, and usually a previous.insecure.log alongside them, so a platform pod’s prior crash is preserved in the archive even after CRI-O has removed the container. The files are zero bytes when there was no previous container, which is a real answer rather than a missing one.
find must-gather.local.*/*/namespaces/openshift-dns -name 'previous.log' | head -3One more gap worth knowing before you hand the archive over: the audit logs from the table earlier are not in it. Red Hat leaves them out of the default set to keep the size down. An audit trail therefore needs its own run, with a gather script the image already carries. Searching a default archive for them comes back empty, which is a bad thing to discover after the support case is open.
oc adm must-gather -- /usr/bin/gather_audit_logsThat writes the same directory layout with the API audit files included.
Node service journals are collected separately under host_service_logs/, bucketed by control plane role. On the test cluster that directory held only masters/, even though the node also carries the worker label, with eleven service logs in it: CRI-O, the kubelet, both machine-config-daemon variants, NetworkManager, four Open vSwitch services (openvswitch, ovs-configuration, ovsdb-server and ovs-vswitchd), ostree-finalize-staged and rpm-ostreed.
To scope a run down, most operators ship their own gather image, which cuts both time and size dramatically compared with the default. Check the cluster version and operator health first with the cluster version and operator status checks so you know which component to target, then pass that operator’s image instead of collecting everything.
Where oc logs stops being enough
Everything above reads logs that still exist, and the retention limits are tighter than most people expect.
Container logs are rotated by the kubelet, not by CRI-O. The log_size_max setting in /etc/crio/crio.conf is commented out and effectively unlimited, so the real cap is the kubelet’s containerLogMaxSize. OpenShift sets that to 50Mi, five times the upstream Kubernetes default of 10Mi. The companion setting containerLogMaxFiles is left unset, so it takes the upstream default of five, and the kubelet rotates through that many files. The catch is that oc logs only ever serves the current one, so 50Mi is the effective ceiling on what you can read back through the API no matter how many rotated files sit on disk.
The journal is bounded too, and far less tightly. Stock RHCOS sets no SystemMaxUse, so unless a MachineConfig says otherwise journald falls back to the systemd default of ten percent of the filesystem capped at 4 GB, and it honours SystemKeepFree alongside that, using whichever limit is smaller.
Rather than deriving that from a disk size, ask the node what it settled on. journald announces it at boot, so the same unit filter you have been using all along will find it:
oc adm node-logs "${NODE}" -u systemd-journald --grep="System Journal"Usage and ceiling in one line:
Aug 19 23:54:55.452126 crc systemd-journald[889]: System Journal (/var/log/journal/fe3a98f305fa4c4b8237f53dffee1d8c) is 22.8M, max 3.0G, 3.0G free.Gigabytes, in other words, against a container log that stops at 50Mi. The container log is the constraint that will actually bite you.
Add a pod restart on top of a rotation and the evidence is simply gone. That is the gap log aggregation fills, and it is the only reason to run it. Shipping to an in-cluster store is what the logging operator does; for an existing SIEM, forwarding OpenShift logs and events to Splunk keeps the retention policy somewhere you already audit.
Two smaller ceilings are worth knowing before you hit them. oc logs reads one namespace at a time, so a problem that spans namespaces means one invocation per namespace or a shell loop. And oc adm node-logs has no follow mode at all, so watching a node service live always means a debug pod on the node. If you came here from Kubernetes and the flags feel familiar, they are: the kubectl reference covers the shared subset, and the pieces that differ are the ones with adm or an OpenShift resource type in them. To practise any of this without touching production, a local OpenShift cluster with CRC reproduces every command in this guide.
