Difference between revisions of "INFO: task blocked for more than 120 seconds."

From Helpful
Jump to: navigation, search
m (Created page with "{{stub}} Under heavy IO load on servers you may see something like: INFO: task nfsd:2252 blocked for more than 120 seconds. "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"...")
 
m
 
(28 intermediate revisions by the same user not shown)
Line 1: Line 1:
{{stub}}
 
 
  
 
Under heavy IO load on servers you may see something like:
 
Under heavy IO load on servers you may see something like:
 
  INFO: task nfsd:2252 blocked for more than 120 seconds.
 
  INFO: task nfsd:2252 blocked for more than 120 seconds.
 
  "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
 
  "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
...probably followed by a call trace that mentions your filesystem, and probably io_schedule and sync_buffer.
+
...probably followed by a call trace that mentions your filesystem, and probably io_schedule and sync_buffer.  
  
  
The code for this sits in <tt>hung_task.c</tt>, which is a kernel thread that detects tasks stuck in D state (basically meaning waiting for IO so long it hasn't been scheduled for 120 seconds (default)).
+
'''This message is not an error''', it's telling you that the named process has not been scheduled on the CPU-time ''at all'' for 120 seconds (default), because it was in [[uninterruptable sleep]] state. {{comment|(The code behind this message sits in <tt>hung_task.c</tt> and was added somewhere around <tt>2.6.30</tt>. This is a kernel thread that detects tasks that stays in the [[D state]] for a while)}}
The code is relatively new, added somewhere around <tt>2.6.30</tt>.
+
  
  
The message is purely informational {{comment|(unless you set sysctl_hung_task_panic, in which case your host is now panicked)}}, but still probably something you want to do something about.
+
120 real-world seconds is an eternity for the CPU and most programs.
  
 +
Not being scheduled for that long typically signals resource starvation, usually IO, specifically disk.
 +
Which means you usually don't want to silence that message,
 +
because you want to find out when and why this happened, and probably avoid it in the future.
  
Notes:
 
* if the mentioned process was ioniced as idle, this may be intended or at least expectable behaviour under load
 
  
* if not, this often means your IO system is slower than your IO use -- often specifically caused by overhead, such as that from head seeking
+
The stack trace can help diagnose what it was doing. {{comment|(which is not so informative of the ''reason'' -
 +
the named program is often the victim of another one misbehaving, though it is sometimes the culprit)}}
 +
 
 +
 
 +
Reasons include
 +
: the system is heavily swapping, or [[trashing]], due to memory allocation issues
 +
 
 +
: the underlying IO system is very slow for some reason
 +
:: I've seen mentions of this happening in VMs that share disks
 +
 
 +
: specific bugs (in kernel code, systemd) have caused this as a side effect
  
* tweaking the linux io scheduler for the device  may help  (See [[Computer hard drives#Drive_specifics]])
 
** if your load is fairly sequential, you may get some relief from using the <tt>noop</tt> io scheduler (instead of <tt>cfq</tt>
 
** if it's relatively random upping the queue depth may help
 
  
* if it happens nightly, it's probably some cron job, and load from something like [http://en.wikipedia.org/wiki/Locate_%28Unix%29 updatedb].
 
  
* if it happens on a fileserver, you may want to consider spreading to more fileservers, or using a parallel filesystem
 
  
* NFS seems to be a common culprit, probably because it's good at filling the writeback cache, something which implies blocking while writeback happens - which is likely to block various things related to the same filesystem. {{verify}}
 
  
  
 
<!--
 
<!--
Side note:
+
{{comment|(...though you can explicitly set <tt>sysctl_hung_task_panic</tt>, in which case your host is now panicked)}}
The D state is uninterruptible sleep (TASK_UNINTERRUPTIBLE), is a deeper sleep than interruptible (which allows signaling). Uninterruptible sleep is generally only used when signals cannot be handled (signal handling is userspace code) such as when a process is waiting on device IO.
+
 
-->
 
-->
 +
 +
 +
 +
 +
Notes:
 +
* if it happens constantly your IO system is slower than your IO use
 +
 +
* most likely to happen ''to'' a process that was [[ionice]]d into the idle class. Which means it's working as intended, becauseidle-class is meant as an extreme politeness thing. It just indicates something else is doing a bunch of IO right now (for at least 120 seconds)
 +
: e.g. [http://en.wikipedia.org/wiki/Locate_%28Unix%29 updatedb] (may be victim if it were ioniced, cause if not)
 +
 +
* if it happens only nightly, look at your cron jobs
 +
 +
* a [[trashing]] system can cause this, and then it's purely a side effect of one program using too much RAM
 +
 +
* being blocked by a desktop-class drive with bad sectors (because they retry for a long while)
 +
 +
 +
* NFS seems to be a common culprit, probably because it's good at filling the writeback cache, something which implies blocking while writeback happens - which is likely to block various things related to the same filesystem. {{verify}}
 +
 +
* if it happens on a fileserver, you may want to consider spreading to more fileservers, or using a parallel filesystem
 +
 +
 +
* tweaking the linux io scheduler for the device  may help  (See [[Computer_data_storage_-_General_%26_RAID_performance_tweaking#OS_scheduling]])
 +
: if your load is fairly sequential, you may get some relief from using the <tt>noop</tt> io scheduler (instead of <tt>cfq</tt>) though note that that disables [[ionice]])
 +
: if your load is relatively random, upping the queue depth may help
 +
 +
 +
 +
[[Category:Unices]]
 +
[[Category:Warnings and errors]]

Latest revision as of 16:27, 22 July 2021

Under heavy IO load on servers you may see something like:

INFO: task nfsd:2252 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

...probably followed by a call trace that mentions your filesystem, and probably io_schedule and sync_buffer.


This message is not an error, it's telling you that the named process has not been scheduled on the CPU-time at all for 120 seconds (default), because it was in uninterruptable sleep state. (The code behind this message sits in hung_task.c and was added somewhere around 2.6.30. This is a kernel thread that detects tasks that stays in the D state for a while)


120 real-world seconds is an eternity for the CPU and most programs.

Not being scheduled for that long typically signals resource starvation, usually IO, specifically disk. Which means you usually don't want to silence that message, because you want to find out when and why this happened, and probably avoid it in the future.


The stack trace can help diagnose what it was doing. (which is not so informative of the reason - the named program is often the victim of another one misbehaving, though it is sometimes the culprit)


Reasons include

the system is heavily swapping, or trashing, due to memory allocation issues
the underlying IO system is very slow for some reason
I've seen mentions of this happening in VMs that share disks
specific bugs (in kernel code, systemd) have caused this as a side effect






Notes:

  • if it happens constantly your IO system is slower than your IO use
  • most likely to happen to a process that was ioniced into the idle class. Which means it's working as intended, becauseidle-class is meant as an extreme politeness thing. It just indicates something else is doing a bunch of IO right now (for at least 120 seconds)
e.g. updatedb (may be victim if it were ioniced, cause if not)
  • if it happens only nightly, look at your cron jobs
  • a trashing system can cause this, and then it's purely a side effect of one program using too much RAM
  • being blocked by a desktop-class drive with bad sectors (because they retry for a long while)


  • NFS seems to be a common culprit, probably because it's good at filling the writeback cache, something which implies blocking while writeback happens - which is likely to block various things related to the same filesystem. (verify)
  • if it happens on a fileserver, you may want to consider spreading to more fileservers, or using a parallel filesystem


if your load is fairly sequential, you may get some relief from using the noop io scheduler (instead of cfq) though note that that disables ionice)
if your load is relatively random, upping the queue depth may help