게시물 1,028건
   
[Ceph] Slow Requests, and Requests are Blocked
글쓴이 : 최고관리자 날짜 : 2018-07-30 (월) 10:34 조회 : 11973
글주소 :
                                
하드웨어 체크
 - 스위치
 - 네트워크 카드
 - RAM/CPU 사용율
 - %iowait 지연


원문
https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/2/html/troubleshooting_guide/troubleshooting-osds#slow-requests-and-requests-are-blocked



5.1.5. Slow Requests, and Requests are Blocked

The ceph-osd daemon is slow to respond to a request and the ceph health detail command returns an error message similar to the following one:

HEALTH_WARN 30 requests are blocked > 32 sec; 3 osds have slow requests
30 ops are blocked > 268435 sec
1 ops are blocked > 268435 sec on osd.11
1 ops are blocked > 268435 sec on osd.18
28 ops are blocked > 268435 sec on osd.39
3 osds have slow requests

In addition, the Ceph logs include an error message similar to the following ones:

2015-08-24 13:18:10.024659 osd.1 127.0.0.1:6812/3032 9 : cluster [WRN] 6 slow requests, 6 included below; oldest blocked for > 61.758455 secs
2016-07-25 03:44:06.510583 osd.50 [WRN] slow request 30.005692 seconds old, received at {date-time}: osd_op(client.4240.0:8 benchmark_data_ceph-1_39426_object7 [write 0~4194304] 0.69848840) v4 currently waiting for subops from [610]
What This Means

An OSD with slow requests is every OSD that is not able to service the I/O operations per second (IOPS) in the queue within the time defined by the osd_op_complaint_time parameter. By default, this parameter is set to 30 seconds.

The main causes of OSDs having slow requests are:

  • Problems with the underlying hardware, such as disk drives, hosts, racks, or network switches
  • Problems with network. These problems are usually connected with flapping OSDs. See Section 5.1.4, “Flapping OSDs” for details.
  • System load

The following table shows the types of slow requests. Use the dump_historic_opsadministration socket command to determine the type of a slow request. For details about the administration socket, see the Using the Administration Socket section in the Administration Guide for Red Hat Ceph Storage 2.

Slow request typeDescription

waiting for rw locks

The OSD is waiting to acquire a lock on a placement group for the operation.

waiting for subops

The OSD is waiting for replica OSDs to apply the operation to the journal.

no flag points reached

The OSD did not reach any major operation milestone.

waiting for degraded object

The OSDs have not replicated an object the specified number of times yet.

To Troubleshoot This Problem
  1. Determine if the OSDs with slow or block requests share a common piece of hardware, for example a disk drive, host, rack, or network switch.
  2. If the OSDs share a disk:

    1. Use the smartmontools utility to check the health of the disk or the logs to determine any errors on the disk.

      NOTE

      The smartmontools utility is included in the smartmontools package.

    2. Use the iostat utility to get the I/O wait report (%iowai) on the OSD disk to determine if the disk is under heavy load.

      NOTE

      The iostat utility is included in the sysstat package.

  3. If the OSDs share a host:

    1. Check the RAM and CPU utilization
    2. Use the netstat utility to see the network statistics on the Network Interface Controllers (NICs) and troubleshoot any networking issues. See also Chapter 3, Troubleshooting Networking Issues for further information.
  4. If the OSDs share a rack, check the network switch for the rack. For example, if you use jumbo frames, verify that the NIC in the path has jumbo frames set.
  5. If you are unable to determine a common piece of hardware shared by OSDs with slow requests, or to troubleshoot and fix hardware and networking problems, open a support ticket. See Chapter 7, Contacting Red Hat Support Service for details.

이름 패스워드
비밀글 (체크하면 글쓴이만 내용을 확인할 수 있습니다.)
왼쪽의 글자를 입력하세요.
   

 



 
사이트명 : 모지리네 | 대표 : 이경현 | 개인커뮤니티 : 랭키닷컴 운영체제(OS) | 경기도 성남시 분당구 | 전자우편 : mojily골뱅이chonnom.com Copyright ⓒ www.chonnom.com www.kyunghyun.net www.mojily.net. All rights reserved.