When accessing daemon / process to certain files on the disk, in this case - /www/site/data/www/site.ua/project/www.site.ua/files/, is sticking this process, kill <pid> not help . It seems like zfs should identify bad blocks, but this does not happen. #ls /www/site/data/www/site.ua/project/www.site.ua/files and the session hangs... # lsof | grep /www/site/data/www/site.ua/project/www.site.ua/ httpd 2720 www txt VREG 146,2806775880 18446742975789345120 1148769 /www/site/data/www/site.ua/project/www.site.ua/files/catalog.files/100-516070727.jpg httpd 2720 www 57r VREG 146,2806775880 18446742975789345120 1148769 /www/site/data/www/site.ua/project/www.site.ua/files/catalog.files/100-516070727.jpg mc 3067 root cwd VDIR 146,2806775880 18446742975782687200 1148593 /www/site/data/www/site.ua/project/www.site.ua/files httpd 3068 www txt VREG 146,2806775880 18446742975782685024 1148707 /www/site/data/www/site.ua/project/www.site.ua/files/catalog.files/100-2d0e4312d8f09c72ede95fccdc012b02.jpg httpd 3068 www 57r VREG 146,2806775880 18446742975782685024 1148707 /www/site/data/www/site.ua/project/www.site.ua/files/catalog.files/100-2d0e4312d8f09c72ede95fccdc012b02.jpg csh 3069 root cwd VDIR 146,2806775880 18446742975782687200 1148593 /www/site/data/www/site.ua/project/www.site.ua/files mc 8807 root cwd VDIR 146,2806775880 18446742975782622624 1148609 /www/site/data/www/site.ua/project/www.site.ua/files/catalog.files csh 8808 root cwd VDIR 146,2806775880 18446742975782622624 1148609 /www/site/data/www/site.ua/project/www.site.ua/files/catalog.files It does not help: # kill -9 2720 3067 3068 3069 Only helps to reboot and reset the system unit. # df -h Filesystem Size Used Avail Capacity Mounted on tank 1.0T 808M 1.0T 0% / devfs 1.0K 1.0K 0B 100% /dev tank/backup 1.2T 170G 1.0T 14% /backup tank/tmp 1.0T 26M 1.0T 0% /tmp tank/usr 1.0T 3.4G 1.0T 0% /usr tank/usr/home 1.0T 30K 1.0T 0% /usr/home tank/usr/ports 1.0T 2.2G 1.0T 0% /usr/ports tank/usr/src 1.0T 333M 1.0T 0% /usr/src tank/var 1.0T 1.8M 1.0T 0% /var tank/var/crash 1.0T 22K 1.0T 0% /var/crash tank/var/db 1.0T 61M 1.0T 0% /var/db tank/mysql 1.0T 294M 1.0T 0% /var/db/mysql tank/mysql/ibdata 1.0T 3.2G 1.0T 0% /var/db/mysql/ibdata tank/mysql/iblogs 1.0T 10M 1.0T 0% /var/db/mysql/iblogs tank/var/log 1.0T 100M 1.0T 0% /var/log tank/var/mail 1.0T 538K 1.0T 0% /var/mail tank/var/run 1.0T 83K 1.0T 0% /var/run tank/var/tmp 1.0T 13M 1.0T 0% /var/tmp tank/www 1.1T 21G 1.0T 2% /www devfs 1.0K 1.0K 0B 100% /var/named/dev ZFS filesystem version 4 ZFS pool version 15 tank version=15 name='tank' state=0 txg=2790 pool_guid=15415411259239146062 hostid=814717323 hostname='second.site.com' vdev_tree type='root' id=0 guid=15415411259239146062 children[0] type='mirror' id=0 guid=16020562126957161505 whole_disk=0 metaslab_array=23 metaslab_shift=33 ashift=9 asize=1483117166592 is_log=0 children[0] type='disk' id=0 guid=11217068100198816386 path='/dev/gpt/disk0' whole_disk=0 DTL=120 children[1] type='disk' id=1 guid=4665162630340381592 path='/dev/gpt/disk1' whole_disk=0 DTL=118 # gpart show => 34 2930277101 ad4 GPT (1.4T) 34 128 1 freebsd-boot (64K) 162 33554432 2 freebsd-swap (16G) 33554594 2896722541 3 freebsd-zfs (1.3T) => 34 2930277101 ad8 GPT (1.4T) 34 128 1 freebsd-boot (64K) 162 33554432 2 freebsd-swap (16G) 33554594 2896722541 3 freebsd-zfs (1.3T) # smartctl -a /dev/ad4 smartctl 5.39.1 2010-01-28 r3054 [FreeBSD 9.0-CURRENT amd64] (local build) Copyright (C) 2002-10 by Bruce Allen, http://smartmontools.sourceforge.net === START OF INFORMATION SECTION === Model Family: Seagate Barracuda 7200.11 family Device Model: ST31500341AS Serial Number: 9VS2B140 Firmware Version: CC1H User Capacity: 1 500 301 910 016 bytes Device is: In smartctl database [for details use: -P show] ATA Version is: 8 ATA Standard is: ATA-8-ACS revision 4 Local Time is: Tue Dec 21 21:01:28 2010 EET SMART support is: Available - device has SMART capability. SMART support is: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED General SMART Values: Offline data collection status: (0x82) Offline data collection activity was completed without error. Auto Offline Data Collection: Enabled. Self-test execution status: ( 0) The previous self-test routine completed without error or no self-test has ever been run. Total time to complete Offline data collection: ( 609) seconds. Offline data collection capabilities: (0x7b) SMART execute Offline immediate. Auto Offline data collection on/off support. Suspend Offline collection upon new command. Offline surface scan supported. Self-test supported. Conveyance Self-test supported. Selective Self-test supported. SMART capabilities: (0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability: (0x01) Error logging supported. General Purpose Logging supported. Short self-test routine recommended polling time: ( 1) minutes. Extended self-test routine recommended polling time: ( 255) minutes. Conveyance self-test routine recommended polling time: ( 2) minutes. SCT capabilities: (0x103f) SCT Status supported. SCT Feature Control supported. SCT Data Table supported. SMART Attributes Data Structure revision number: 10 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x000f 115 099 006 Pre-fail Always - 93576265 3 Spin_Up_Time 0x0003 100 099 000 Pre-fail Always - 0 4 Start_Stop_Count 0x0032 100 100 020 Old_age Always - 44 5 Reallocated_Sector_Ct 0x0033 100 100 036 Pre-fail Always - 0 7 Seek_Error_Rate 0x000f 084 060 030 Pre-fail Always - 274870085 9 Power_On_Hours 0x0032 091 091 000 Old_age Always - 8230 10 Spin_Retry_Count 0x0013 100 100 097 Pre-fail Always - 0 12 Power_Cycle_Count 0x0032 100 037 020 Old_age Always - 60 184 End-to-End_Error 0x0032 100 100 099 Old_age Always - 0 187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0 188 Command_Timeout 0x0032 100 090 000 Old_age Always - 50 189 High_Fly_Writes 0x003a 073 073 000 Old_age Always - 27 190 Airflow_Temperature_Cel 0x0022 070 046 045 Old_age Always - 30 (Lifetime Min/Max 28/44) 194 Temperature_Celsius 0x0022 030 054 000 Old_age Always - 30 (0 20 0 0) 195 Hardware_ECC_Recovered 0x001a 049 017 000 Old_age Always - 93576265 197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 0 198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 0 240 Head_Flying_Hours 0x0000 100 253 000 Old_age Offline - 4359391810201 241 Total_LBAs_Written 0x0000 100 253 000 Old_age Offline - 367084491 242 Total_LBAs_Read 0x0000 100 253 000 Old_age Offline - 1582503175 SMART Error Log Version: 1 No Errors Logged SMART Self-test log structure revision number 1 No self-tests have been logged. [To run self-tests, use: smartctl -t] SMART Selective self-test log data structure revision number 1 SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS 1 0 0 Not_testing 2 0 0 Not_testing 3 0 0 Not_testing 4 0 0 Not_testing 5 0 0 Not_testing Selective self-test flags (0x0): After scanning selected spans, do NOT read-scan remainder of disk. If Selective self-test is pending on power-up, resume after 0 minute delay. # smartctl -a /dev/ad8 smartctl 5.39.1 2010-01-28 r3054 [FreeBSD 9.0-CURRENT amd64] (local build) Copyright (C) 2002-10 by Bruce Allen, http://smartmontools.sourceforge.net === START OF INFORMATION SECTION === Model Family: Seagate Barracuda 7200.11 family Device Model: ST31500341AS Serial Number: 9VS2B0HA Firmware Version: CC1H User Capacity: 1 500 301 910 016 bytes Device is: In smartctl database [for details use: -P show] ATA Version is: 8 ATA Standard is: ATA-8-ACS revision 4 Local Time is: Tue Dec 21 21:01:07 2010 EET SMART support is: Available - device has SMART capability. SMART support is: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED General SMART Values: Offline data collection status: (0x82) Offline data collection activity was completed without error. Auto Offline Data Collection: Enabled. Self-test execution status: ( 0) The previous self-test routine completed without error or no self-test has ever been run. Total time to complete Offline data collection: ( 617) seconds. Offline data collection capabilities: (0x7b) SMART execute Offline immediate. Auto Offline data collection on/off support. Suspend Offline collection upon new command. Offline surface scan supported. Self-test supported. Conveyance Self-test supported. Selective Self-test supported. SMART capabilities: (0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability: (0x01) Error logging supported. General Purpose Logging supported. Short self-test routine recommended polling time: ( 1) minutes. Extended self-test routine recommended polling time: ( 255) minutes. Conveyance self-test routine recommended polling time: ( 2) minutes. SCT capabilities: (0x103f) SCT Status supported. SCT Feature Control supported. SCT Data Table supported. SMART Attributes Data Structure revision number: 10 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x000f 112 099 006 Pre-fail Always - 44718699 3 Spin_Up_Time 0x0003 100 100 000 Pre-fail Always - 0 4 Start_Stop_Count 0x0032 100 100 020 Old_age Always - 41 5 Reallocated_Sector_Ct 0x0033 100 100 036 Pre-fail Always - 1 7 Seek_Error_Rate 0x000f 084 060 030 Pre-fail Always - 265723778 9 Power_On_Hours 0x0032 091 091 000 Old_age Always - 8197 10 Spin_Retry_Count 0x0013 100 100 097 Pre-fail Always - 0 12 Power_Cycle_Count 0x0032 100 037 020 Old_age Always - 58 184 End-to-End_Error 0x0032 100 100 099 Old_age Always - 0 187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0 188 Command_Timeout 0x0032 100 099 000 Old_age Always - 7 189 High_Fly_Writes 0x003a 080 080 000 Old_age Always - 20 190 Airflow_Temperature_Cel 0x0022 070 047 045 Old_age Always - 30 (Lifetime Min/Max 27/44) 194 Temperature_Celsius 0x0022 030 053 000 Old_age Always - 30 (0 19 0 0) 195 Hardware_ECC_Recovered 0x001a 053 014 000 Old_age Always - 44718699 197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 0 198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 0 240 Head_Flying_Hours 0x0000 100 253 000 Old_age Offline - 70901320127123 241 Total_LBAs_Written 0x0000 100 253 000 Old_age Offline - 3213332020 242 Total_LBAs_Read 0x0000 100 253 000 Old_age Offline - 1673372002 SMART Error Log Version: 1 No Errors Logged SMART Self-test log structure revision number 1 No self-tests have been logged. [To run self-tests, use: smartctl -t] SMART Selective self-test log data structure revision number 1 SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS 1 0 0 Not_testing 2 0 0 Not_testing 3 0 0 Not_testing 4 0 0 Not_testing 5 0 0 Not_testing Selective self-test flags (0x0): After scanning selected spans, do NOT read-scan remainder of disk. If Selective self-test is pending on power-up, resume after 0 minute delay.
Responsible Changed From-To: freebsd-amd64->freebsd-fs reclassify.
zfs snapshot tank/www@temp zfs rollback tank/www@temp also did not help. :( -- Vladislav V. Prodan VVP24-UANIC +38[067]4584408 +38[099]4060508 vlad11@jabber.ru
Now lock the other directory: ( how to stop this process? # ll /www/cargotek/data/logs/cargotek.ru.error.log.1 -rw-r----- 1 root cargotek 984039 7 Ñнв 21:23 /www/cargotek/data/logs/cargotek.ru.error.log.1 # ps -axww | grep ls 82467 ?? D 7:46,78 find -sx / /backup /usr /usr/home /var /var/db/mysql /var/db/mysql/ibdata /var/db/mysql/iblogs /www /dev/null -type f ( -perm -u+x -or -perm -g+x -or -perm -o+x ) ( -perm -u+s -or -perm -g+s ) -exec ls -liTd {} + 51731 2 D+ 0:00,00 ls -lA /www/cargotek/data 51736 9 D+ 0:00,00 ls /www/cargotek/data 52498 11 D+ 0:00,00 ls -lA /www/cargotek/data/ # procstat -kk 82467 52498 51731 51736 PID TID COMM TDNAME KSTACK 82467 100263 find - mi_switch+0x174 sleepq_wait+0x42 __lockmgr_args+0x73b vop_stdlock+0x39 VOP_LOCK1_APV+0x46 _vn_lock+0x47 vget+0x70 cache_lookup+0x4fe vfs_cache_lookup+0xc0 VOP_LOOKUP_APV+0x40 lookup+0x49a namei+0x53a vn_open_cred+0x3db kern_openat+0x181 syscallenter+0x2cf syscall+0x4b Xfast_syscall+0xe2 52498 100301 ls - mi_switch+0x174 sleepq_wait+0x42 __lockmgr_args+0x73b vop_stdlock+0x39 VOP_LOCK1_APV+0x46 _vn_lock+0x47 vget+0x70 cache_lookup+0x4fe vfs_cache_lookup+0xc0 VOP_LOOKUP_APV+0x40 lookup+0x49a namei+0x53a vn_open_cred+0x3db kern_openat+0x181 syscallenter+0x2cf syscall+0x4b Xfast_syscall+0xe2 51731 100307 ls - mi_switch+0x174 sleepq_wait+0x42 __lockmgr_args+0x73b vop_stdlock+0x39 VOP_LOCK1_APV+0x46 _vn_lock+0x47 vget+0x70 cache_lookup+0x4fe vfs_cache_lookup+0xc0 VOP_LOOKUP_APV+0x40 lookup+0x49a namei+0x53a vn_open_cred+0x3db kern_openat+0x181 syscallenter+0x2cf syscall+0x4b Xfast_syscall+0xe2 51736 100321 ls - mi_switch+0x174 sleepq_wait+0x42 __lockmgr_args+0x73b vop_stdlock+0x39 VOP_LOCK1_APV+0x46 _vn_lock+0x47 vget+0x70 cache_lookup+0x4fe vfs_cache_lookup+0xc0 VOP_LOOKUP_APV+0x40 lookup+0x49a namei+0x53a vn_open_cred+0x3db kern_openat+0x181 syscallenter+0x2cf syscall+0x4b Xfast_syscall+0xe2 -- Vladislav V. Prodan VVP24-UANIC +38[067]4584408 +38[099]4060508 vlad11@jabber.ru
For bugs matching the following criteria: Status: In Progress Changed: (is less than) 2014-06-01 Reset to default assignee and clear in-progress tags. Mail being skipped