Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Symptoms


SoftNAS Appliance in a frozen or hung state during normal operation due mostly to high CPU and MEMORY utilization

 Purpose

In this KB we would discuss the various ways you can recover a hung SoftNAS instance without a reboot. However, to effectively recovery the SoftNAS instance without a reboot one must know exactly what is the culprit of the issue to begin with.

...

  1.  MEMORY (RAM) :- A system can go out of memory or even swap if there are a lot of read/write operations going on at the same time requiring extra CPU and RAM to process
  2. CPU :- As we know the CPU is the heart of the system, when it is over utilized it can cause so many problems that are but not limited to system lockups, Slow response time or even system crash etc.
  3. PROCESSES: Sometimes Linux processes can go rogue, consuming a noticeable amount of resources hence staving other processes to function properly.

Resolution

 Launch an ssh session (PUTTY or your favorite Client)

...

  1. Now that we've identified the processes that are taking up most of the resources it is time to get rid of some of them to bring down the system load. You can use the command below to kill some of the Protocol services (CIFS and NFS)
    # kill <process id>
    # kill -9 <process id>

  2. Then you can restart the NAS services
    /var/www/softnas/scripts/start-nasservices.sh

  3. Another very important thing to look for would be to check if you are not running out of file descriptors (FDs). Running out of FDs will render your system unusable
    grep max /var/log/messagesYou should get something like the output below

    Feb 19 06:42:18 Localhost kernel: VFS: file-max limit 1000000 reached

    Feb 19 06:42:18 Localhost kernel: VFS: file-max limit 1000000 reached

    Feb 19 06:42:18 Localhost kernel: VFS: file-max limit 1000000 reached

    Feb 19 06:42:18 Localhost kernel: VFS: file-max limit 1000000 reached

    Feb 19 06:42:18 Localhost kernel: VFS: file-max limit 1000000 reached

    Feb 19 06:42:18 Localhost kernel: VFS: file-max limit 1000000 reached

    Feb 19 06:42:18 Localhost kernel: VFS: file-max limit 1000000 reached

    Feb 19 06:42:18 Localhost kernel: VFS: file-max limit 1000000 reached

    Feb 19 06:42:18 Localhost kernel: VFS: file-max limit 1000000 reached

    Feb 19 06:42:18 Localhost kernel: VFS: file-max limit 1000000 reached

  4. If you get any output above after running the command in step #4 then it is time to increase your file descriptors. Below are the commands to increase file descriptors

    sysctl -w fs.file-max=1000000

    then verify by doing 'cat /proc/sys/fs/file-max'

    To make it permanent, add this to /etc/sysctl.conf

    Edit this line in /etc/sysctl.conf fs.file-max = 500000 to fs.file-max = 1000000 for example

    End result: fs.file-max=1000000

    # sysctl -p :- To commit the change without a reboot

Additional Information

You can read a great deal of performance related problems that can affect the overall functionality of SoftNAS here


Update History


 

 KB Created 
  
  

...