NodeJS Socket.io : Many connections in state CLOSE_WAIT and FIN_WAIT2 without release

hi , I used ubuntu(12.04) + nodejs (v0.10.22) + socket.io (v0.9.14) to transmit messages.

There are ~300 simultaneous connections. After some hours (about 1 or 2 hours above, it doesn't show up immediately), some connections will persistent in the state CLOSE_WAIT or FIN_WAIT2.

And these un-dead connections grows linearly with time. The users will hard to connect socket server when the connections number reach the limit (Default 1024 - see also Linux TCP/IP tuning for scalability) , unless some connections released normally.

The following was socket service connections status, running about 3 hours.

netstat -anl | grep <PORT_OF_NODE_PROCESS> | awk '/^tcp/ {t[$NF]++}END{for(state in t){print state, t[state]} }'

FIN_WAIT2 23
LISTEN 1
CLOSE_WAIT 27
TIME_WAIT 12
ESTABLISHED 333
FIN_WAIT1 12

Probably Solutions

1. Touch js file in regular periods

Using Nodemon Package to run js file, when change the file's last modified time, nodemon will restart service, and release all previous un-dead connections (CLOSEWAIT or FINWAIT2)

2. Increase connections limit

sudo vim /etc/security/limits.conf

*        soft    nofile  1024
*        hard    nofile  2048
root        soft    nofile  4096
root        hard    nofile  8192
user1   soft    nofile  2048
user1   hard    nofile  2048

Try to let connections hard to reach limit.

3. Decrease keep-alive timeout

Let operation system to close connections automatically in the short time, but I'm not try it yet.

see also : Using TCP keepalive under Linux

Question

I found some probably solution to fix the problem. But the above solutions were not really solved the persistent connections with state CLOSE_WAIT or FIN_WAIT2 problem. I could find this is a result of server(CLOSE_WAIT) or clients (FIN_WAIT2) not correctly closing connections. I think socket.io will force-close these incorrectly connection after some timeout. But it seems like not work correctly.

I try to reappear the state CLOSE_WAIT or FIN_WAIT2 problem in my test environment. But it never show up these connection situation.

  1. After connect socket server and disconnect network
  2. Connect socket server for a long time

I found @njam ask related question before (Many stale connections in state CLOSE_WAIT and FIN_WAIT2), but still can't find the solution. Does anyone know how to solve this problem??

Thanks

References

  1. Many stale connections in state CLOSE_WAIT and FIN_WAIT2
  2. TIME_WAIT and its design implications for protocols and scalable client server systems
  3. Using TCP keepalive under Linux
  4. Linux TCP/IP tuning for scalability

    Read on

Laravel 4 Migrate介紹

PHP ARTISAN MIGRATE指令

在命令列輸入php artisan後,可以看到migrate指令

$ php artisan
migrate
  migrate:install    Create the migration repository
  migrate:make       Create a new migration file
  migrate:refresh    Reset and re-run all migrations
  migrate:reset      Rollback all database migrations
  migrate:rollback   Rollback the last database migration

migrate指令用途

指令 說明
migration 執行尚未執行的migration
migrate:install 在資料庫建立migration管理資料表
migrate:make 建立新的migration檔案
migrate:refresh 重設資料庫所有的migration,並重新執行所有的migration
migrate:reset 重設資料庫所有的migration
migrate:rollback 復原最後一筆migration紀錄
Read on

在Laravel 4使用artisan建立app key

清除檔案中的KEY

app/config/app.php檔案中你會找到一組key的位置,這個key是一組32個字元的key,我們可以自己輸入想要的key,或者使用Laravel 4提供的Artisan命令列去自動亂數產生一組key

使用ARTISAN指令產生KEY

在命令列輸入php artisan可以看到所有artisan支援的指令,你可以找到key:generate可以產生一組亂數的key。

$ php artisan
Laravel Framework version 4.0.0
Usage:
  [options] command [arguments]
Options:
  --help           -h Display this help message.
  --quiet          -q Do not output any message.
  --verbose        -v|vv|vvv Increase the verbosity of messages: 1 for normal output, 2 for                                                                  more verbose output and 3 for debug
  --version        -V Display this application version.
  --ansi              Force ANSI output.
  --no-ansi           Disable ANSI output.
  --no-interaction -n Do not ask any interactive question.
  --env               The environment the command should run under.
Available commands:
  clear-compiled     Remove the compiled class file
  down               Put the application into maintenance mode
  dump-autoload      Regenerate framework autoload files
  help               Displays help for a command
  list               Lists commands
  migrate            Run the database migrations
  optimize           Optimize the framework for better performance
  routes             List all registered routes
  serve              Serve the application on the PHP development server
  tinker             Interact with your application
  up                 Bring the application out of maintenance mode
  workbench          Create a new package workbench
asset
  asset:publish      Publish a package's assets to the public directory
auth
  auth:reminders     Create a migration for the password reminders table
cache
  cache:clear        Flush the application cache
command
  command:make       Create a new Artisan command
config
  config:publish     Publish a package's configuration to the application
controller
  controller:make    Create a new resourceful controller
db
  db:seed            Seed the database with records
key
  key:generate       Set the application key
migrate
  migrate:install    Create the migration repository
  migrate:make       Create a new migration file
  migrate:refresh    Reset and re-run all migrations
  migrate:reset      Rollback all database migrations
  migrate:rollback   Rollback the last database migration
queue
  queue:listen       Listen to a given queue
  queue:subscribe    Subscribe a URL to an Iron.io push queue
  queue:work         Process the next job on a queue
session
  session:table      Create a migration for the session database table

所以我們輸入php artisan key:generate即可產生我們需要的key了

php artisan key:generate
Application key [2wgjsIpDDIJvc4AgMV6VV1QybWznB6nY] set successfully.

同步發表於