Max File Descriptors set in config but not being acknowledged

We have just just upgraded our server to 1.4.1 and implemented some of the logging features. For some time now we have been running your suggested OS tuning settings.

“maxFileDescriptors”: 250000

After upgrading we noticed the max file descriptors was capped to 4096 in the log which is not inline with the config file or the settings on the server. It does not look like the config setting is being acknowledged which might be an issue with accounts with lots of small docs.

Log Entries:
2017-07-10T09:49:13.855Z ==== Couchbase Sync Gateway/1.4.1(3;ddffd85) ====
2017-07-10T09:49:13.855Z requestedSoftFDLimit >= currentHardFdLimit (250000 >= 4096) capping at 4096

OS Settings:
cat /proc/sys/fs/file-max shows
500000

As the sync gateway user max file descriptors is correct as per your doco.
sync_gateway@sg02-ln-uk-p:/var/log/sync_gateway$ ulimit -n
250000

Thanks for raising this. Which OS/distro/version are you running?

I looked into this and noticed that on CentOS7 in the /usr/lib/systemd/system/sync_gateway.service file, it was defaulting to a hardcoded limit of LimitNOFILE=65535. Maybe you are running into a similar situation where a service definition (systemd or otherwise) is setting the value to 4096.

I proposed a documentation change to make this more clear.

Ubuntu 14.04 but moving up to 16.04 shortly. I can’t find a service file in /etc/systemd/system for sync_gateway. Is this started with old init scripts?

Nothing hardcoded in here.

description "Sync Gateway Upstart Script"
version "0.1.0"
author “Andrew Reslan”

Upstart env vars

env RUNAS=sync_gateway
env RUNBASE=/home/sync_gateway
env PIDFILE=/var/run/sync-gateway.pid
env GATEWAY=/opt/couchbase-sync-gateway/bin/sync_gateway
env CONFIG=/home/sync_gateway/construct.json
env LOGS=/home/sync_gateway/logs
env NAME=sync_gateway

Keep the server running on crash or machine reboot

start on runlevel [2345]
stop on shutdown
respawn

pre-start script
mkdir -p $LOGS
chown -R ${RUNAS}:${RUNAS} $LOGS

mkdir -p ${RUNBASE}/data
chown -R ${RUNAS}:${RUNAS} ${RUNBASE}/data
end script

exec start-stop-daemon --start --chuid $RUNAS --chdir $RUNBASE --make-pidfile --pidfile $PIDFILE --startas $GATEWAY – $CONFIG >> ${LOGS}/${NAME}_access.log 2>> ${LOGS}/${NAME}_error.log

OK on 16.04 using systemd the default is 65535 however I can’t find a sync_gateway.service file in /etc/systemd/system. It’s a fresh VM. Any ideas!

2017-07-13T06:42:17.344Z Enabling logging: [HTTP+]
2017-07-13T06:42:17.344Z ==== Couchbase Sync Gateway/1.4.1(3;ddffd85) ====
2017-07-13T06:42:17.344Z requestedSoftFDLimit >= currentHardFdLimit (250000 >= 65535) capping at 65535
2017-07-13T06:42:17.344Z Configured process to allow 65535 open file descriptors
2017-07-13T06:42:17.344Z Opening db /db as bucket “db”, pool “default”, server walrus:data
2017-07-13T06:42:17.344Z Opening Walrus database db on walrus:data
2017-07-13T06:42:17.345Z Using default sync function ‘channel(doc.channels)’ for database "db"
2017-07-13T06:42:17.441Z Reset guest user to config
2017-07-13T06:42:17.441Z Starting admin server on 127.0.0.1:4985
2017-07-13T06:42:17.450Z Starting server on 0.0.0.0:4984 …

For the benefit of others having this issue. I have not found anything in the Ubuntu 14.04 startup scripts to uncap this. However on 16.04 sync gateway installs with systemd startup scripts and you can uncap the 65535 default in /var/lib/lxcfs/cgroup/pids/system.slice/sync_gateway.service

This can be fixed by setting

limit nofile 250000 250000

in the upstart script, typically present at /etc/init/sync_gateway.conf

System ulimit gets set, but the upstart script has to be told to make use of so many file descriptors.