mongodb: directory-sync: open: Too many open files

  mongodb, question

Problem description

mongodb log:

2019-05-03T15:57:05.627+0800 I STORAGE  [conn2] createCollection: test.site_userid_31917 with generated UUID: a7deb9ff-dfa5-4272-b10d-2da69c7a2e3c
 2019-05-03T15:57:05.653+0800 I STORAGE  [conn2] createCollection: test.site_userid_31918 with generated UUID: 8ae5dfdc-c416-482e-be28-ac0af623a512
 2019-05-03T15:57:06.016+0800 E -        [ftdc] Assertion: Location13538: couldn't open [/proc/7276/stat] Unknown error src/mongo/util/processinfo_linux.cpp 81
 2019-05-03T15:57:06.172+0800 E STORAGE  [conn2] WiredTiger error (24) [1556870226:172133][7276:0x7f135e8b7700],   WT_SESSION.create: __posix_directory_sync,   145: /var/lib/mongodb/: directory-sync: open: Too many open files Raw: [1556870226:172133][7276:0x7f135e8b7700],   WT_SESSION.create: __posix_directory_sync, 145: /var/lib/mongodb/: directory-sync: open: Too many open files
 2019-05-03T15:57:06.172+0800 E STORAGE  [conn2] WiredTiger error (24) [1556870226:172364][7276:0x7f135e8b7700],   WT_SESSION.create: __posix_directory_sync,   160: /var/lib/mongodb/index-63839--3770517573993711728.wt: directory-sync: Too many open files Raw: [1556870226:172364][7276:0x7f135e8b7700],   WT_SESSION.create: __posix_directory_sync, 160: /var/lib/mongodb/index-63839--3770517573993711728.wt: directory-sync: Too many open files
 2019-05-03T15:57:06.172+0800 E STORAGE  [conn2] WiredTiger error (-31804) [1556870226:172415][7276:0x7f135e8b7700], WT_SESSION.create: __wt_panic,   520: the process must exit and restart: WT_PANIC: WiredTiger library panic Raw: [1556870226:172415][7276:0x7f135e8b7700],   WT_SESSION.create: __wt_panic, 520: the process must exit and restart: WT_PANIC: WiredTiger library panic
 2019-05-03T15:57:06.176+0800 F -        [conn2] Fatal Assertion 50853 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 409
 2019-05-03T15:57:06.178+0800 F -        [conn2]
 
 ***aborting after fassert() failure
 
 
 2019-05-03T15:57:06.305+0800 F -        [conn2] Got signal: 6 (Aborted).
 0x55c4d32c1141 0x55c4d32c0359 0x55c4d32c083d 0x7f13709df890 0x7f137061ae97 0x7f137061c801 0x55c4d18c1609 0x55c4d19bfe06 0x55c4d1a32569 0x55c4d1848a58 0  x55c4d1848e72 0x55c4d19fc5fe 0x55c4d19fce3b 0x55c4d19f8dfb 0x55c4d1a548d6 0x55c4d1a17d46 0x55c4d1a1870b 0x55c4d1a18061 0x55c4d1a2d23a 0x55c4d1a2d5b6 0x  55c4d1988320 0x55c4d1993d92 0x55c4d1b2f38a 0x55c4d205c5e2 0x55c4d205f27c 0x55c4d2054c3c 0x55c4d204ebf3 0x55c4d1f9f397 0x55c4d1f9f704 0x55c4d1f9fa0a 0x5  5c4d1fa43bc 0x55c4d1f9ddcf 0x55c4d1f9c022 0x55c4d19730a8 0x55c4d19750cb 0x55c4d19769e9 0x55c4d1977921 0x55c4d196306a 0x55c4d196fb8a 0x55c4d196a907 0x55  c4d196e061 0x55c4d2b17b42 0x55c4d1968af0 0x55c4d196bbe5 0x55c4d196a047 0x55c4d196a98d 0x55c4d196e061 0x55c4d2b180a5 0x55c4d3218af4 0x7f13709d46db 0x7f13706fd88f
 ----- BEGIN BACKTRACE -----

The environmental background of the problem and what methods have you tried?

Copy, Test Case:

In [19]: prefix = 'site_userid_'
 In [20]: for i in range(100000):
 ...:     coll_name = prefix + str(i)
 ...:     coll = db[coll_name]
 ...:     coll.insert_one({"name": coll_name})
 ...:

GIF before creating 30000 collection:

About 3,000 collection went wrong.

Plans that have been tested:

  1. Modify unix-like filelimit
  2. Modify Code-“Quantitative” Restart MongoClient

Invalid.


What is your expected result? What is the actual error message?

Welcome friends, if you can point out that the above is my MongoDB where the configuration is wrong?
Or is coding used incorrectly?

After searching google for a day, there was no solution.


Note:
If you have any questions, please comment in the comment area.
Don’t paste unverified solution by feeling.

It is totally acceptable to say that the way I use mongdb is wrong, but please explain the reason and the correct way to use it.

The idea of this test case is a website site hundreds of thousands of userids.
Each user id corresponds to a collection according to its unique user id, which contains, for example, user’s profile and other contents.

Please do not answer if you are not clear. Please communicate in the comment area. Thank you.

After discussion, I think the answer to the question is clearer. Clear the following principles:

  1. Each collection’s data file creates a separate file (collection-xxxxxx.wt)
  2. Each index file creates a separate file (index-xxxxxx.wt)
  3. Each collection requires a data file and a default index (_id index)

The above problem means that every set created takes up 2 file handles, and 64,000 handles can create up to 32,000 sets. But you also need to remove the handles MongoDB itself needs to use (internal files such as wiredTiger.wt, plus network connections), so when you createsite_userid_31918There are not enough handles, which is normal behavior.
The problem is still “why do you want to create so many collections?” As I said in my comments, MongoDB recommends slicing instead of dividing libraries and tables. The latter will have a strong invasion on the program logic in use, causing the logic to be complicated. In addition, it will also cause other problems, such as a long restart time. Because this is not MongoDB’s recommended usage.