Bug with SDK 3, missing native epoll/kqueue libs

Hi

The core-io v2 lib (com.couchbase.client.core-io.2.0.*), used for the latest v3 java sdk are missing the native netty libraries, this is because of a misconfiguration in the core-io-deps. You can see as per https://netty.io/wiki/native-transports.html#wiki-h2-0 the classifier is required, the fix for https://github.com/couchbase/couchbase-jvm-clients/blob/master/core-io-deps/pom.xml is:

--- a/core-io-deps/pom.xml
+++ b/core-io-deps/pom.xml
@@ -43,11 +43,13 @@
             <groupId>io.netty</groupId>
             <artifactId>netty-transport-native-kqueue</artifactId>
             <version>${netty.version}</version>
+            <classifier>osx-x86_64</classifier>
         </dependency>
         <dependency>
             <groupId>io.netty</groupId>
             <artifactId>netty-transport-native-epoll</artifactId>
             <version>${netty.version}</version>
+            <classifier>linux-x86_64</classifier>
         </dependency>
         <dependency>
             <groupId>io.netty</groupId>
@@ -100,6 +102,7 @@
                                     <groupId>io.netty</groupId>
                                     <artifactId>netty-transport-native-kqueue</artifactId>
                                     <outputDirectory>${native.folder}</outputDirectory>
+                                    <classifier>osx-x86_64</classifier>
                                     <includes>META-INF/native/*</includes>
                                 </artifactItem>
                                 <artifactItem>
@@ -107,6 +110,7 @@
                                     <artifactId>netty-transport-native-epoll</artifactId>
                                     <outputDirectory>${native.folder}</outputDirectory>
                                     <includes>META-INF/native/*</includes>
+                                    <classifier>linux-x86_64</classifier>
                                 </artifactItem>
                                 <artifactItem>
                                     <groupId>io.netty</groupId>
@@ -208,4 +212,4 @@
         </plugins>
     </build>

Before:

unzip -l core-io-2.0.2.jar | grep "/native/"
        0  2020-02-04 20:55   META-INF/native/
  2630627  2020-02-04 20:08   META-INF/native/libcom_couchbase_client_core_deps_netty_tcnative_linux_x86_64.so
  2690856  2020-02-04 20:08   META-INF/native/libcom_couchbase_client_core_deps_netty_tcnative_osx_x86_64.jnilib
  2434048  2020-02-04 20:08   META-INF/native/com_couchbase_client_core_deps_netty_tcnative_windows_x86_64.dll

After fix:

unzip -l core-io/target/core-io-2.1.0-SNAPSHOT.jar | grep "/native/" 
        0  2020-02-26 17:11   META-INF/native/
  2630627  2020-02-26 15:38   META-INF/native/libcom_couchbase_client_core_deps_netty_tcnative_linux_x86_64.so
  2690856  2020-02-26 15:38   META-INF/native/libcom_couchbase_client_core_deps_netty_tcnative_osx_x86_64.jnilib
  2434048  2020-02-26 15:38   META-INF/native/com_couchbase_client_core_deps_netty_tcnative_windows_x86_64.dll
    40608  2020-01-13 12:29   META-INF/native/libnetty_resolver_dns_native_macos_x86_64.jnilib
    82248  2020-02-26 16:35   META-INF/native/libcom_couchbase_client_core_deps_netty_transport_native_epoll_x86_64.so
    52444  2020-02-26 16:35   META-INF/native/libcom_couchbase_client_core_deps_netty_transport_native_kqueue_x86_64.jnilib
1 Like

Thanks for reporting - this is actually a regression in 2.0.2 (2.0.0 and 2.0.1 contained the natives properly) because we optimized what gets included from netty to save a bit on the jar size. I’ll fix it for the next release.

2.0.0

META-INF/native
β”œβ”€β”€ com_couchbase_client_core_deps_netty_tcnative_windows_x86_64.dll
β”œβ”€β”€ libcom_couchbase_client_core_deps_netty_tcnative_linux_x86_64.so
β”œβ”€β”€ libcom_couchbase_client_core_deps_netty_tcnative_osx_x86_64.jnilib
β”œβ”€β”€ libcom_couchbase_client_core_deps_netty_transport_native_epoll_x86_64.so
β”œβ”€β”€ libcom_couchbase_client_core_deps_netty_transport_native_kqueue_x86_64.jnilib
└── libnetty_resolver_dns_native_macos_x86_64.jnilib

2.0.1

META-INF/native
β”œβ”€β”€ com_couchbase_client_core_deps_netty_tcnative_windows_x86_64.dll
β”œβ”€β”€ libcom_couchbase_client_core_deps_netty_tcnative_linux_x86_64.so
β”œβ”€β”€ libcom_couchbase_client_core_deps_netty_tcnative_osx_x86_64.jnilib
β”œβ”€β”€ libcom_couchbase_client_core_deps_netty_transport_native_epoll_x86_64.so
β”œβ”€β”€ libcom_couchbase_client_core_deps_netty_transport_native_kqueue_x86_64.jnilib
└── libnetty_resolver_dns_native_macos_x86_64.jnilib

2.0.2

META-INF/native
β”œβ”€β”€ com_couchbase_client_core_deps_netty_tcnative_windows_x86_64.dll
β”œβ”€β”€ libcom_couchbase_client_core_deps_netty_tcnative_linux_x86_64.so
└── libcom_couchbase_client_core_deps_netty_tcnative_osx_x86_64.jnilib

Ah yes I see it introduced on this commit https://github.com/couchbase/couchbase-jvm-clients/commit/d62395c3e3cbeee69f1326ef41237ca8297cb177

Thanks for fixing

Tracking in https://issues.couchbase.com/browse/JVMCBC-824, will be part of 3.0.2

I built the couchbase java client from the latest master and it still throws the error which is available here.

@PShri actually if you look at the bottom of your log it actually could load it. It just looked in different places first and couldn’t find it there, so you should be all set.

Ah you are right. Thanks for pointing that out.

Is there anything I could do to make it look at the right place first time it looks for it?

No it just has to do how netty tries to look it up, and since we shadow it, it looks there a bit later. But note that it is debug output, so if you configure it at info level you won’t see it anyways.