Issue with eventing. curl() does nothing

Eventing
1

Hello in the Couchbase forum.

I’m ferociously trying to get Couchbase Eventing to work in the following way:

SCENARIO:
When a document is created >> post the document to a remote endpoint via the curl() function. Documents are created by another entity I control.

STEPS:

  • Setup Couchbase 5.5 via the Docker image on DockerHub.
  • Created a Couchbase Function via Couchbase Eventing. This function is very simple and has the following code.
    function OnUpdate(doc, meta) { var http = SELECT CURL( "http://192.168.49.111:8080/api/test", { "request": "GET", "header": "application/json" } ); http.execQuery(); }
  • Deployed the function.
  • Whitelisted > The endpoint via ‘Settings’ > ‘Advanced Query Settings’ > ‘CURL() function access’
  • Setup another container, wherein the API endpoint is hosted. On the same Docker host as the Couchbase container. The GET endpoint just writes a file to disk. To confirm that ‘x’ happened when the request from Couchbase Eventing was retrieved.
  • Posted a document to Couchbase.
  • Confirmed that the document is written to Couchbase.
  • Confirmed that the OnUpdate() function in my Eventing Couchbase function was processed. This via the ‘Buckets’ >> ‘Statistics’ >> ‘Eventing Stats: “MyFunction”’

ISSUE:
The endpoint is not successfully called via the curl() method in the OnUpdate() function. I can confirm this because the code invoked when calling the specified endpoint should write a file to disk. This does not happen when Couchbase eventing executes.

TROUBLESHOOTING
I have tried the following in order to troubleshoot this.

  • Confirmed that the called API endpoint works. Basically just opened a browser and accessed it from there (GET method, so okay). A file is written to disk.
  • Tried whitelisting the root endpoint name in Couchbase (so no /api/test part).
  • Tried having the OnUpdate() function copy a CREATED document to another bucket. This works. So "stuff " can/do happen when using Couchbase Eventing.
  • Scrolled through the eventing.log stored in /opt/couchbase/var/lib/couchbase/logs. There is nothing in that indicating why nothing happens. It actually indicates a success:
    2018-05-30T10:11:43.449+00:00 [Info] eventing-consumer [worker_TestingShit_1:/tmp/127.0.0.1:8091_worker_TestingShit_1.sock:21602] v8worker execution stats:{"on_update_success":1, "on_update_failure":0, "on_delete_success":0, "on_delete_failure":0, "doc_timer_create_failure":0, "messages_parsed":1113, "cron_timer_msg_counter":0, "dcp_delete_msg_counter":0, "dcp_mutation_msg_counter":1, "doc_timer_msg_counter":0, "enqueued_cron_timer_msg_counter":0, "enqueued_dcp_delete_msg_counter":0, "enqueued_dcp_mutation_msg_counter":1, "enqueued_doc_timer_msg_counter":0, "doc_timer_responses_sent":0, "uv_try_write_failure_counter":0, "agg_queue_size":0, "feedback_queue_size":0, "timestamp":"2018-05-30T10:11:43Z"}
    – The above is from tests when using the N1QL curl approach. This seems to work, at least in the logs, however nothing happens (no file is written).
    – When using curl() directly Couchbase Eventing states that the OnUpdate() code failed. Same endpoint, same everything.
  • Tried curling the API endpoint from within the Couchbase container. No problem, a request is retrieved and a file is written.
    ** That confirms access.
    ** Confirms, once more, that the remote */api/test endpoint works.

I would like pointers, advice … basically help on why the above is giving me trouble :face_with_raised_eyebrow:

Also. I would like to know how to, and where to view the output of the log() function when used inside an OnUpdate() Couchbase Eventing function.

I’m looking forward to hearing from you. Thank you very much in advance.

2

Hi,

Thanks for details.

I presume, you’re trying against 5.5 Beta build?

I haven’t tried Eventing on docker builds yet, let me request our QE folks if they could simulate this scenario.

It goes inside @eventing folder inside “Index Storage Path” you would configured for the cluster. 37%20PM

Meanwhile, you could try spawning up a debugger session and check out if n1ql calls are throwing some error? But based on log counter you pasted - “on_update_success”:1 , that might not be the case.

Thanks,
Abhishek

3

Hi @asingh,

Thank you for looking into this. It would surely be lovely if you could get your QE folks to have a peep at it. Because yes, it might have something to do with some Docker setting or circumstance occuring when inside the Docker Container. Let me know what they find out.

And. Sorry for not mentioning the version I’m trying this one. But yes. It is v5.5 build 2473 I think. The newest version build to DockerHub. (I don’t have access to the Couchbase where I am now).

Thank you for pointing me to storage location for calls to log(). Is that info somewhere in the documentation? If not that would be lovely to have. None else, for people in need of the info as I was.

Thank you and I will be looking forward to hearing from you again. :smile:

4

More info. I debugged this some more. Findings:

Thank you again @asingh for the info on the log() method. I found the folder and can see data. Nice help in the future.

Now I’m just hoping for this curl() method call to work in my function by the advice of some clever Couchbase minds.

Thank you :slight_smile:

5

Also I will have a hard time debugging this as Couchbase is running inside a Docker container. Because of the random port assignment of the debugging mechanics.
As Matthew Groves mentions in this video > https://youtu.be/tBuliswRiU4?t=224

6

Hi,

Yes, debugger right now asks OS to provide a free port that’s available - and then uses it. There is bug open for it to make the debugger port static - so easier to whitelist in container environment, we will tentatively ship the fix for it in next release as 5.5 is almost ready to be shipped.

I’ve brought docker testing before out PM @venkat as well.

Thanks,
Abhishek

7

Splendid. Thank you.

8

Any news on this?

@asingh or @venkat :slight_smile:

9

@NotLayingOnTheCouch : apologies, but do let us know if we missed answering any query of yours. I guess @asingh answered you?

  • The doc will get a refresh when the 5.5 release goes GA.
  • On the Docker test, we have it in the roadmap and will test with the rest of the features. As of now, the curl calls(be it N1QL or native) are DP features and are yet to undergo exhaustive testing. I would recommend not to use them for any production workloads for now.
10

Thank you for following up @venkat. I will wait with using it for production until the GA release of v5.5 comes out.

Again, thank you.

11

FYI, a couple more examples of using cURL (through N1QL)

function OnUpdate(doc, meta) {
let here = db['here'];

let url = "https://places.cit.api.here.com/places/v1/discover/around";
let data = `app_id=${here.id}&app_code=${here.code}&at=${doc.lat},${doc.lon}`;

let query = SELECT CURL($url, {
    "request": "GET",
    "header": ["Content-Type: application/json", "accept: application/json"],
    "data": $data
}).results.items AS poi;

let response = query.execQuery();
db['poi'] = response[0].poi;

url = "http://localhost:8080/events/poi";
data = JSON.stringify(response[0].poi);
query = SELECT CURL($url, {
    "request": "POST",
    "header": ["Content-Type: application/json", "accept: application/json"],
    "data": $data
});

response = query.execQuery();
}
12

hi

If your couchbase is in Docker container and using [Docker-for-mac] or [Docker-for-Windows],
Connect to your couchbase service using the host host.docker.internal (instead of the 127.0.0.1 or localhost in your connection string).

Like : url = “http://host.docker.internal:8080/events/poi”;

Thanks

13

Hi @NotLayingOnTheCouch,

Please look at the examples at basicCurlGet and basicCurlPost (valid for both 6.X and 7.X releases).

Please, note the final committed version for direct curl() in the Eventing Service occured in version 6.5.

I know it’s been about three years, but bphalak comment triggered a notify to me and I thought I wold provide some clarity and samples.

Best

Jon Strabala
Principal Product Manager - Server‌