Hello again,
I am having serious trouble figuring this out. Spent way to much time on it already. Here is the gist.
{“couchdb”:“Welcome”,“version”:“v4.5.1-108-g45731d9”,“couchbase”:“6.0.0-1693-community”}
ES version 5.6.16
connector version 4.1
I am trying to set up a simple Remote cluster reference just as I did with the plugin and previous versions of ES/CB.
My prior ES version was 5.4 and CB version was 5.1 and I was using the correct associated plugin for those versions. It Worked great.
couchbase.password: ******
couchbase.username: Administrator
couchbase.port: 9091
couchbase.maxConcurrentRequests: 1024
http.cors.enabled: true
http.cors.allow-origin : /http://localhost(:[0-9]+)?/
couchbase.ignoreFailures: true
couchbase.keyFilter: org.elasticsearch.transport.couchbase.capi.RegexKeyFilter
couchbase.keyFilter.type: exclude
couchbase.keyFilter.keyFiltersRegex.Histoty: ^Histo.*$"
Now with the connector ver I cannot figure out how to setup the remote cluster reference port. Ive tried 9300 and adding the settings above to the new yaml and/or changing the toml file.
This is what is open
systemd-r 529 systemd-resolve 13u IPv4 14196 0t0 TCP 127.0.0.53:53 (LISTEN)
cupsd 606 root 6u IPv6 26156 0t0 TCP [::1]:631 (LISTEN)
cupsd 606 root 7u IPv4 26157 0t0 TCP 127.0.0.1:631 (LISTEN)
node 608 kibana 10u IPv4 29903 0t0 TCP 127.0.0.1:5601 (LISTEN)
beam.smp 920 couchbase 7u IPv4 28828 0t0 TCP *:21100 (LISTEN)
epmd 1306 couchbase 3u IPv4 27131 0t0 TCP *:4369 (LISTEN)
epmd 1306 couchbase 4u IPv6 27132 0t0 TCP *:4369 (LISTEN)
beam.smp 1414 couchbase 25u IPv4 31756 0t0 TCP *:21101 (LISTEN)
beam.smp 1414 couchbase 35u IPv4 34136 0t0 TCP *:8091 (LISTEN)
beam.smp 1681 couchbase 8u IPv4 32806 0t0 TCP *:21102 (LISTEN)
beam.smp 1681 couchbase 19u IPv4 34129 0t0 TCP *:8092 (LISTEN)
projector 1917 couchbase 6u IPv6 31487 0t0 TCP *:9999 (LISTEN)
indexer 1933 couchbase 6u IPv6 34215 0t0 TCP *:9101 (LISTEN)
indexer 1933 couchbase 10u IPv6 32007 0t0 TCP *:9100 (LISTEN)
indexer 1933 couchbase 11u IPv6 31515 0t0 TCP *:9102 (LISTEN)
indexer 1933 couchbase 35u IPv6 32025 0t0 TCP *:9105 (LISTEN)
cbq-engin 1951 couchbase 7u IPv6 31494 0t0 TCP *:8093 (LISTEN)
goxdcr 1968 couchbase 9u IPv4 34542 0t0 TCP 127.0.0.1:9998 (LISTEN)
cbft 1983 couchbase 5u IPv6 32938 0t0 TCP *:8094 (LISTEN)
memcached 1995 couchbase 10u IPv4 34187 0t0 TCP *:11210 (LISTEN)
memcached 1995 couchbase 11u IPv6 34188 0t0 TCP *:11210 (LISTEN)
memcached 1995 couchbase 12u IPv4 34191 0t0 TCP *:11209 (LISTEN)
memcached 1995 couchbase 13u IPv6 34192 0t0 TCP *:11209 (LISTEN)
expressvp 2075 root 6u IPv4 33516 0t0 TCP 127.0.0.1:2015 (LISTEN)
expressvp 3241 adtracktiv 5u IPv4 36848 0t0 TCP 127.0.0.1:36785 (LISTEN)
code 17666 adtracktiv 59u IPv4 314061 0t0 TCP 127.0.0.1:32888 (LISTEN)
java 22056 elasticsearch 152u IPv6 408569 0t0 TCP [::1]:9300 (LISTEN)
java 22056 elasticsearch 154u IPv6 410625 0t0 TCP 127.0.0.1:9300 (LISTEN)
java 22056 elasticsearch 175u IPv6 406428 0t0 TCP [::1]:9200 (LISTEN)
java 22056 elasticsearch 180u IPv6 408908 0t0 TCP 127.0.0.1:9200 (LISTEN)
java 22956 adtracktiv 151u IPv6 422450 0t0 TCP *:31415 (LISTEN)
and this is my toml
# Introduction to TOML config file format:
# https://npf.io/2014/08/intro-to-toml/
# The work of replicating a Couchbase bucket may be distributed between
# several connector worker processes, known collectively as a "group".
#
# The connector name uniquely identifies a group of connector workers.
# All workers in the group must use the same name. The group name
# is required even if there is only one worker.
#
# TIP: Changing the name invalidates the replication status checkpoint.
# To preserve the checkpoint, run the `cbes-checkpoint-backup` command
# before renaming, and `cbes-checkpoint-restore` afterwards.
[group]
name = 'example-group'
# The simplest way to deploy the connector is "static" mode. In this mode,
# each worker process is independent of the others, and must be manually
# configured to know which partitions it is responsible for.
[group.static]
# Each member in a static group must have a unique member number.
memberNumber = 1 # A value from 1 to 'totalMembers', inclusive.
# The total number of workers in the static group.
#
# TIP: To safely add or remove workers from a static group, first stop all
# running workers, then reconfigure them with the new 'totalMembers' value,
# and finally start all the workers again.
totalMembers = 1
# Performance metrics may be written to a log file and/or exposed via HTTP.
[metrics]
httpPort = 31415 # Positive = fixed, 0 = ephemeral, -1 = disable HTTP server
logInterval = '1m' # Set to '0m' to disable metrics logging
# Required if secure connections are enabled (SSL/TLS).
[truststore]
# Use the Java keytool command to create this file and import the root
# certificate from the Security section of the Couchbase admin console
# and/or the CA certificate used by Elasticsearch.
path = 'path/to/truststore'
pathToPassword = 'secrets/truststore-password.toml'
[couchbase]
hosts = ['localhost', '127.0.0.1:9300']
network = 'auto'
bucket = 'bankable'
# By default the connector stores replication checkpoint documents
# in the source bucket. If you'd rather store those documents in a
# separate existing bucket, put that bucket's name here.
metadataBucket = ''
# At a minimum, the Couchbase user must have the "Data DCP Reader"
# role for the source bucket, and "Data Reader" & "Data Writer"
# roles for the bucket where replication checkpoints are stored.
username = 'Administrator'
# Path to a separate TOML file with a single 'password' key.
# The base for a relative path is the connector installation directory.
pathToPassword = 'secrets/couchbase-password.toml'
# Optionally use SSL/TLS to encrypt the connection between Couchbase and
# the connector. Requires Couchbase Enterprise Edition, and requires the
# [truststore] section to be configured.
secureConnection = false
# Options for the Couchbase Database Change Protocol (DCP).
[couchbase.dcp]
compression = true
flowControlBuffer = '128mb'
persistencePollingInterval = '100ms'
[elasticsearch]
hosts = ['localhost', '127.0.0.1:9300', '127.0.0.1:9091']
username = 'elastic'
# Path to a separate TOML file with a single 'password' key.
# The base for a relative path is the connector installation directory.
pathToPassword = 'secrets/elasticsearch-password.toml'
# Optionally use SSL/TLS to encrypt the connection between Elasticsearch and
# the connector. Requires the [truststore] section to be configured. See also:
# https://www.elastic.co/guide/en/elasticsearch/reference/current/configuring-tls.html
secureConnection = false
[elasticsearch.bulkRequestLimits]
bytes = '10mb'
actions = 1000
timeout = '1m'
concurrentRequests = 2
[elasticsearch.docStructure]
# The Elasticsearch document may optionally contain Couchbase metadata
# (cas, revision, expiry, etc). If present, this will be a top-level field
# of the Elasticsearch document. If you have configured the connector
# to put document content at the top level, choose a field name that will
# not conflict with any document fields. To omit the metadata field,
# use an empty string.
metadataFieldName = 'meta'
# false means the ES document root will have a 'doc' field whose value is
# the Couchbase document.
# true means the ES document will be identical to the Couchbase document,
# with the possible addition of the metadata field.
documentContentAtTopLevel = false
# false means ignore Couchbase counter documents.
# true means replicate them as Object nodes like {"value":<counter>}
wrapCounters = false
# The values in this section may be overridden by specific type definitions.
[elasticsearch.typeDefaults]
# Write matching documents to this index.
# Empty string means "no default".
index = ''
# Send matching documents though this pipeline.
# Empty string means "no pipeline".
pipeline = ''
# Assign this type to matching documents.
# For ES 5.x remove leading underscore!
typeName = 'doc'
# If true, ignore matching documents entirely (implies 'ignoreDeletes').
ignore = false
# If true, never delete matching documents from Elasticsearch.
ignoreDeletes = false
# Sample document type definitions for the travel-sample bucket.
# Replace these to match your own data model.
#
# At a minimum, each type must specify a 'regex' (Java regular expression)
# or 'prefix' property. The type config matches any Couchbase document
# whose IDs matches the regex or prefix.
#
# The order of type definitions is significant. If a document ID matches
# more than one type, the assigned type will be the one that occurs first
# in this configuration. If none match, the document will not be indexed.
[[elasticsearch.type]]
prefix = '_sync:' # Couchbase Sync Gateway metadata
ignore = true
[[elasticsearch.type]]
prefix = 'airline_'
index = 'airlines'
pipeline = ''
[[elasticsearch.type]]
# Regex just for example. Matches prefixes "airport_", "seaport_", etc.
regex = '.*port_.*'
index = 'airports'
[[elasticsearch.type]]
# Index can be inferred from document ID by including a capturing group
# named "index". This example matches IDs that start with one or more
# characters followed by "::". It directs "user::alice" to index "user",
# and "foo::bar::123" to index "foo".
regex = '(?<index>.+?)::.*'
# In the travel-sample data model, a route is the child of an airline.
# Each route document has an `airlineid` field that holds the parent ID.
# The following type definition ensures each route document is stored
# in the same index and shard as its parent airline document.
# For more information about Elasticsearch parent/join fields, see
# https://www.elastic.co/guide/en/elasticsearch/reference/current/parent-join.html
[[elasticsearch.type]]
prefix = 'route_'
index = 'airlines'
routing = '/airlineid' # JSON pointer to the parent ID field.
ignoreDeletes = true # Must always be true if `routing` is specified.
[[elasticsearch.type]]
prefix = '' # Empty prefix matches any document ID.
index = 'etc'
# When Elasticsearch rejects a document (usually due to a type mapping error)
# a rejection log entry document is written to Elasticsearch. The log entry's
# document ID is the ID of the rejected Couchbase document. The log entry's
# content has the following fields:
# "index" - (string) name of the index the connector tried to write to
# "type" - (string) document type name used for the write attempt
# "action" - (string) failed action type ("INDEX" or "DELETE")
# "error" - (string) error message received from Elasticsearch
[elasticsearch.rejectionLog]
index = 'cbes-rejects'
typeName = 'doc' # For ES 5.x remove leading underscore!
“please help me obi wan kenobi your my only hope”
