moxi + S3 + Elasticache ?
Hi!
It is possible to use a standalone moxi configuration where the cluster configuration is a static file retrieved via HTTP and that config has only memcached servers?
Specifically, I would like my moxi-cluster.cfg to contain only "url=http://s3.amazonaws.com/mybucket/myconfig.json" and the json on S3 would be crafted to be a list of Elasticache endpoints. Then as I add more Elasticache endpoints, I could update the json config file on S3 and all my autoscaled webservers would have their standalone moxi clients updated via HTTP to use the new endpoint.
Thanks,
Tony
There's not a lot of detail here, but this might help: http://www.couchbase.org/wiki/display/membase/Moxi+with+Memcached
The JSON format is really only useful for transferring a vbucketmap which is pretty closely tied to Membase.
You'll want to look at using Moxi with just standard memcached.
Perry
Thanks for the reply. I have tried the -z flag with "11211=one.example.com:11211,two.example.com:11211,three.example.com:11211" in a local config file and it worked fine. However if I were to add another node, reconfiguring and restarting moxi on a pool of EC2 servers in an AutoScaling group would be difficult.
After having a few days to work on HTTP/REST setup, I arrived at this non-working example of 'myconfig.json':
{ "hashAlgorithm": "CRC", "numReplicas": 0, "serverList": [ "one.example.com:11211", "two.example.com:11211", "three.example.com:11211" ], "vBucketMap": [ [ 0 ] ] }
In looking at the moxi logs and the source code, I think the problem was that moxi was trying SASL auth with Elasticache's memcached - specifically this error: http://www.couchbase.org/wiki/display/membase/Moxi+Error+Responses#MoxiE... . Is there a simple way to turn off SASL?
Tony
Take a look at this post: http://www.couchbase.org/forums/thread/new-moxi-error
Let me know if changing the vbucketmap helps...
Perry
Thanks again for your help in trying to debug this config.
I added "nodeLocator": "vbucket" and tried with three vbucketmap configs: 1)"vBucketMap": [ [0] ] 2)"vBucketMap": [ [0],[1] ] and 3)"vBucketMap": [ [0],[1],[0],[1] ](that's 2^0, 2^1 and 2^2 nodes). None worked. After adding '-vvv -O /var/log/moxi/moxi.log' to /opt/moxi/etc/moxi-init.d from the rpm http://packages.couchbase.com/releases/1.7.1/moxi-server_x86_1.7.1.rpm I attempted to retrieve a key with ( echo "get testkey"; sleep 0.2; echo "quit" ) | nc -C localhost 11211. The log output was:
2011-09-06 20:40:01: (cproxy.c.1760) assign_downstream
2011-09-06 20:40:01: (cproxy.c.871) cproxy_add_downstream 0 1024
2011-09-06 20:40:01: (cproxy.c.1280) cproxy_create_downstream: {
"hashAlgorithm": "CRC",
"numReplicas": 0,
"nodeLocator": "vbucket",
"serverList": [
"one.example.com:11211",
"two.example.com:11211"
],
"vBucketMap": [
[0],[1]
]
}, 6, 8
2011-09-06 20:40:01: (mcs.c.52) mcs_create using libvbucket
2011-09-06 20:40:01: (cproxy.c.989) -1: release_downstream
2011-09-06 20:40:01: (cproxy.c.1381) check_downstream_config 1
2011-09-06 20:40:01: (cproxy.c.1381) check_downstream_config 1
2011-09-06 20:40:01: (cproxy.c.1858) 54: assign_downstream, matched to upstream
2011-09-06 20:40:01: (cproxy.c.1932) 54: cproxy_forward prot 5 to prot 8
2011-09-06 20:40:01: (cproxy.c.1421) 54: cproxy_connect_downstream server_index 0 in 2
2011-09-06 20:40:01: (cproxy.c.1549) 55: cproxy_connect_downstream_conn one.example.com:11211 0 115
2011-09-06 20:40:01: (memcached.c.403) <55 new auto-negotiating client connection
2011-09-06 20:40:01: (memcached.c.670) 55: going from conn_pause to conn_connecting
2011-09-06 20:40:01: (cproxy.c.1885) assign_downstream, done
2011-09-06 20:40:01: (memcached.c.3457) 54: drive_machine conn_pause
2011-09-06 20:40:01: (memcached.c.3457) 55: drive_machine conn_connecting
2011-09-06 20:40:01: (cproxy.c.3085) 55: cproxy_on_connect_downstream_conn for one.example.com:11211:default:(null):0
2011-09-06 20:40:01: (cproxy.c.2801) cproxy_auth_downstream usr: default pwd: (0)
2011-09-06 20:40:01: (cproxy.c.2899) auth_downstream failure for default (81)
2011-09-06 20:40:01: (cproxy.c.3235) z_error, one.example.com:11211:default:(null):0, 1, 1, 1, 8580
2011-09-06 20:40:01: (cproxy.c.648) <55 cproxy_on_close_downstream_conn quit_server
2011-09-06 20:40:01: (memcached.c.670) 55: going from conn_connecting to conn_closing
2011-09-06 20:40:01: (cproxy.c.1932) 54: cproxy_forward prot 5 to prot 8
2011-09-06 20:40:01: (cproxy.c.1421) 54: cproxy_connect_downstream server_index 0 in 2
2011-09-06 20:40:01: (cproxy.c.1911) 54: could not forward upstream to downstream
2011-09-06 20:40:01: (cproxy.c.2004) 54: upstream_error: SERVER_ERROR proxy write to downstream one.example.com
2011-09-06 20:40:01: (memcached.c.670) 54: going from conn_pause to conn_mwrite
2011-09-06 20:40:01: (cproxy.c.989) -1: release_downstream
2011-09-06 20:40:01: (cproxy.c.1381) check_downstream_config 1
2011-09-06 20:40:01: (memcached.c.3457) 55: drive_machine conn_closing
2011-09-06 20:40:01: (cproxy.c.669) <55 cproxy_on_close_downstream_conn
2011-09-06 20:40:01: (cproxy.c.3235) z_error, one.example.com:11211:default:(null):0, 1, 0, 2, 8580
2011-09-06 20:40:01: (cproxy.c.710) 55: skipping release dc in on_close_dc
2011-09-06 20:40:01: (memcached.c.556) <55 connection closed.You can see that in the line 2011-09-06 20:40:01: (cproxy.c.2801) cproxy_auth_downstream usr: default pwd: (0) the username and password are not present. My question is why is it trying to auth at all? SASL auth is not needed for plain memcached. Just to try it, if I add the "usr=" and "pwd=" fields to /opt/moxi/etc/moxi.cfg it attempts to use basic auth with S3 when trying to configure via HTTP, which is not permitted by S3.
Thanks,
Tony
Hi,
Catching up on this thread.
One thought is you just want to use moxi in front of memcached (instead of membase), then your cfg file should specify 'ketama' as the nodeLocator algorithm instead of the 'vbucket' algorithm. For example...
{"name":"default",
"nodeLocator":"ketama",
"nodes":[
{"hostname":"mc-001", "ports":{"proxy":11213,"direct":11212}}
{"hostname":"mc-002", "ports":{"proxy":11213,"direct":11212}}
]}
I typed that without checking it, so YMMV. Let me know if the mileage wasn't good enough (might have forgotten something).
Cheers,
Steve
Steve,
Thanks for the off-the-cuff config - there was only 1 comma missing :-) I was uncertain what to set "proxy" and "direct" to. Since there's only 1 port for memcached/Elasticache I set them both set to 11211.
Unfortunately, I'm still getting the "cproxy_auth_downstream usr: default pwd: (0)" + "auth_downstream failure for default (81)" error when trying to use it. The sanitized -vvv debug logs are below. The command I used to test a set was:
# ( echo "set testkey 0 3600 1"; echo "1"; sleep 0.2; echo "quit" ) | nc -C localhost 11211
SERVER_ERROR proxy write to downstream one.example.com
I believe it is still trying to SASL to memcache/Elasticache. In the output of 'tcpdump -A -s 65535 one.example.com', I see moxi sending "PLAIN.default" to memcache/Elasticache and it responding with "Unknown command".
To confirm, the only configuration I've done is add "url=http://s3.amazonaws.com/mytestbucket/testconfig.json" to moxi-cluster.cfg and add to the daemon line in moxi-init.d "-vvv -O /var/log/moxi/moxi.log".
Thoughts?
Thanks again,
Tony
---8<---
2011-09-07 14:48:32: (agent_config.c.397) configuration received
2011-09-07 14:48:32: (agent_config.c.1216) conc new_config_ver 129
2011-09-07 14:48:32: (agent_config.c.556) conjo contents config from <a href="http://s3.amazonaws.com/mytestbucket/testconfig.json:" title="http://s3.amazonaws.com/mytestbucket/testconfig.json:">http://s3.amazonaws.com/mytestbucket/testconfig.json:</a> {
"name": "default",
"nodeLocator": "ketama",
"nodes": [
{ "hostname": "one.example.com", "ports": { "proxy":11211,"direct":11211 }},
{ "hostname": "two.example.com", "ports": { "proxy":11211,"direct":11211 }}
]
}
2011-09-07 14:48:32: (agent_config.c.730) parsing config nodeLocator:ketama
2011-09-07 14:48:32: (agent_config.c.761) conjk nodes_num: 2
2011-09-07 14:48:32: (agent_config.c.1409) conp existing config change 11211
2011-09-07 14:48:32: (agent_config.c.1472) conp changed false, shutdown false
2011-09-07 14:48:32: (agent_config.c.1594) update_ptd_config 11211, 128 = 129 no change
2011-09-07 14:48:32: (agent_config.c.1594) update_ptd_config 11211, 128 = 129 no change
2011-09-07 14:48:32: (agent_config.c.1594) update_ptd_config 11211, 128 = 129 no change
2011-09-07 14:48:32: (agent_config.c.1594) update_ptd_config 11211, 128 = 129 no change
2011-09-07 14:48:32: (agent_config.c.1518) conp changed false, 129
2011-09-07 14:48:33: (memcached.c.3457) 51: drive_machine conn_listening
2011-09-07 14:48:33: (memcached.c.403) <54 new auto-negotiating client connection
2011-09-07 14:48:33: (cproxy.c.489) <54 assigning to default bucket: default
2011-09-07 14:48:33: (memcached.c.3457) 54: drive_machine conn_new_cmd
2011-09-07 14:48:33: (memcached.c.670) 54: going from conn_new_cmd to conn_waiting
2011-09-07 14:48:33: (memcached.c.3457) 54: drive_machine conn_waiting
2011-09-07 14:48:33: (memcached.c.670) 54: going from conn_waiting to conn_read
2011-09-07 14:48:33: (memcached.c.3457) 54: drive_machine conn_read
2011-09-07 14:48:33: (memcached.c.670) 54: going from conn_read to conn_parse_cmd
2011-09-07 14:48:33: (memcached.c.3457) 54: drive_machine conn_parse_cmd
2011-09-07 14:48:33: (memcached.c.3125) 54: Client using the proxy-upstream-ascii protocol
2011-09-07 14:48:33: (cproxy_protocol_a.c.34) <54 cproxy_process_upstream_ascii set testkey 0 3600 1
2011-09-07 14:48:33: (memcached.c.670) 54: going from conn_parse_cmd to conn_nread
2011-09-07 14:48:33: (memcached.c.3457) 54: drive_machine conn_nread
2011-09-07 14:48:33: (memcached.c.3457) 54: drive_machine conn_nread
2011-09-07 14:48:33: (memcached.c.3457) 54: drive_machine conn_nread
2011-09-07 14:48:33: (cproxy.c.2175) 54: pause_upstream_for_downstream
2011-09-07 14:48:33: (memcached.c.670) 54: going from conn_nread to conn_pause
2011-09-07 14:48:33: (cproxy.c.2228) wait_queue_timeout started
2011-09-07 14:48:33: (cproxy.c.1760) assign_downstream
2011-09-07 14:48:33: (cproxy.c.871) cproxy_add_downstream 0 1024
2011-09-07 14:48:33: (cproxy.c.1280) cproxy_create_downstream: one.example.com:11211,two.example.com:11211, 129, 8
2011-09-07 14:48:33: (mcs.c.60) mcs_create using libmemcached
2011-09-07 14:48:33: (cproxy.c.989) -1: release_downstream
2011-09-07 14:48:33: (cproxy.c.1381) check_downstream_config 1
2011-09-07 14:48:33: (cproxy.c.1381) check_downstream_config 1
2011-09-07 14:48:33: (cproxy.c.1858) 54: assign_downstream, matched to upstream
2011-09-07 14:48:33: (cproxy.c.1932) 54: cproxy_forward prot 5 to prot 8
2011-09-07 14:48:33: (cproxy.c.1421) 54: cproxy_connect_downstream server_index 0 in 2
2011-09-07 14:48:33: (cproxy.c.1549) 55: cproxy_connect_downstream_conn one.example.com:11211 0 115
2011-09-07 14:48:33: (memcached.c.403) <55 new auto-negotiating client connection
2011-09-07 14:48:33: (memcached.c.670) 55: going from conn_pause to conn_connecting
2011-09-07 14:48:33: (cproxy.c.1885) assign_downstream, done
2011-09-07 14:48:33: (memcached.c.3457) 54: drive_machine conn_pause
2011-09-07 14:48:33: (memcached.c.3457) 55: drive_machine conn_connecting
2011-09-07 14:48:33: (cproxy.c.3085) 55: cproxy_on_connect_downstream_conn for one.example.com:11211:default:(null):0
2011-09-07 14:48:33: (cproxy.c.2801) cproxy_auth_downstream usr: default pwd: (0)
2011-09-07 14:48:33: (cproxy.c.2899) auth_downstream failure for default (81)
2011-09-07 14:48:33: (cproxy.c.3235) z_error, one.example.com:11211:default:(null):0, 1, 1, 1, 134114
2011-09-07 14:48:33: (cproxy.c.648) <55 cproxy_on_close_downstream_conn quit_server
2011-09-07 14:48:33: (memcached.c.670) 55: going from conn_connecting to conn_closing
2011-09-07 14:48:33: (cproxy.c.1932) 54: cproxy_forward prot 5 to prot 8
2011-09-07 14:48:33: (cproxy.c.1421) 54: cproxy_connect_downstream server_index 0 in 2
2011-09-07 14:48:33: (cproxy.c.1911) 54: could not forward upstream to downstream
2011-09-07 14:48:33: (cproxy.c.2004) 54: upstream_error: SERVER_ERROR proxy write to downstream one.example.com
2011-09-07 14:48:33: (memcached.c.670) 54: going from conn_pause to conn_mwrite
2011-09-07 14:48:33: (cproxy.c.989) -1: release_downstream
2011-09-07 14:48:33: (cproxy.c.1381) check_downstream_config 1
2011-09-07 14:48:33: (memcached.c.3457) 55: drive_machine conn_closing
2011-09-07 14:48:33: (cproxy.c.669) <55 cproxy_on_close_downstream_conn
2011-09-07 14:48:33: (cproxy.c.3235) z_error, one.example.com:11211:default:(null):0, 1, 0, 2, 134114
2011-09-07 14:48:33: (cproxy.c.710) 55: skipping release dc in on_close_dc
2011-09-07 14:48:33: (memcached.c.556) <55 connection closed.
2011-09-07 14:48:33: (memcached.c.3457) 54: drive_machine conn_mwrite
2011-09-07 14:48:33: (memcached.c.3457) 54: drive_machine conn_mwrite
2011-09-07 14:48:33: (memcached.c.670) 54: going from conn_mwrite to conn_new_cmd
2011-09-07 14:48:33: (memcached.c.3457) 54: drive_machine conn_new_cmd
2011-09-07 14:48:33: (memcached.c.670) 54: going from conn_new_cmd to conn_waiting
2011-09-07 14:48:33: (memcached.c.3457) 54: drive_machine conn_waiting
2011-09-07 14:48:33: (memcached.c.670) 54: going from conn_waiting to conn_read
2011-09-07 14:48:33: (memcached.c.3457) 54: drive_machine conn_read
2011-09-07 14:48:33: (memcached.c.670) 54: going from conn_read to conn_parse_cmd
2011-09-07 14:48:33: (memcached.c.3457) 54: drive_machine conn_parse_cmd
2011-09-07 14:48:33: (cproxy_protocol_a.c.34) <54 cproxy_process_upstream_ascii quit
2011-09-07 14:48:33: (memcached.c.670) 54: going from conn_parse_cmd to conn_closing
2011-09-07 14:48:33: (memcached.c.3457) 54: drive_machine conn_closing
2011-09-07 14:48:33: (cproxy.c.551) <54 cproxy_on_close_upstream_conn
2011-09-07 14:48:33: (memcached.c.556) <54 connection closed.
2011-09-07 14:48:33: (cproxy.c.2250) wait_queue_timeout
2011-09-07 14:48:33: (cproxy.c.2264) wait_queue_timeout clearedJust to try some other ideas, I've also attempted adding -B ascii to the moxi startup and also tried making the S3 URL http://s3.amazonaws.com/pools/default/bucketsStreaming/default as well as http://s3.amazonaws.com/pools/default/saslBucketsStreaming. No success on all of them - AFAICT, moxi still attempted to SASL auth to memcached/Elasticache.
Tony
I'll have to let Steve jump back in, but I'm thinking that Moxi must use SASL when being configured with a URL...it expects this to be a Membase cluster.
Given that this works fine when you give Moxi a specific configuration, could you do something where you point Moxi at a config file, and have that file synced across all of your app nodes? Then a quick bounce of Moxi would be the only blip required when adding/removing servers.
Perry
That might be a "plan b", but harder to implement because an EC2 AutoScaling group can change from moment-to-moment. There are "right ways" of doing it with SNS, but since moxi already supports dynamic reconfiguration, it feels like it should be possible with memcached/Elasticache as well. I've attempted to dig through the source at https://github.com/membase/moxi to try to see where the key decisions are made, but I haven't been able to find it just yet.
Tony
Hi, moxi should be using SASL optionally, especially if you provided it a password.
What cmd-line are you using to launch moxi?
And, what's your current exact config file?
Cheers,
STeve
There is one other thing about moxi's dynamic reconfiguration support (that you might have already realized)...
The web-server needs to keep it's connection open and 'stream' any config changes to moxi. The embedded web-server in membase, for example, does this; so that the membase cluster manager/orchestrator can rapidly inform any moxi's (or other smart clients) of any cluster changes.
If that's not available, then moxi will repeatedly poll the web-server to see if there are changes. I really never liked this poll'ing here, but it was a pragmatic fall-back "Plan B" approach when all this was coming together. This polling is what you'd end up with by having the REST config json content served from S3 -- not optimal, but just something to keep in mind.
Cheers,
Steve
Steve,
Below is how I'm running moxi on an Amazon Linux AMI on EC2. It is a very vanilla setup - only edits are for the HTTP config file and to add logging/debugging. I also don't keep the EC2 instance around, so I'm terminating when I'm done testing and re-launching when someone replies to this thread ;-)
Tony
---8<---
1) install steps:
ssh ec2-user@ec2-aaa-bbb-ccc-ddd.compute-1.amazonaws.com
sudo bash -o vi
wget http://packages.couchbase.com/releases/1.7.1/moxi-server_x86_1.7.1.rpm
yum -y --nogpgcheck localinstall moxi-server_x86_1.7.1.rpm
2) configure steps:
mkdir -p /var/log/moxi
touch /var/log/moxi/moxi.log
chown -Rh moxi:moxi /var/log/moxi
cd /opt/moxi/etc
vi moxi-init.d
# edit startup line to be:
daemon "/bin/su -c \"$DAEMON -vvv -O /var/log/moxi/moxi.log -d -P $PIDFILE -Z $MOXI_CFG -z $MOXI_CLUSTER_CFG\" moxi"
vi moxi-cluster.cfg
# add after last commented line:
url=http://s3.amazonaws.com/mybucket/testconfig.json
3) start-up moxi:
service moxi-server start
The web-server needs to keep it's connection open and 'stream' any config changes to moxi. The embedded web-server in membase, for example, does this; so that the membase cluster manager/orchestrator can rapidly inform any moxi's (or other smart clients) of any cluster changes.
If that's not available, then moxi will repeatedly poll the web-server to see if there are changes. I really never liked this poll'ing here, but it was a pragmatic fall-back "Plan B" approach when all this was coming together. This polling is what you'd end up with by having the REST config json content served from S3 -- not optimal, but just something to keep in mind.
I hadn't realized that. Doing the math at 1 request per second for a year, that's going to be an additional cost of $31.54/yr in S3 GET charges ($0.01 per 10k) for each moxi instance. This might motivate me to find a pair of HTTP servers that I already have running on EC2 that I can utilize for the config but I think it might even be worth the S3 costs at the end of the day for small webserver clusters.
Tony
I just tested this with moxi-server_x86_1.7.1.1.rpm as well, same results:
2011-09-15 18:23:30: (cproxy.c.3085) 55: cproxy_on_connect_downstream_conn for one.example.com:11211:default:(null):0
2011-09-15 18:23:30: (cproxy.c.2801) cproxy_auth_downstream usr: default pwd: (0)
2011-09-15 18:23:30: (cproxy.c.2899) auth_downstream failure for default (81)
2011-09-15 18:23:30: (cproxy.c.3235) z_error, one.example.com:11211:default:(null):0, 1, 1, 1, 9388
2011-09-15 18:23:30: (cproxy.c.648) <55 cproxy_on_close_downstream_conn quit_server
Gah forgot the config file that lives at http://s3.amazonaws.com/mybucket/testconfig.json:
{ "name": "default", "nodeLocator": "ketama", "nodes": [ { "hostname": "one.example.com", "ports": { "proxy":11211,"direct":11211 }}, { "hostname": "two.example.com", "ports": { "proxy":11211,"direct":11211 }} ] }
bump? Any additional thoughts on this?
So if we put the moxi config on s3 ad documented above and generate signed url valid for 20 years it will happily pull the config as json right?
what happens if it fails to get a config on one of it's polls - does it just carry on with the one it has or do bad things happen?
It's all laid out in the documentation:
When the libconflate receives a proper REST response, it invokes moxi callbacks (so the dynamic re-configuration callbacks are happening on libconflate's dedicated thread).
Assuming the REST response wasn't proper (e.g. it failed) the callback for dynamic reconfiguration would not occur.
http://www.couchbase.com/docs/moxi-manual-1.7/moxi-internals-reconfig.html
Tony
P.S. It WILL happily pull the json configuration. However last time I tried, the ElastiCache version of memcached wasn't SASL enabled and moxi only attempting SASL connections to it. Thus moxi + S3 + Elasticache was not working properly.
... and if it is possible, what is the proper JSON to configure a cluster of 3 nodes, say one.example.com:11211, two.example.com:11211 and three.example.com:11211 - I haven't found too much documentation on this config format, only http://www.couchbase.org/wiki/display/membase/Developing+Membase+Clients... which provides some example, but it is for membase and not memcached