View Source

Here we store some pointers to things that we use to beat up membase.

h2. Test Clients

h1.

The [Testing cluster with load generator] page has a lot of information on test clients. We should probably have more.

h2. Test Servers

h1.

The [Couchbase Server Simulator] documentation describes how we can have a server that creates all kinds of faults and allows us to tests clients and proxies.

h2. Peripheral Test Tools

h1.

[The labrea project wiki|https://github.com/dustin/labrea/wiki] can interpose lua scripts against key system calls. This is an incredibly useful hook to simulate failure conditions. The labrea project site has the full documentation, but here are the basics:

Project source: [https://github.com/dustin/labrea|https://github.com/dustin/labrea]

On linux:

1. download and compile from [https://github.com/dustin/labrea/tarball/master] Or, 64-bit rpm available at: [http://github.com/downloads/dustin/labrea/labrea-0.1-1.x86_64.rpm]

2. extract the tarball to /usr/local/labrea

3. rename and edit the following scripts:

Add a test.lua script. For example:
{code}
[root@localhost memcached]# cat /usr/local/labrea/test.lua
function before_lseek(fd, offset, whence)
-- if math.random(1, 100) == 13 then
io.write(string.format("Slowing down a seek on fd=%d to %d (%d)\n",
fd, offset, whence))
usleep(1000000)
-- end
end

function before_read(...)
usleep(1000)
end

function before_write(...)
usleep(1000)
end
{code}

{code}
cd /opt/membase/bin/memcached/
mv memcached memcached.bin
{code}

create a new memcached script
{code}
#!/bin/sh

LB=/usr/local/labrea

exec $LB/labrea $LB/test.lua `dirname $0`/memcached.bin "$@"
{code}

Restart memcached. Keep in mind that if you slow down the disk seeks / reads, it will slow down the startup time of the process significantly.
{code}
killall -9 memcached
{code}

h3. For Machines

(describing using firewall rules to make machines appear to have plug pulls)

h3. For Processes

{{kill \-STOP <pid>}} will completely suspend a process, hanging anything trying to communicate with it synchronously. The process can be resumed with {{kill \-CONT <pid>}}

To slow a process down a bit, we can {{renice}} the process and run something CPU intensive that will greatly limit the CPU time available.

Another approach to slowing down the process would be to use {{ptrace}} to intercept it and make certain calls just take longer.

h3. For Networks

(someone dig up the documentation we had for the network simulation stuff we were using before)

h1. Testlink Tools

[testlink-cli|http://github.com/kbatten/testlink-cli] \- An example script and class to access testlink api through python. The current version allows you to execute a specific test.