Decompressing inside a view

juan_musacchio · February 27, 2014, 7:30pm

Hi,

Currently I’m working in a project where we use Couchbase as a caching layer server.
The thing is that we are storing big json documents (between 1MB and 2MB) and based on the caching layer section of Couchbase documentation, those documents that are frequently queried/accesed are kept in RAM to avoid Disk swap.
I know that I have several options like stripping the JSON, removing data that we may not need, but involves tracking those properties that we remove and code changes.
I decided to investigate how to perform client side compression using Couchbase Java SDK (I created my own Transcoder) and performed ZLIB compression using DeflaterOutputStream. I tried it with a 100Kb document and was reduced up to 3.5Kb (zlib compression + base64 encoding).
The only problematic thing as we know are the Couchbase views that expect doc in a JSON format, so I decided to use a minified version of https://github.com/beatgammit/deflate-js/blob/master/lib/rawinflate.js inside a view to test the inflate process (previous I had to call decodeBase64 Couchbase built in function).
After all this research/testing work, I could generate a view index based on a compressed document.

My questions are:

Is it worth to do all this compression/decompression work when documents are big or Couchbase can handle it without problem? I know that depends on RAM sizing for node and bucket and another config parameters.
If I still want to go with the compression approach, inflating in the view is gonna be a problem? I know that indexes generation are incremental and will depend on the inflate javascript function also, but a first index creation will inflate all documents for a specific view.

Thanks!
Juanma

prasad_karkera · October 3, 2014, 9:10pm

I am sorry I cannot help you with your question but I have a question for you

I am trying to use the same library but its not inflating my data. I know that couchbase encodes the data as base64 too. I tried to decode it and then pass it on to inflate as well. But still it doesnt work. Can you tell me how did you get inflate to work?

//Added the js file you mentioned here

var data = inflate(doc);
emit(meta.id, data);

juan_musacchio · October 4, 2014, 10:35pm

Hi,

First of all are you storing JSON or serialzed java objects? the thing is that the default transcoder (SerializingTranscoder) doesn’t compress json strings, so what I did was to create my own transcoder to be able to use deflate algorithm (the default transcoder uses GZIP algorithm) and also compress json strings. Then you need to override the default transcoder (CouchbaseConnectionFactoryBuilder).
Heres is the transcoder code

import java.io.ByteArrayOutputStream; import java.io.IOException; import java.util.Date; import java.util.zip.Deflater; import java.util.zip.DeflaterOutputStream;

import net.spy.memcached.CachedData;
import net.spy.memcached.compat.CloseUtil;
import net.spy.memcached.transcoders.SerializingTranscoder;
import net.spy.memcached.transcoders.TranscoderUtils;
import net.spy.memcached.util.StringUtils;

/**

Transcoder that serializes and compresses objects.
*/
public class CouchbaseCompressionTranscoder extends SerializingTranscoder {

// General flags
static final int SERIALIZED = 1;
static final int COMPRESSED = 2;

// Special flags for specially handled types.
static final int SPECIAL_BOOLEAN = (1 << 8);
static final int SPECIAL_INT = (2 << 8);
static final int SPECIAL_LONG = (3 << 8);
static final int SPECIAL_DATE = (4 << 8);
static final int SPECIAL_BYTE = (5 << 8);
static final int SPECIAL_FLOAT = (6 << 8);
static final int SPECIAL_DOUBLE = (7 << 8);
static final int SPECIAL_BYTEARRAY = (8 << 8);

private final TranscoderUtils tu = new TranscoderUtils(true);

/*

(non-Javadoc)
@see net.spy.memcached.Transcoder#encode(java.lang.Object)
*/
public CachedData encode(Object o) {
byte[] b = null;
int flags = 0;
if (o instanceof String) {
b = encodeString((String) o);
if (StringUtils.isJsonObject((String) o)) {
if (b.length > compressionThreshold) {
byte[] compressed = compress(b);
if (compressed.length < b.length) {
b = compressed;
flags |= COMPRESSED;
}
}
```
 return new CachedData(flags, b, getMaxSize());
```
}
} else if (o instanceof Long) {
b = tu.encodeLong((Long) o);
flags |= SPECIAL_LONG;
} else if (o instanceof Integer) {
b = tu.encodeInt((Integer) o);
flags |= SPECIAL_INT;
} else if (o instanceof Boolean) {
b = tu.encodeBoolean((Boolean) o);
flags |= SPECIAL_BOOLEAN;
} else if (o instanceof Date) {
b = tu.encodeLong(((Date) o).getTime());
flags |= SPECIAL_DATE;
} else if (o instanceof Byte) {
b = tu.encodeByte((Byte) o);
flags |= SPECIAL_BYTE;
} else if (o instanceof Float) {
b = tu.encodeInt(Float.floatToRawIntBits((Float) o));
flags |= SPECIAL_FLOAT;
} else if (o instanceof Double) {
b = tu.encodeLong(Double.doubleToRawLongBits((Double) o));
flags |= SPECIAL_DOUBLE;
} else if (o instanceof byte[]) {
b = (byte[]) o;
flags |= SPECIAL_BYTEARRAY;
} else {
b = serialize(o);
flags |= SERIALIZED;
}
assert b != null;
if (b.length > compressionThreshold) {
byte[] compressed = compress(b);
if (compressed.length < b.length) {
getLogger().debug(“Compressed %s from %d to %d”,
o.getClass().getName(), b.length, compressed.length);
b = compressed;
flags |= COMPRESSED;
} else {
getLogger().info(“Compression increased the size of %s from %d to %d”,
o.getClass().getName(), b.length, compressed.length);
}
}
return new CachedData(flags, b, getMaxSize());
}

@Override
protected byte[] compress(byte[] in) {
if (in == null) {
throw new NullPointerException(“Can’t compress null”);
}
ByteArrayOutputStream bos = new ByteArrayOutputStream();
DeflaterOutputStream def = null;
try {
// create deflater without header
def = new DeflaterOutputStream(bos, new Deflater(Deflater.DEFAULT_COMPRESSION, true));
def.write(in);
} catch (IOException e) {
throw new RuntimeException(“IO exception compressing data”, e);
} finally {
CloseUtil.close(def);
CloseUtil.close(bos);
}
byte[] rv = bos.toByteArray();
getLogger().debug(“Compressed %d bytes to %d”, in.length, rv.length);
return rv;
}
}

Then put this script in your view:

// data inflated
var data = inflate.inflate(decodeBase64(doc));

// if it is a json you can parse it to access properties
var json = JSON.parse(String.fromCharCode.apply(String, data));

// emit data or json, is not a good practice to emit the json as value, just use it for testing
emit(meta.id, data);
emit(meta.id, json);
}

Send me an email if you want the files.

prasad_karkera · October 5, 2014, 7:39pm

Thanks so much for the code. Unfortunately it didn’t work for me.

To answer your question, I am not using JSON. Its serialized php objects.

To give you a background, I am working on an existing PHP project where it uses couchbase to store session data. The data stored, is not JSON and it is compressed using ZLIB. I am trying to write a view in which I want to uncompress the data and parse it. When I tried your code the output I get is just ‘[ ]’. I guess probably because you are using GZIP and my compression method is ZLIB. Any thoughts?

Also, did you get answers/benchmark results to your questions you mentioned in here?

Thanks a lot for your help.

juan_musacchio · October 5, 2014, 8:54pm

Hi

Im currently using Java Deflater Class

new Deflater(Deflater.DEFAULT_COMPRESSION, true)

and the constructor java doc says:

/**
* Creates a new compressor using the specified compression level.
* If ‘nowrap’ is true then the ZLIB header and checksum fields will
* not be used in order to support the compression format used in
* both GZIP and PKZIP.
* @param level the compression level (0-9)
* @param nowrap if true then use GZIP compatible compression
*/
public Deflater(int level, boolean nowrap) {

The implementation I’m using (DeflaterOutputStream) differs on GZIPOutputStream in that is not writing gzip header and CRC.
Didn’t get any benchmark, maybe you can calculate it tracking a timestamp before the inflate and base64 calls and after them so you can emit the result in an index to see how much it takes for each document.

prasad_karkera · October 5, 2014, 9:06pm

Ah! I will have to figure out a way to decompress. Thanks for your help.