<!-- 
RSS generated by JIRA (5.2.4#845-sha1:c9f4cc41abe72fb236945343a1f485c2c844dac9) at Thu May 23 08:19:07 CDT 2013

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary add field=key&field=summary to the URL of your request.
For example:
http://www.couchbase.com/issues/si/jira.issueviews:issue-xml/MB-7631/MB-7631.xml?field=key&field=summary
-->
<rss version="0.92" >
<channel>
    <title>Couchbase</title>
    <link>http://www.couchbase.com/issues</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>5.2.4</version>
        <build-number>845</build-number>
        <build-date>26-12-2012</build-date>
    </build-info>

<item>
            <title>[MB-7631] Bodies for deleted documents shouldn&apos;t be read (causes garbage data to be read)</title>
                <link>http://www.couchbase.com/issues/browse/MB-7631</link>
                <project id="10010" key="MB">Couchbase Server</project>
                        <description>Steps to reproduce:&lt;br/&gt;
&lt;br/&gt;
1.  Set up XDCR (observed when sending to ElasticSearch but not believe to be related to this)&lt;br/&gt;
2.  Create JSON document&lt;br/&gt;
3.  Observe that document is replicated as expected to remote side&lt;br/&gt;
&lt;br/&gt;
[{meta={id=2, rev=2-000015818d7d3c0a0000000000000000, expiration=0, flags=0}, base64=eyJjbGljayI6InRvIGVkaXQiLCJuZXcgaW4gMi4wIjoidGhlcmUgYXJlIG5vIHJlc2VydmVkIGZpZWxkIG5hbWVzIn0=}]&lt;br/&gt;
&lt;br/&gt;
in the _bulk_docs request the document appears to be JSON and the base64 data correctly represents the document&lt;br/&gt;
&lt;br/&gt;
4.  Delete the document&lt;br/&gt;
5.  Observe that the document deletion is replicated across to remote side, but with some unusual settings:&lt;br/&gt;
&lt;br/&gt;
[{meta={id=2, rev=3-000015818d7d3c0b0000000000000000, att_reason=non-JSON mode, expiration=0, flags=0, deleted=true}, base64=V1QBAOAAAENfbG9jYWwvdmJzdGF0ZXsiBQdoIjogImFjdGl2ZSIsICJjaGVja3BvaW50X2lkARsAMQEWXG1heF9kZWxldGVkX3NlcW5vIjogIjAifQ==}]&lt;br/&gt;
&lt;br/&gt;
Note that the document is indicated as binary (non-JSON mode) and that the the base64 encoded data appears to be some random memory on the server:&lt;br/&gt;
&lt;br/&gt;
echo &amp;quot;V1QBAOAAAENfbG9jYWwvdmJzdGF0ZXsiBQdoIjogImFjdGl2ZSIsICJjaGVja3BvaW50X2lkARsAMQEWXG1heF9kZWxldGVkX3NlcW5vIjogIjAifQ==&amp;quot; | base64 -D&lt;br/&gt;
WT?C_local/vbstate{&amp;quot;h&amp;quot;: &amp;quot;active&amp;quot;, &amp;quot;checkpoint_id\max_deleted_seqno&amp;quot;: &amp;quot;0&amp;quot;}&lt;br/&gt;
&lt;br/&gt;
</description>
                <environment>Observed on Mac and Linux platforms</environment>
            <key id="22057">MB-7631</key>
            <summary>Bodies for deleted documents shouldn&apos;t be read (causes garbage data to be read)</summary>
                <type id="1" iconUrl="http://www.couchbase.com/issues/images/icons/issuetypes/bug.png">Bug</type>
                                <priority id="2" iconUrl="http://www.couchbase.com/issues/images/icons/priorities/critical.png">Critical</priority>
                    <status id="5" iconUrl="http://www.couchbase.com/issues/images/icons/statuses/resolved.png">Resolved</status>
                    <resolution id="1">Fixed</resolution>
                    <security id="10011">Public</security>
                        <assignee username="FilipeManana">Filipe Manana</assignee>
                                <reporter username="mschoch">Marty Schoch</reporter>
                        <labels>
                    </labels>
                <created>Tue, 29 Jan 2013 18:40:41 -0600</created>
                <updated>Thu, 14 Feb 2013 12:28:35 -0600</updated>
                    <resolved>Thu, 14 Feb 2013 12:28:35 -0600</resolved>
                            <version>2.0</version>
                                <fixVersion>2.0.1</fixVersion>
                                <component>storage-engine</component>
                                <votes>0</votes>
                        <watches>9</watches>
                                                    <comments>
                    <comment id="48967" author="mschoch" created="Tue, 29 Jan 2013 18:42:39 -0600"  >Also, although the decoded data here is very text-like, in other tests the base64 data observed was just garbage.</comment>
                    <comment id="48968" author="junyi" created="Tue, 29 Jan 2013 18:46:33 -0600"  >Hi Aaron,&lt;br/&gt;
&lt;br/&gt;
When XDCR fetched deletion mutation from storage, XDCR would see some garbage in  document body as Marty described and it is marked as NON-JSON. This seems waster of bandwidth because destination side (either a regular XDCR receiver or ES cluster) just need the key and metadata, it does not need anything encoded in base64. &lt;br/&gt;
&lt;br/&gt;
Please be free assign to the right person if necessary. Thanks.&lt;br/&gt;
&lt;br/&gt;
&lt;br/&gt;
</comment>
                    <comment id="48969" author="junyi" created="Tue, 29 Jan 2013 18:47:33 -0600"  >add storage engine in component list since the garbage comes from storage. </comment>
                    <comment id="48977" author="dipti" created="Wed, 30 Jan 2013 00:51:06 -0600"  >is this something that we need to fix in 2.0.1? Do we run into the elasticsearch problems because of this? </comment>
                    <comment id="48978" author="mschoch" created="Wed, 30 Jan 2013 00:58:57 -0600"  >This does not cause any problem for Elasticsearch, when processing a delete we do not look at the binary flag or the body of the base64 data.  As Junyi mentioned, it may waste bandwidth transferring bytes thats are not used for anything.</comment>
                    <comment id="48979" author="aaron" created="Wed, 30 Jan 2013 01:21:59 -0600"  >Couchstore considers it &amp;quot;valid&amp;quot; for deleted items to have bodies, and will use whatever content the upstream user provided. It should be fine for XDCR to consider deleted items to be &amp;quot;empty&amp;quot; and send no body, though, as from the perspective of the rest of the components in the system, a deleted item is effectively empty.&lt;br/&gt;
&lt;br/&gt;
There may be some code in CouchDB/XDCR misinterpreting the empty body and reading some garbage from the beginning of the file erroneously though (Location 0 is considered to be &amp;quot;empty&amp;quot;), as I believe ep-engine correctly sets the body to empty when doing deletes. I will investigate. Do we know if this happens with *every* deleted item, or just some?</comment>
                    <comment id="48980" author="dipti" created="Wed, 30 Jan 2013 01:32:41 -0600"  >BTW, I forget, do we propagate expired documents? if yes, then for use cases in which we have TTL, this may be a huge overhead ? &lt;br/&gt;
&lt;br/&gt;
I don&amp;#39;t know if this is something we want to fix in 2.0.1 given the timing, perhaps 2.0.2? </comment>
                    <comment id="48982" author="mschoch" created="Wed, 30 Jan 2013 01:55:01 -0600"  >@aaron after we discussed the issue and theorized what was happening, the first doc I put through these steps exhibited this behavior.  i can&amp;#39;t say it always happens, but it has been very easy to reproduce.&lt;br/&gt;
&lt;br/&gt;
@dipti regarding &amp;quot;do we propagate expired documents?&amp;quot;  a good question the more i think about it.  while the remote cluster should expire the item on its own, when the expiration hits the disk on the source cluster, i would expect that docid/revision to go across in a _revs_diff request.  in the case of the elasticsearch plugin this would seem result in the doc/rev being considered missing (we already deleted it), since its missing, the _bulk_docs would then send the delete across the wire (introducing the overhead in your question)  but, i&amp;#39;m not sure thats actually how it works in practice</comment>
                    <comment id="49059" author="junyi" created="Wed, 30 Jan 2013 17:52:19 -0600"  >@Dipti, XDCR does replicate expired item as a deletion. But since the item can expire on its own on remote cluster, there is a race condition that no guarantee the deletion will be replicated to destination,  depending on  on which cluster, the expiration happens first.  For ES, it does not impact the correctness, but may cause some network bandwidth overhead. &lt;br/&gt;
&lt;br/&gt;
&amp;nbsp;</comment>
                    <comment id="49063" author="dipti" created="Wed, 30 Jan 2013 18:34:27 -0600"  >Bandwidth for sure, but also latency. We need to optimize for latency. &lt;br/&gt;
How hard is the fix? &lt;br/&gt;
&lt;br/&gt;
KV + XDCR (particularly for caching use case is very important) </comment>
                    <comment id="49064" author="mschoch" created="Wed, 30 Jan 2013 18:40:16 -0600"  >@Dipti I&amp;#39;m not sure I understand the part about latency.  If you&amp;#39;ve configured TTL on the Elasticsearch cluster, the expired document will be removed from the index within the &amp;quot;indices.ttl.interval&amp;quot; (default 60s).  The only other thing is that at some later time the document expiration will also come across the wire via XDCR.  This will involve both a _revs_diff and a _bulk_docs request.  The net effect of these is nothing (the document has already been removed from the index)  So what latency would be affected?</comment>
                    <comment id="49577" author="jin" created="Sun, 3 Feb 2013 23:47:39 -0600"  >Based on the above communication thread, there are two obvious things we can discuss here:&lt;br/&gt;
* overhead of replicating expired items (especially with the race condition mentioned above) - yes waste of bandwidth but can this affect maintaining consistent XDCR performance including latency when the number of expired items are huge?&lt;br/&gt;
* Aaron&amp;#39;s suggestion - XDCR to consider a deleted/expired item to be empty thus send no body over wire.&lt;br/&gt;
&lt;br/&gt;
We will discuss above two points and make a decision for feasible steps. Thanks.</comment>
                    <comment id="49578" author="jin" created="Sun, 3 Feb 2013 23:48:28 -0600"  >Assign it back to Junyi to follow up for next bug scrub meeting.</comment>
                    <comment id="49659" author="jin" created="Mon, 4 Feb 2013 15:35:44 -0600"  >Per bug scrubs, we will provide a fix that XDCR sends nobody across clusters for a deleted/expired item.</comment>
                    <comment id="49668" author="junyi" created="Mon, 4 Feb 2013 17:00:24 -0600"  >Talked to Damien and Jin briefly. &lt;br/&gt;
&lt;br/&gt;
XDCR just read all mutations from storage and try to send to remote side. XDCR itself does not create or parse the document body. All info XDCR need is the key and metadata, not the body of document.  The content in doc boy is totally beyond XDCR&amp;#39;s control.&lt;br/&gt;
&lt;br/&gt;
Looks like the &amp;quot;garbage in doc body&amp;quot; is persisted in storage because by Marty&amp;#39;s testcase, the garbage is something created in memory. So even without XDCR, we may waste potentially a lot of storage space by persisting a bunch of garbage for deleted items. XDCR just expose this issue.&lt;br/&gt;
&lt;br/&gt;
On XDCR side, ofcourse we can parse the doc body before sending it to avoid the garbage, but IHMO it is NOT a good fix, because it just &amp;quot;hides&amp;quot; the issue instead of fixing it. Parsing doc body is not supposed to happen in XDCR layer.&lt;br/&gt;
&lt;br/&gt;
Per Aaron&amp;#39;s comment, &amp;quot;ep-engine correctly sets the body to empty when doing deletes&amp;quot;,  therefore I would suggest to CouchDB/CouchStore team look at this issue first why we read garbage for deleted item.&lt;br/&gt;
&lt;br/&gt;
</comment>
                    <comment id="49669" author="junyi" created="Mon, 4 Feb 2013 17:03:37 -0600"  >remove xdcr component</comment>
                    <comment id="49676" author="junyi" created="Mon, 4 Feb 2013 17:59:06 -0600"  >Per comment&amp;#39;s from Damien, it is also possible a bug in ep_engine which messed up the doc_body ptr of deleted item.  Add ep_engine in the component list.</comment>
                    <comment id="50000" author="dipti" created="Mon, 11 Feb 2013 02:16:54 -0600"  >is this a small change that we can fix in 2.0.1? </comment>
                    <comment id="50021" author="junyi" created="Mon, 11 Feb 2013 11:46:25 -0600"  >Dipti,&lt;br/&gt;
&lt;br/&gt;
My feeling is if we can find the culprit, it may be a small fix. But we need to figure out where the problem is. As I mentioned above, we incorrectly store some garbage of deleted item.  XDCR just exposed that. &lt;br/&gt;
&lt;br/&gt;
IMHO, ep_engine should first look at this issue to see if there is any dangling pointer of doc_body. If not, couchdb and couchstore will need to investigate. Thanks!</comment>
                    <comment id="50029" author="jin" created="Mon, 11 Feb 2013 12:21:50 -0600"  >EP Engine (Jin) will take a look at this issue this week. As Junyi stated earlier if this turns out to be an ep engine issue, this should be a small fix. Thanks.</comment>
                    <comment id="50035" author="farshid" created="Mon, 11 Feb 2013 12:27:11 -0600"  >Jin/Dipti/Yaseen,&lt;br/&gt;
&lt;br/&gt;
given that this is not a regression from 2.0 and it does not impact xdcr functionality and does not cause huge bandwidth issue i recommend moving this to 2.0.2</comment>
                    <comment id="50038" author="dipti" created="Mon, 11 Feb 2013 12:41:03 -0600"  >I think it does cause a huge bandwidth issue for large documents and expirations. This is particularly the case for ad targeting. Also, many people have complained about unusual disk growth with TTL, I suspect this may be the issue. In addition to storing tombstones, we are also storing data in couchstore. questions that will help us decide are: &lt;br/&gt;
&lt;br/&gt;
- do we store extra data for every deleted key? &lt;br/&gt;
- is this also the case for expired items? &lt;br/&gt;
- is it a fixed amount of additional data that gets stored? how much per document? &lt;br/&gt;
&lt;br/&gt;
I&amp;#39;m not so concerned about deletes since typically very small percentage of items are deleted &amp;lt;1%. But more concerned about expired items. </comment>
                    <comment id="50039" author="dipti" created="Mon, 11 Feb 2013 12:42:39 -0600"  >Forum post just for reference: &lt;a href=&quot;http://www.couchbase.com/forums/thread/persistence-becoming-issue-can-it-be-disabled&quot;&gt;http://www.couchbase.com/forums/thread/persistence-becoming-issue-can-it-be-disabled&lt;/a&gt; </comment>
                    <comment id="50045" author="jin" created="Mon, 11 Feb 2013 13:11:20 -0600"  >I agree we will still continue to look into this if there is a small fix. We just didn&amp;#39;t have enough engineering cycles for debugging this, but will do so this week.&lt;br/&gt;
Please see my comments below for Dipti&amp;#39;s questions above;&lt;br/&gt;
&lt;br/&gt;
- do we store extra data for every deleted key? yes on src node but not dst node (Junyi please correct me if wrong)&lt;br/&gt;
- is this also the case for expired items? yes since ep engine handles both explicitly deleted and expired items with the same disk write operation (del).&lt;br/&gt;
- is it a fixed amount of additional data that gets stored? it appears to be not much but random according to Junyi&amp;#39;s latest debugging&lt;br/&gt;
&lt;br/&gt;
Again, will do a little bit of more engineering debugging and make a final call. Thanks for practical suggestion from QE though.</comment>
                    <comment id="50050" author="junyi" created="Mon, 11 Feb 2013 13:27:39 -0600"  >Jin,&lt;br/&gt;
&lt;br/&gt;
Thanks a lot for looking into this.  &lt;br/&gt;
&lt;br/&gt;
&lt;br/&gt;
Dipti,&lt;br/&gt;
&lt;br/&gt;
Here is my take on your questions&lt;br/&gt;
&lt;br/&gt;
- do we store extra data for every deleted key? &lt;br/&gt;
It is fairly easy to reproduce, but I am not sure if it is for &amp;quot;every&amp;quot; deleted key&lt;br/&gt;
&lt;br/&gt;
- is this also the case for expired items?&lt;br/&gt;
Expired time will end up a del op from ep_engine, if the code path is the same, probably it will be a victim of this bug too.&lt;br/&gt;
&lt;br/&gt;
- is it a fixed amount of additional data that gets stored? how much per document? &lt;br/&gt;
The garbage seems from a random segment of memory (possibly due to some dangling pointer), if it is the case, we store a random chunk of data in doc_body, the size could be variant.&lt;br/&gt;
&lt;br/&gt;
&amp;nbsp;&lt;br/&gt;
</comment>
                    <comment id="50333" author="jin" created="Wed, 13 Feb 2013 16:47:10 -0600"  >I finally did some debugging this issue and found that ep engine is handling deleted items correctly. It never passes doc body but NULL to couchstore for deleting docs.&lt;br/&gt;
However, with Aaron&amp;#39;s help, it appears to be that open_doc_int() in couch_db.erl still tries to read the doc body for a deleted item instead of trasting it as a empty doc. Assign it to couchdb team (Filipe) for further debugging and a fix. Thanks.&lt;br/&gt;
&amp;nbsp;</comment>
                    <comment id="50335" author="jin" created="Wed, 13 Feb 2013 16:49:58 -0600"  >Hi Filipe, it looks like a fix is needed in couchdb. If appropriate, please assign it to Aaron as he know the root cause as well. Thanks.</comment>
                    <comment id="50336" author="FilipeManana" created="Wed, 13 Feb 2013 16:55:32 -0600"  >Is it that hard to add to the replicator code that doesn&amp;#39;t read bodies for deleted items, or make couch_db:open_doc/2 not read bodies for deleted items?</comment>
                    <comment id="50341" author="aaron" created="Wed, 13 Feb 2013 17:12:23 -0600"  >&lt;a href=&quot;http://review.couchbase.org/#/c/24586/&quot;&gt;http://review.couchbase.org/#/c/24586/&lt;/a&gt; Here.</comment>
                    <comment id="50344" author="junyi" created="Wed, 13 Feb 2013 17:20:51 -0600"  >Filipe,&lt;br/&gt;
&lt;br/&gt;
It is not hard but replicator is not supposed to do that, and more importantly, that does not solve the problem. You still store a bunch of garbage on disk, which caused more serious disk usage issue as Dipit mentioned.</comment>
                    <comment id="50345" author="aaron" created="Wed, 13 Feb 2013 17:25:25 -0600"  >We aren&amp;#39;t storing garbage, I think. We&amp;#39;ve only been _reading_ garbage.</comment>
                    <comment id="50346" author="FilipeManana" created="Wed, 13 Feb 2013 17:27:43 -0600"  >From what I understood from Jin&amp;#39;s comments, ep-engine/couchstore side are doing the right thing, that is, not persisting garbage to disk for deleted documents.&lt;br/&gt;
It&amp;#39;s only a problem of couchdb not knowing that it shouldn&amp;#39;t read bodies for deleted documents (unlike in Apache CouchDB).&lt;br/&gt;
&lt;br/&gt;
Issues like this could well be solved by people who work on the replicator imho.</comment>
                    <comment id="50348" author="jin" created="Wed, 13 Feb 2013 17:36:04 -0600"  >Aaron/Filipe/Junyi, thanks for your input and quick help here. It looks like we can address this issue fairly simple, wether from the replicator or couchdb sides. If I may I will leave this up to your decision. However, please let me know since it is fairly simple, good fix wether we should consider push it to 2.0.1 or not. Thanks.</comment>
                    <comment id="50351" author="junyi" created="Wed, 13 Feb 2013 17:38:24 -0600"  >It is good news that we do not actually store garbage. If the couchdb read some garbage (as Aaron pointed out), then couchdb should fix it. &lt;br/&gt;
&lt;br/&gt;
Replicator is just one consumer of this API and relies on couchdb to fetch all mutations,  and IMHO it should not look into doc body. Put some code in replicator does not solve the problem, other caller of couchdb:open_doc() will still read garbage. &lt;br/&gt;
&amp;nbsp;</comment>
                    <comment id="50352" author="junyi" created="Wed, 13 Feb 2013 17:42:14 -0600"  >based on inputs from Aaron, change the bug title</comment>
                    <comment id="50353" author="jin" created="Wed, 13 Feb 2013 17:58:08 -0600"  >OK, someone changed it to 2.0.1 fix. Everyone OK with it?</comment>
                    <comment id="50354" author="junyi" created="Wed, 13 Feb 2013 18:00:19 -0600"  >I am ok with that. But the fix is in couchdb so Filipe should make the final decision. &lt;br/&gt;
</comment>
                    <comment id="50356" author="FilipeManana" created="Wed, 13 Feb 2013 18:03:19 -0600"  >Yes, fine for me.</comment>
                    <comment id="50399" author="thuan" created="Thu, 14 Feb 2013 04:27:50 -0600"  >Integrated in github-couchdb-preview #563 (See [&lt;a href=&quot;http://qa.hq.northscale.net/job/github-couchdb-preview/563/&quot;&gt;http://qa.hq.northscale.net/job/github-couchdb-preview/563/&lt;/a&gt;])&lt;br/&gt;
&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&lt;a href=&quot;http://www.couchbase.com/issues/browse/MB-7631&quot; title=&quot;Bodies for deleted documents shouldn&amp;#39;t be read (causes garbage data to be read)&quot;&gt;&lt;strike&gt;MB-7631&lt;/strike&gt;&lt;/a&gt; Interpret bp of 0 as empty body (Revision c086371a81e43021966620c469134e08530fe38a)&lt;br/&gt;
Revert &amp;quot;&lt;a href=&quot;http://www.couchbase.com/issues/browse/MB-7631&quot; title=&quot;Bodies for deleted documents shouldn&amp;#39;t be read (causes garbage data to be read)&quot;&gt;&lt;strike&gt;MB-7631&lt;/strike&gt;&lt;/a&gt; Interpret bp of 0 as empty body&amp;quot; (Revision bfd853e859ab961f8ee6c8fb8b38e667b3dd0a9c)&lt;br/&gt;
&lt;br/&gt;
&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;Result = SUCCESS&lt;br/&gt;
Filipe David Borba Manana : &lt;br/&gt;
Files : &lt;br/&gt;
* src/couchdb/couch_db.erl&lt;br/&gt;
&lt;br/&gt;
Filipe David Borba Manana : &lt;br/&gt;
Files : &lt;br/&gt;
* src/couchdb/couch_db.erl&lt;br/&gt;
</comment>
                    <comment id="50417" author="thuan" created="Thu, 14 Feb 2013 10:32:40 -0600"  >Integrated in github-couchdb-preview #564 (See [&lt;a href=&quot;http://qa.hq.northscale.net/job/github-couchdb-preview/564/&quot;&gt;http://qa.hq.northscale.net/job/github-couchdb-preview/564/&lt;/a&gt;])&lt;br/&gt;
&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&lt;a href=&quot;http://www.couchbase.com/issues/browse/MB-7631&quot; title=&quot;Bodies for deleted documents shouldn&amp;#39;t be read (causes garbage data to be read)&quot;&gt;&lt;strike&gt;MB-7631&lt;/strike&gt;&lt;/a&gt; Interpret bp of 0 as empty body (Revision 8e961880696e0398229e7a837344b73053beed84)&lt;br/&gt;
&lt;br/&gt;
&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;Result = SUCCESS&lt;br/&gt;
Filipe David Borba Manana : &lt;br/&gt;
Files : &lt;br/&gt;
* src/couchdb/couch_db.erl&lt;br/&gt;
</comment>
                    <comment id="50427" author="jin" created="Thu, 14 Feb 2013 12:28:35 -0600"  >&lt;a href=&quot;http://review.couchbase.org/#/c/24586/&quot;&gt;http://review.couchbase.org/#/c/24586/&lt;/a&gt; merged.</comment>
                </comments>
                    <attachments>
                </attachments>
            <subtasks>
        </subtasks>
                <customfields>
                                                                        <customfield id="customfield_10180" key="com.atlassian.jira.ext.charting:firstresponsedate">
                <customfieldname>Date of First Response</customfieldname>
                <customfieldvalues>
                    <customfieldvalue>Tue, 29 Jan 2013 18:46:33 -0600</customfieldvalue>

                </customfieldvalues>
            </customfield>
                                                                                                                                                                                                            <customfield id="customfield_10081" key="com.pyxis.greenhopper.jira:gh-global-rank">
                <customfieldname>Rank</customfieldname>
                <customfieldvalues>
                    <customfieldvalue>3033</customfieldvalue>
                </customfieldvalues>
            </customfield>
                                                                                                                                                <customfield id="customfield_10052" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                <customfieldname>Sprint Status</customfieldname>
                <customfieldvalues>
                        <customfieldvalue key="10027"><![CDATA[Current Sprint]]></customfieldvalue>

                </customfieldvalues>
            </customfield>
                                                                <customfield id="customfield_10181" key="com.atlassian.jira.ext.charting:timeinstatus">
                <customfieldname>Time In Status</customfieldname>
                <customfieldvalues>
                    
                </customfieldvalues>
            </customfield>
                                                </customfields>
    </item>
</channel>
</rss>