How can I join 3 documents using LET

Hi Guys,

 Need some help, how can I join 3 documents using LET or Join that is optimized and easy to manage.

These are my documents:

// product docs

[
    {
    "id": product::0793",
    "name": “product01”
  },
  {
    "id": "product::0018",
    "name": "product03”
  },
  {
    "id": "product::0054",
    "name": "product02”
  },
  {
    "id": "product::0020",
    "name": "product04”
  },
]

// product-purchased

[
   {
    	"product": "product::0793",
  	"location": "location::6099247",
  	"period": 30,
  	"type": "product-purchased",
 	 "hasPurchased": 1
   },
  {
 	"product": "product::0793",
  	"location": “location::914779A",
  	"period": 30,
  	"type": "product-purchased",
 	 "hasPurchased": 1
  },
  {
  	"product": "product::0054",
  	"location": "location::6099247",
  	"period": 30,
  	"type": "product-purchased",
  	"hasPurchased": 0
  }
]

// product-sales-average

[
  {
  	"product": "product::0793",
  	"entityId": "location::6099247",
  	"periodType": “month,
  	"averageSales": 1.73,
 	 "averageQtyShipped": 0.25,
 	 "type": "product-sales-average",
  },
  {
  	"product": "product::0054",
  	"entityId": "location::6099247",
  	"periodType": “month,
  	"averageSales": 1.73,
 	 "averageQtyShipped": 0.25,
 	 "type": "product-sales-average",
  }
]

Use ANSI JOIN https://blog.couchbase.com/ansi-join-support-n1ql/

Or is there way we could use LET ? I found Let more manageable :slight_smile:

If the queries are independent you can use. If you need JOIN LET query becomes correlated subquery . At present correlated subqueries requires USE KEYS.

If you have JOIN query post it. Will help in optimized or if possible to convert to LET.
The input (duplicate records) and output doesn’t give how you are achieving multiple to single document

SELECT d.*, ps AS salesAverage
FROM (SELECT p.*, pp AS purchased
                 FROM default AS p
                 LEFT NEST default AS pp 
                 ON p.id = pp.product AND pp.type = "product-purchased"
                 WHERE p.type  = "product") AS d
LEFT  NEST default AS ps
ON p.id = ps.product AND ps.type = "product-sales-average";

Hi @vsr1,
Thank you for the immediate response… :slight_smile:
Sorry, for the input(duplicate records) I’ve updated my first post for the 3 documents. My bad.

So, my expected result something like this:

[
   {
    	"id": product::0793",
    	"name": “product01”,
	“purchased”: {
    	    "product": "product::0793",
  	    "location": "location::6099247",
  	    "period": 30,
  	    "type": "product-purchased",
 	    "hasPurchased": 1
         },
        “average”: {
  	    "product": "product::0793",
  	    "entityId": "location::6099247",
  	    "periodType": “month,
  	    "averageSales": 1.73,
 	    "averageQtyShipped": 0.25,
 	   "type": "product-sales-average",
        }
   },
   {
    	"id": "product::0018",
    	"name": "product03”,
   },
   {
    	"id": "product::0054",
    	"name": "product02”
        “purchased”: {
             "product": "product::0054",
  	     "location": "location::6099247",
  	     "period": 30,
  	      "type": "product-purchased",
  	      "hasPurchased": 0
        },
        “average”: {
  	     "product": "product::0054",
  	     "entityId": "location::6099247",
  	     "periodType": “month,
  	     "averageSales": 1.73,
 	     "averageQtyShipped": 0.25,
 	     "type": "product-sales-average",
        }
   },
   {
    	"id": "product::0020",
    	"name": "product04”
   },
]

The reason, I was doubtful of using JOIN is I just want join the documents ‘product-purchased’ and ‘product-sales-average’ to ‘product’ docs that satisfies the product id. And retained all other products that are not referenced.

Also, is it possible the properties of “purchased” and “average” will be use on WHERE and ORDER clause. For example I can ORDER purchased.hasPurchased ASC

OR is it easier to just flatten the “purchased” and “average” properties? like this:

"product": "product::0054",
"entityId": "location::6099247",
 "periodType": “month,
   "averageSales": 1.73,
  "averageQtyShipped": 0.25,
  "type": "product-sales-average",

Please help… :slight_smile:

// product-purchased

“product”: “product::0793” has two (it can be more) which one you choose In flatten ??

SELECT p.*, pp AS purchased, ps AS salesAverage
FROM default AS p
LEFT JOIN default AS pp ON p.id = pp.product AND pp.type = "product-purchased"
LEFT JOIN default AS ps p.id = ps.product AND pp.location = ps.entityId AND ps.type = "product-sales-average"
WHERE p.type  = "product" AND p.id IS NOT NULL
ORDER BY purchased.hasPurchased

Hi @vsr1,

Cool the query seems to work :slight_smile: the my problem here now is the query seems to run very slow :frowning:
Even though, I just LIMIT it to 1.

FYI, number of docs I have.
“product” -> 653,
“product-purchased” -> 169,
“product-sales-average” -> 145848,

Can you give some help or tips to optimized ? :frowning: really appreciate you help.

Here’s my query.

SELECT 
  meta(p).id,
  p.name,
  p.rank,
  pp AS purchased,
  ps AS average
FROM `bucket` AS p
LEFT JOIN `bucket` AS pp 
  ON META(p).id = pp.product
  AND pp.location = “location::6099247"
  AND pp.type = "product-purchased"

LEFT JOIN `bucket` AS ps 
  ON META(p).id = ps.product
  AND ps.periodType = "month"
  AND ps.type = "product-sales-average"
  AND ps.entityId = "location::6099247"

WHERE p.type = "product"

ORDER BY purchased.hasPurchased ASC, p.rank.level ASC, p.rank.position ASC

LIMIT 1

Once you have ORDER BY it must produce all possible values (irrespective of LIMIT) because last value can be be first in the order.

By removing ORDER BY it will produce very quick.

Ok… but ORDER BY is really needed, for the sorting functionality. ORDER BY is needed in my query :frowning:

Hi @vsr1
perhaps you could give some tips on how to index it or create index on the query. :slight_smile:

CREATE INDEX ix1 ON  `bucket` (name, rank) WHERE type = "product";
CREATE INDEX ix2 ON  `bucket` (product, location) WHERE type = "product-purchased";
CREATE INDEX ix3 ON  `bucket` (product, entityId, periodType ) WHERE type = "product-sales-average";

 SELECT 
  meta(p).id,
  p.name,
  p.rank,
  pp AS purchased,
  ps AS average
FROM `bucket` AS p
LEFT JOIN `bucket` AS pp 
  ON META(p).id = pp.product
  AND pp.location = “location::6099247"
  AND pp.type = "product-purchased"

LEFT JOIN `bucket` AS ps 
  ON META(p).id = ps.product
  AND ps.periodType = "month"
  AND ps.type = "product-sales-average"
  AND ps.entityId = "location::6099247"
WHERE p.type = "product" AND p.name IS NOT NULL
ORDER BY purchased.hasPurchased ASC, p.rank.level ASC, p.rank.position ASC
LIMIT 1

If you are willing to change your order you can consider this https://blog.couchbase.com/offset-keyset-pagination-n1ql-query-couchbase/
AS you doing LEFT JOIN take fixed number of documents from LEFT most one and then project all the join results with required sort then use key set pagination

start $id “” , next iteration give $id as last value

CREATE INDEX ix1 ON  `bucket` (META().id, name, rank) WHERE type = "product";
CREATE INDEX ix2 ON  `bucket` (product, location) WHERE type = "product-purchased";
CREATE INDEX ix3 ON  `bucket` (product, entityId, periodType ) WHERE type = "product-sales-average";

SELECT  p.*, pp AS purchased, ps AS average

FROM (SELECT meta(p1).id, p1.name, p1.rank
      FROM `bucket` AS p1
      WHERE p.type = "product" AND meta(p1).id > $id
      LIMIT 10) AS p
LEFT JOIN `bucket` AS pp ON p.id = pp.product
                         AND pp.location = "location::6099247"
                         AND pp.type = "product-purchased"
LEFT JOIN `bucket` AS ps ON p.id = ps.product
                         AND ps.periodType = "month"
                         AND ps.type = "product-sales-average"
                         AND ps.entityId = "location::6099247"
ORDER BY p.id, purchased.hasPurchased ASC, p.rank.level ASC, p.rank.position ASC

https://index-advisor.couchbase.com/indexadvisor/#1

Hi @vsr1
Thank you for you immediate response :slight_smile:

I have error creating the index:

CREATE INDEX ix1 ON  `bucket` (META().id, name, rank) WHERE type = "product";
CREATE INDEX ix2 ON  `bucket` (product, location) WHERE type = "product-purchased";
CREATE INDEX ix2 ON  `bucket` (product, entityId, periodType ) WHERE type = "product-sales-average";
  {
    "_sequence_num": 3,
    "_sequence_query": "\nCREATE INDEX ix2 ON  `bucket` (product, entityId, periodType ) WHERE type = \"product-sales-average\";",
    "_sequence_query_status": 409,
    "_sequence_result": [
      {
        "code": 4300,
        "msg": "The index ix2 already exists.",
        "query": "\nCREATE INDEX ix2 ON  `bucket` (product, entityId, periodType ) WHERE type = \"product-sales-average\";"
      }
    ]
  }

Change index name to ix3

Hi @vsr1
Thanks man!.. I will check this all out. thank you for your help :slight_smile:

Hi @vsr1,
I already apply the indexes you advice me to create and it works great. The performance of the query with 145K documents was optimized. Also, the query that you provided works well. Thank you

I have a little bit problem with the current query I have:

 SELECT 
  meta(p).id,
  p.name,
  p.rank,
  pp AS purchased,
  ps AS average
FROM `bucket` AS p

LEFT JOIN `bucket` AS pp 
  ON META(p).id = pp.product
  AND pp.location = “location::6099247"
  AND pp.type = "product-purchased"

LEFT JOIN `bucket` AS ps 
  ON META(p).id = ps.product
  AND ps.periodType = "month"
  AND ps.type = "product-sales-average"
  AND ps.entityId = "location::6099247"

WHERE p.type = "product" AND p.name IS NOT NULL
ORDER BY purchased.hasPurchased ASC, p.rank.level ASC, p.rank.position ASC
LIMIT 1

So, my problem right now, is how can I add another JOIN to my current query. with an Array of entityId:

 [
    'territory::11234',
    'territory::0045',
    'territory::0036'
 ]

This is the query I’ve been trying to add:

LEFT JOIN `bucket` AS territory 
  ON META(p).id = territory.product
  AND territory.periodType = "month"
  AND territory.type = "product-sales-average"
  AND ANY v IN [ 'territory::11234', 'territory::0045', 'territory::0036'] SATISFIES v =  territory.entityId END

This query seems to work, but it doesn’t form an array of territory
instead, it creates another objects adding to product lists :frowning:

// so the expected result I want to achieve it this:

[
   {
    	"id": product::0793",
    	"name": “product01”,
	“purchased”: {
    	    "product": "product::0793",
  	    "location": "location::6099247",
  	    "period": 30,
  	    "type": "product-purchased",
 	    "hasPurchased": 1
         },
        “average”: {
  	    "product": "product::0793",
  	    "entityId": "location::6099247",
  	    "periodType": “month,
  	    "averageSales": 1.73,
 	    "averageQtyShipped": 0.25,
 	   "type": "product-sales-average",
        },
       "territory": [
           {
  	       "product": "product::0793",
  	       "entityId": "territory::11234",
  	       "periodType": “month,
  	       "averageSales": 1.73,
 	       "averageQtyShipped": 0.25,
 	       "type": "product-sales-average",
           },
          {
  	       "product": "product::0793",
  	       "entityId": "territory::0045'",
  	       "periodType": “month,
  	       "averageSales": 11.73,
 	       "averageQtyShipped": 10.25,
 	       "type": "product-sales-average",
           },
        ]
   },
   {
    	"id": "product::0018",
    	"name": "product03”,
   },
   {
    	"id": "product::0020",
    	"name": "product04”
   },
]

Another problem with this result is I can’t do GROUP BY, because I want to get AVG() for averageSales and averageQtyShipped GROUP BY product. So, it would be like this:

"territory": [
           {
  	       "averageSales": 6.73,
 	       "averageQtyShipped": 5.25,
           }
]

I really need help, :slight_smile:

Also, I was trying use LET instead of JOIN but I couldn’t get result when I do

USE KEYS META(p).id

territory.entityId IN [ ‘territory::11234’, ‘territory::0045’, ‘territory::0036’]
Do LEFT NEST

Hi @vsr1,
Ah ok… cool you mean I will use LEFT NEST instead of JOIN?
Can I do GROUP BY inside the LEFT JOIN I created ? :frowning:

SELECT 
  meta(p).id,
  p.name,
  p.rank,
  pp AS purchased,
  ps AS average,
  territory
FROM `bucket` AS p

LEFT JOIN `bucket` AS pp 
  ON META(p).id = pp.product
  AND pp.location = “location::6099247"
  AND pp.type = "product-purchased"

LEFT JOIN `bucket` AS ps 
  ON META(p).id = ps.product
  AND ps.periodType = "month"
  AND ps.type = "product-sales-average"
  AND ps.entityId = "location::6099247"

LEFT NEST  `bucket` AS territory 
  ON META(p).id = territory.product
  AND territory.periodType = "month"
  AND territory.type = "product-sales-average"
  AND   territory.entityId  IN [ 'territory::11234', 'territory::0045', 'territory::0036'] 

WHERE p.type = "product" AND p.name IS NOT NULL
ORDER BY purchased.hasPurchased ASC, p.rank.level ASC, p.rank.position ASC
LIMIT 1

OR

 SELECT 
      meta(p).id,
      p.name,
      p.rank,
      pp AS purchased,
      ps AS average,
       (SELECT AVG(t.averageSales) AS  averageSales, AVG(t. averageQtyShipped) AS averageQtyShipped
        FROM territory AS t 
        GROUP BY t. product) AS territory
    FROM `bucket` AS p

    LEFT JOIN `bucket` AS pp 
      ON META(p).id = pp.product
      AND pp.location = “location::6099247"
      AND pp.type = "product-purchased"

    LEFT JOIN `bucket` AS ps 
      ON META(p).id = ps.product
      AND ps.periodType = "month"
      AND ps.type = "product-sales-average"
      AND ps.entityId = "location::6099247"

    LEFT NEST  `bucket` AS territory 
      ON META(p).id = territory.product
      AND territory.periodType = "month"
      AND territory.type = "product-sales-average"
      AND   territory.entityId  IN [ 'territory::11234', 'territory::0045', 'territory::0036'] 

    WHERE p.type = "product" AND p.name IS NOT NULL
    ORDER BY purchased.hasPurchased ASC, p.rank.level ASC, p.rank.position ASC
    LIMIT 1

Hi @vsr1,
Wow, cool thanks man ! thanks for the immediate response.
But, how can I do GROUP BY for the territory?

Checkout previous post