In Part 1, we improved application performance by concatenating fields, changing data types, and shortening field names. In Part 2, we implemented the Bucket Pattern and Computed Pattern and optimized the aggregation pipeline to achieve even better performance.

In this final article, we address the issues and improvements identified in appV5R4. Specifically, we focus on reducing the document size in our application to alleviate the disk throughput bottleneck on the MongoDB server. This reduction will be accomplished by adopting a dynamic schema and modifying the storage compression algorithm.

All the application versions and revisions from this article were developed by a senior MongoDB developer, as they are built on all the previous versions and utilize the Dynamic Schema pattern, which isn't very common to see.

Application version 6 revision 0 (appV6R0): A dynamic monthly bucket document

As mentioned in the Issues and Improvements of appV5R4 from the previous article, the primary limitation of our MongoDB server is its disk throughput. To address this, we need to reduce the size of the documents being stored.

Consider the following document from appV5R3, which has provided the best performance so far:

const document = {
  _id: Buffer.from("...01202202"),
  items: [
    { date: new Date("2022-06-05"), a: 10, n: 3 },
    { date: new Date("2022-06-16"), p: 1, r: 1 },
    { date: new Date("2022-06-27"), a: 5, r: 1 },
    { date: new Date("2022-06-29"), p: 1 },
  ],
};

The items array in this document contains only four elements, but on average, it will have around 10 elements, and in the worst-case scenario, it could have up to 90 elements. These elements are the primary contributors to the document size, so they should be the focus of our optimization efforts.

One commonality among the elements is the presence of the date field, with its value including the year and month, for the previous document. By rethinking how this field and its value could be stored, we can reduce storage requirements.

An unconventional solution we could use is:

• Changing the items field type from an array to a document.

• Using the date value as the field name in the items document.

• Storing the status totals as the value for each date field.

Here is the previous document represented using the new schema idea:

const document = {
  _id: Buffer.from("...01202202"),
  items: {
    20220605: { a: 10, n: 3 },
    20220616: { p: 1, r: 1 },
    20220627: { a: 5, r: 1 },
    20220629: { p: 1 },
  },
};

While this schema may not significantly reduce the document size compared to appV5R3, we can further optimize it by leveraging the fact that the year is already embedded in the _id field. This eliminates the need to repeat the year in the field names of the items document.

With this approach, the items document adopts a Dynamic Schema, where field names encode information and are not predefined.

To demonstrate various implementation possibilities, we will revisit all the bucketing criteria used in the appV5RX implementations, starting with appV5R0.

For appV6R0, which builds upon appV5R0 but uses a dynamic schema, data is bucketed by year and month. The field names in the items document represent only the day of the date, as the year and month are already stored in the _id field.

A detailed explanation of the bucketing logic and functions used to implement the current application can be found in the appV5R0 introduction.

The following document stores data for January 2022 (2022-01-XX), applying the newly presented idea:

const document = {
  _id: Buffer.from("...01202201"),
  items: {
    "05": { a: 10, n: 3 },
    16: { p: 1, r: 1 },
    27: { a: 5, r: 1 },
    29: { p: 1 },
  },
};

Schema

The application implementation presented above would have the following TypeScript document schema denominated SchemaV6R0:

export type SchemaV6R0 = {
  _id: Buffer;
  items: Record<
    string,
    {
      a?: number;
      n?: number;
      p?: number;
      r?: number;
    }
  >;
};

Bulk upsert

Based on the specification presented, we have the following updateOne operation for each event generated by this application version:

const DD = getDD(event.date); // Extract the `day` from the `event.date`

const operation = {
  updateOne: {
    filter: { _id: buildId(event.key, event.date) }, // key + year + month
    update: {
      $inc: {
        [`items.${DD}.a`]: event.approved,
        [`items.${DD}.n`]: event.noFunds,
        [`items.${DD}.p`]: event.pending,
        [`items.${DD}.r`]: event.rejected,
      },
    },
    upsert: true,
  },
};

filter:

• Target the document where the _id field matches the concatenated value of key, year, and month.

• The buildId function converts the key+year+month into a binary format.

update:

• Uses the $inc operator to increment the fields corresponding to the same DD as the event by the status values provided.

• If a field does not exist in the items document and the event provides a value for it, $inc treats the non-existent field as having a value of 0 and performs the operation.

• If a field exists in the items document but the event does not provide a value for it (i.e., undefined), $inc treats it as 0 and performs the operation.

upsert:

• Ensures a new document is created if no matching document exists.

Get reports

To fulfill the Get Reports operation, five aggregation pipelines are required, one for each date interval. Each pipeline follows the same structure, differing only in the filtering criteria in the $match stage:

const pipeline = [
  { $match: docsFromKeyBetweenDate },
  { $addFields: buildTotalsField },
  { $group: groupSumTotals },
  { $project: { _id: 0 } },
];

The complete code for this aggregation pipeline is quite complicated. Because of that, we will have just a pseudocode for it here.

1: { $match: docsFromKeyBetweenDate }

• Range-filters documents by _id to retrieve only buckets within the report date range. It has the same logic as appV5R0.

2: { $addFields: buildTotalsField }

• The logic is similar to the one used in the Get Reports of appV5R3.

• The $objectToArray operator is used to convert the items document into an array, enabling a $reduce operation.

• Filtering the items fields within the report's range involves extracting the year and month from the _id field and the day from the field names in the items document.

• The following JavaScript code is logic equivalent to the real aggregation pipeline code.

// Equivalent JavaScript logic:
const [MM] = _id.slice(-2).toString(); // Get month from _id
const [YYYY] = _id.slice(-6, -2).toString(); // Get year from _id
const items_array = Object.entries(items); // Convert the object to an array of [key, value]

const totals = items_array.reduce(
  (accumulator, [DD, status]) => {
    let statusDate = new Date(`${YYYY}-${MM}-${DD}`);

    if (statusDate >= reportStartDate && statusDate < reportEndDate) {
      accumulator.a += status.a || 0;
      accumulator.n += status.n || 0;
      accumulator.p += status.p || 0;
      accumulator.r += status.r || 0;
    }

    return accumulator;
  },
  { a: 0, n: 0, p: 0, r: 0 }
);

3: { $group: groupCountTotals }

• Group the totals of each document in the pipeline into final status totals using $sum operations.

4: { $project: { _id: 0 } }

• Format the resulting document to have the reports format.

Indexes

No additional indexes are required, maintaining the single _id index approach established in the appV4 implementation.

Initial scenario statistics

Collection statistics

To evaluate the performance of appV6R0, we inserted 500 million event documents into the collection using the schema and Bulk Upsert function described earlier. For comparison, the tables below also include statistics from previous comparable application versions:

Event statistics

To evaluate the storage efficiency per event, the Event Statistics are calculated by dividing the total data size and index size by the 500 million events.

It is challenging to make a direct comparison between appV6R0 and appV5R0 from a storage perspective. The appV5R0 implementation is the simplest bucketing possible, where event documents were merely appended to the items array without bucketing by day, as is done in appV6R0.

However, we can attempt a comparison between appV6R0 and appV5R3, the best solution so far. In appV6R0, data is bucketed by month, whereas in appV5R3, it is bucketed by quarter. Assuming document size scales linearly with the bucketing criteria (though this is not entirely accurate), the appV6R0 document would be approximately 3 * 125 = 375 bytes, which is 9.4% smaller than appV5R3.

Another indicator of improvement is the Data Size/Events metric in the Event Statistics table. For appV6R0, each event uses an average of 23.8 bytes, compared to 27.7 bytes for appV5R3, representing a 14.1% reduction in size.

Load test results

Executing the load test for appV6R0 and plotting it alongside the results for appV5R0 and Desired rates, we have the following results for Get Reports and Bulk Upsert.

Get Reports rates

The two versions exhibit very similar rate performance, with appV6R0 showing slight superiority in the second and third quarters, while appV5R0 is superior in the first and fourth quarters.

Get Reports latency

The two versions exhibit very similar latency performance, with appV6R0 showing slight advantages in the second and third quarters, while appV5R0 is superior in the first and fourth quarters.

Bulk Upsert rates

Both versions have similar rate values, but it can be seen that appV6R0 has a small edge compared to appV5R0.

Bulk Upsert latency

Although both versions have similar latency values for the first quarter of the test, for the final three-quarters, appV6R0 has a clear advantage over appV5R0.

Performance summary

Despite the significant reduction in document and storage size achieved by appV6R0, the performance improvement was not as substantial as expected. This suggests that the bottleneck in the application when bucketing data by month may not be related to disk throughput.

Examining the collection stats table reveals that the index size for both versions is close to 3GB. This is near the 4GB of available memory on the machine running the database and exceeds the 1.5GB allocated by WiredTiger for cache. Therefore, it is likely that the limiting factor in this case is memory/cache rather than document size, which explains the lack of a significant performance improvement.

Issues and improvements

To address the limitations observed in appV6R0, we propose adopting the same line of improvements applied from appV5R0 to appV5R1. Specifically, we will bucket the events by quarter in appV6R1. This approach not only follows the established pattern of enhancements but also aligns with the need to optimize performance further.

As highlighted in the Load Test Results, the current bottleneck lies in the size of the index relative to the available cache/memory. By increasing the bucketing interval from month to quarter, we can reduce the number of documents by approximately a factor of three. This reduction will, in turn, decrease the number of index entries by the same factor, leading to a smaller index size.

Application version 6 revision 1 (appV6R1): A dynamic quarter bucket document

As discussed in the previous Issues and Improvements section, the primary bottleneck in appV6R0 was the index size nearing the memory capacity of the machine running MongoDB. To mitigate this issue, we propose increasing the bucketing interval from a month to a quarter for appV6R1, following the approach used in appV5R1.

This adjustment aims to reduce the number of documents and index entries by approximately a factor of three, thereby decreasing the overall index size. By adopting a quarter-based bucketing strategy, we align with the established pattern of enhancements applied in appV5R1 versions while addressing the specific memory/cache constraints identified in appV6R0.

The implementation of appV6R1 retains most of the code from appV6R0, with the following key differences:

• The _id field will now be composed of key+year+quarter.

• The field names in the items document will encode both month and day, as this information is necessary for filtering date ranges in the Get Reports operation.

The following example demonstrates how data for June 2022 (2022-06-XX), within the second quarter (Q2), is stored using the new schema:

const document = {
  _id: Buffer.from("...01202202"),
  items: {
    "0605": { a: 10, n: 3 },
    "0616": { p: 1, r: 1 },
    "0627": { a: 5, r: 1 },
    "0629": { p: 1 },
  },
};

Schema

The application implementation presented above would have the following TypeScript document schema denominated SchemaV6R0:

export type SchemaV6R0 = {
  _id: Buffer;
  items: Record<
    string,
    {
      a?: number;
      n?: number;
      p?: number;
      r?: number;
    }
  >;
};

Bulk upsert

Based on the specification presented, we have the following updateOne operation for each event generated by this application version:

const MMDD = getMMDD(event.date); // Extract the month (MM) and day(DD) from the `event.date`

const operation = {
  updateOne: {
    filter: { _id: buildId(event.key, event.date) }, // key + year + quarter
    update: {
      $inc: {
        [`items.${MMDD}.a`]: event.approved,
        [`items.${MMDD}.n`]: event.noFunds,
        [`items.${MMDD}.p`]: event.pending,
        [`items.${MMDD}.r`]: event.rejected,
      },
    },
    upsert: true,
  },
};

This updateOne operation has a similar logic to the one in appV6R0, with the only differences being the filter and update criteria.

filter:

• Target the document where the _id field matches the concatenated value of key, year, and quarter.

• The buildId function converts the key+year+quarter into a binary format.

update:

• Uses the $inc operator to increment the fields corresponding to the same MMDD as the event by the status values provided.

Get reports

To fulfill the Get Reports operation, five aggregation pipelines are required, one for each date interval. Each pipeline follows the same structure, differing only in the filtering criteria in the $match stage:

const pipeline = [
  { $match: docsFromKeyBetweenDate },
  { $addFields: buildTotalsField },
  { $group: groupSumTotals },
  { $project: { _id: 0 } },
];

This aggregation operation has a similar logic to the one in appV6R0, with the only differences being the implementation in the $addFields stage.

{ $addFields: itemsReduceAccumulator }:

• A similar implementation to the one in appV6R0

• The difference relies on extracting the value of year (YYYY) from the _id field and the month and day (MMDD) from the field name.

• The following JavaScript code is logic equivalent to the real aggregation pipeline code.

const [YYYY] = _id.slice(-6, -2).toString(); // Get year from _id
const items_array = Object.entries(items); // Convert the object to an array of [key, value]

const totals = items_array.reduce(
  (accumulator, [MMDD, status]) => {
    let [MM, DD] = [MMDD.slice(0, 2), MMDD.slice(2, 4)];
    let statusDate = new Date(`${YYYY}-${MM}-${DD}`);

    if (statusDate >= reportStartDate && statusDate < reportEndDate) {
      accumulator.a += status.a || 0;
      accumulator.n += status.n || 0;
      accumulator.p += status.p || 0;
      accumulator.r += status.r || 0;
    }

    return accumulator;
  },
  { a: 0, n: 0, p: 0, r: 0 }
);

Indexes

No additional indexes are required, maintaining the single _id index approach established in the appV4 implementation.

Initial scenario statistics

Collection statistics

To evaluate the performance of appV6R1, we inserted 500 million event documents into the collection using the schema and Bulk Upsert function described earlier. For comparison, the tables below also include statistics from previous comparable application versions:

Event statistics

To evaluate the storage efficiency per event, the Event Statistics are calculated by dividing the total data size and index size by the 500 million events.

In the previous Initial Scenario Statistics analysis, we assumed that document size would scale linearly with the bucketing range. However, this assumption proved inaccurate. The average document size in appV6R1 is approximately twice as large as in appV6R0, even though it stores three times more data. Already a win for this new implementation.

Since appV6R1 buckets data by quarter at the document level and by day within the items sub-document, a fair comparison would be with appV5R3, the best-performing version so far. From the tables above, we observe a significant improvement in Document Size and consequently Data Size when transitioning from appV5R3 to appV6R1. Specifically, there was a 31.4% reduction in Document Size. From an index size perspective, there was no change, as both versions bucket events by quarter.

Load test results

Executing the load test for appV6R0 and plotting it alongside the results for appV5R0 and Desired rates, we have the following results for Get Reports and Bulk Upsert.

Get Reports rates

For the first three-quarters of the test, both versions have similar rate values, but, for the final quarter, appV6R1 has a notable edge over appV5R3.

Get Reports latency

The two versions exhibit very similar latency performance, with appV6R0 showing slight advantages in the second and third quarters, while appV5R0 is superior in the first and fourth quarters.

Bulk Upsert rates

Both versions have similar rate values, but it can be seen that appV6R0 has a small edge compared to appV5R0.

Bulk Upsert latency

Although both versions have similar latency values for the first quarter of the test, for the final three-quarters, appV6R0 has a clear advantage over appV5R0.

Performance summary

Despite the significant reduction in document and storage size achieved by appV6R0, the performance improvement was not as substantial as expected. This suggests that the bottleneck in the application when bucketing data by month may not be related to disk throughput.

Examining the collection stats table reveals that the index size for both versions is close to 3GB. This is near the 4GB of available memory on the machine running the database and exceeds the 1.5GB allocated by WiredTiger for cache. Therefore, it is likely that the limiting factor in this case is memory/cache rather than document size, which explains the lack of a significant performance improvement.

Issues and improvements

To address the limitations observed in appV6R0, we propose adopting the same line of improvements applied from appV5R0 to appV5R1. Specifically, we will bucket the events by quarter in appV6R1. This approach not only follows the established pattern of enhancements but also aligns with the need to optimize performance further.

As highlighted in the Load Test Results, the current bottleneck lies in the size of the index relative to the available cache/memory. By increasing the bucketing interval from month to quarter, we can reduce the number of documents by approximately a factor of three. This reduction will, in turn, decrease the number of index entries by the same factor, leading to a smaller index size.

Application version 6 revision 1 (appV6R1): A dynamic quarter bucket document

As discussed in the previous Issues and Improvements section, the primary bottleneck in appV6R0 was the index size nearing the memory capacity of the machine running MongoDB. To mitigate this issue, we propose increasing the bucketing interval from a month to a quarter for appV6R1, following the approach used in appV5R1.

This adjustment aims to reduce the number of documents and index entries by approximately a factor of three, thereby decreasing the overall index size. By adopting a quarter-based bucketing strategy, we align with the established pattern of enhancements applied in appV5R1 versions while addressing the specific memory/cache constraints identified in appV6R0.

The implementation of appV6R1 retains most of the code from appV6R0, with the following key differences:

• The _id field will now be composed of key+year+quarter.

• The field names in the items document will encode both month and day, as this information is necessary for filtering date ranges in the Get Reports operation.

The following example demonstrates how data for June 2022 (2022-06-XX), within the second quarter (Q2), is stored using the new schema:

const document = {
  _id: Buffer.from("...01202202"),
  items: {
    "0605": { a: 10, n: 3 },
    "0616": { p: 1, r: 1 },
    "0627": { a: 5, r: 1 },
    "0629": { p: 1 },
  },
};

Schema

The application implementation presented above would have the following TypeScript document schema denominated SchemaV6R0:

export type SchemaV6R0 = {
  _id: Buffer;
  items: Record<
    string,
    {
      a?: number;
      n?: number;
      p?: number;
      r?: number;
    }
  >;
};

Bulk upsert

Based on the specification presented, we have the following updateOne operation for each event generated by this application version:

const MMDD = getMMDD(event.date); // Extract the month (MM) and day(DD) from the `event.date`

const operation = {
  updateOne: {
    filter: { _id: buildId(event.key, event.date) }, // key + year + quarter
    update: {
      $inc: {
        [`items.${MMDD}.a`]: event.approved,
        [`items.${MMDD}.n`]: event.noFunds,
        [`items.${MMDD}.p`]: event.pending,
        [`items.${MMDD}.r`]: event.rejected,
      },
    },
    upsert: true,
  },
};

This updateOne operation has a similar logic to the one in appV6R0, with the only differences being the filter and update criteria.

filter:

• Target the document where the _id field matches the concatenated value of key, year, and quarter.

• The buildId function converts the key+year+quarter into a binary format.

update:

• Uses the $inc operator to increment the fields corresponding to the same MMDD as the event by the status values provided.

Get reports

To fulfill the Get Reports operation, five aggregation pipelines are required, one for each date interval. Each pipeline follows the same structure, differing only in the filtering criteria in the $match stage:

const pipeline = [
  { $match: docsFromKeyBetweenDate },
  { $addFields: buildTotalsField },
  { $group: groupSumTotals },
  { $project: { _id: 0 } },
];

This aggregation operation has a similar logic to the one in appV6R0, with the only differences being the implementation in the $addFields stage.

{ $addFields: itemsReduceAccumulator }:

• A similar implementation to the one in appV6R0

• The difference relies on extracting the value of year (YYYY) from the _id field and the month and day (MMDD) from the field name.

• The following JavaScript code is logic equivalent to the real aggregation pipeline code.

const [YYYY] = _id.slice(-6, -2).toString(); // Get year from _id
const items_array = Object.entries(items); // Convert the object to an array of [key, value]

const totals = items_array.reduce(
  (accumulator, [MMDD, status]) => {
    let [MM, DD] = [MMDD.slice(0, 2), MMDD.slice(2, 4)];
    let statusDate = new Date(`${YYYY}-${MM}-${DD}`);

    if (statusDate >= reportStartDate && statusDate < reportEndDate) {
      accumulator.a += status.a || 0;
      accumulator.n += status.n || 0;
      accumulator.p += status.p || 0;
      accumulator.r += status.r || 0;
    }

    return accumulator;
  },
  { a: 0, n: 0, p: 0, r: 0 }
);

Indexes

No additional indexes are required, maintaining the single _id index approach established in the appV4 implementation.

Initial scenario statistics

Collection statistics

To evaluate the performance of appV6R1, we inserted 500 million event documents into the collection using the schema and Bulk Upsert function described earlier. For comparison, the tables below also include statistics from previous comparable application versions:

Event statistics

To evaluate the storage efficiency per event, the Event Statistics are calculated by dividing the total data size and index size by the 500 million events.

As expected, we had an 11.2% increase in the Document Size by adding a totals field in each document of appV6R2. When comparing to appV5R3, we still have a reduction of 23.9% in the Document Size. Let's review the Load Test Results to see if the trade-off between storage and computation cost is worthwhile.

Load test results

Executing the load test for appV6R2 and plotting it alongside the results for appV6R1 and Desired rates, we have the following results for Get Reports and Bulk Upsert.

Get Reports rates

We can see that appV6R2 has better rates than appV6R1 throughout the test, but it’s still not reaching the top rate of 250 reports per second.

Get Reports latency

As shown in the rates graph, appV6R2 consistently provides lower latency than appV6R1 throughout the test.

Bulk Upsert rates

Both versions exhibit very similar rate values throughout the test, with appV6R2 performing slightly better than appV6R1 in the final 20 minutes, yet still failing to reach the desired rate.

Bulk Upsert latency

Although appV6R2 had better rate values than appV6R1, their latency performance is not conclusive, with appV6R2 being superior in the first and final quarters and appV6R1 in the second and third quarters.

Performance summary

The two "maybes" from the previous Issues and Improvements made up for their promises, and we got the best performance for appV6R2 when comparing to appV6R1. This is the redemption of the Computed Pattern applied on a document level. This revision is one of my favorites because it shows that the same optimization on very similar applications can lead to different results. In our case, the difference was caused by the application being very bottlenecked by the disk throughput.

Issues and improvements

Let's tackle the last improvement on an application level. Those paying close attention to the application versions may have already questioned it. In every Get Reports section, we have "To fulfill the Get Reports operation, five aggregation pipelines are required, one for each date interval." Do we really need to run five aggregation pipelines to generate the reports document? Isn't there a way to calculate everything in just one operation? The answer is yes, there is.

The reports documents are composed of fields oneYear, threeYears, fiveYears, sevenYears, and tenYears, where each one was generated by its respective aggregation pipeline until now. Generating the reports this way is a waste of processing power because we are doing some part of the calculation multiple times. For example, to calculate the status totals for tenYears, we will also have to calculate the status totals for the other fields, as from a date range perspective, they are all contained in the tenYears date range.

So, for our next application revision, we'll condense the Get Reports five aggregation pipelines into one, avoiding wasting processing power on repeated calculation.

Application version 6 revision 3 (appV6R3): Getting everything at once

As discussed in the previous Issues and Improvements section, in this revision, we'll improve the performance of our application by changing the Get Reports functionality to generate the reports document using only one aggregation pipeline instead of five.

The rationale behind this improvement is that when we generate the tenYears totals, we have also calculated the other totals, oneYear, threeYears, fiveYears, and sevenYears. As an example, when we request to Get Reports with the key ...0001 with the date 2022-01-01, the totals will be calculated with the following date range:

• oneYear: from 2021-01-01 to 2022-01-01

• threeYears: from 2020-01-01 to 2022-01-01

• fiveYears: from 2018-01-01 to 2022-01-01

• sevenYears: from 2016-01-01 to 2022-01-01

• tenYear: from 2013-01-01 to 2022-01-01

As we can see from the list above, the date range for tenYears encompasses all the other date ranges.

Although we successfully implemented the Computed Pattern in the previous revision, appV6R2, achieving better results than appV6R1, we will not use it as a base for this revision. There were two reasons for that:

1. Based on the results of our previous implementation of the Computed Pattern on a document level, from appV5R3 to appV5R4, I didn't expect it to get better results.

2. Implementing Get Reports to retrieve the reports document through a single aggregation pipeline, utilizing pre-computed field totals generated by the Computed Pattern would require significant effort. By the time of the latest versions of this series, I just wanted to finish it.

So, this revision will be built based on the appV6R1.

Schema

The application implementation presented above would have the following TypeScript document schema denominated SchemaV6R0:

export type SchemaV6R0 = {
  _id: Buffer;
  items: Record<
    string,
    {
      a?: number;
      n?: number;
      p?: number;
      r?: number;
    }
  >;
};

Bulk upsert

Based on the specifications, the following bulk updateOne operation is used for each event generated by the application:

const YYYYMMDD = getYYYYMMDD(event.date); // Extract the year(YYYY), month(MM), and day(DD) from the `event.date`

const operation = {
  updateOne: {
    filter: { _id: buildId(event.key, event.date) }, // key + year + quarter
    update: {
      $inc: {
        [`items.${YYYYMMDD}.a`]: event.approved,
        [`items.${YYYYMMDD}.n`]: event.noFunds,
        [`items.${YYYYMMDD}.p`]: event.pending,
        [`items.${YYYYMMDD}.r`]: event.rejected,
      },
    },
    upsert: true,
  },
};

This updateOne has almost exactly the same logic as the one for appV6R1. The difference is that the name of the fields in the items document will be created based on year, month, and day (YYYYMMDD) instead of just month and day (MMDD). This change was made to reduce the complexity of the aggregation pipeline of the Get Reports.

Get reports

To fulfill the Get Reports operation, one aggregation pipeline is required:

const pipeline = [
  { $match: docsFromKeyBetweenDate },
  { $addFields: buildTotalsField },
  { $group: groupCountTotals },
  { $project: format },
];

This aggregation operation has a similar logic to the one in appV6R1, with the only differences being the implementation in the $addFields stage.

{ $addFields: buildTotalsField }

• It follows a similar logic to the previous revision, where we first convert the items document into an array using $objectToArray, and then use the reduce function to iterate over the array, accumulating the status.

• The difference lies in the initial value and the logic of the reduce function.

• The initial value in this case is an object/document with one field for each of the report date ranges. These fields for each report date range are