I’m trying to interpret telemetry data for household tasks (#5886) and I want to make sure I am understanding the data correctly. The telemetry documentation states that each metric includes “performance” data such as min, max and sumrepresented as time in milliseconds as well as a count, which represents the number of times the metric was recorded. However, for some metrics I don’t think this is what it represents.
Metric → tasks:group:ready
Definition → Number of tasks in “Ready” state for the household, when displaying household tasks page.
Example data
This is recorded per user per day. Here are a few example entries I’m seeing, 4 different users on the same day.
user
metric
min
max
sum
count
user_1
tasks:group:ready
0
1
1
4
user_2
tasks:group:ready
0
6
121
52
user_3
tasks:group:ready
0
2
8
15
user_4
tasks:group:ready
0
0
0
1
Interpretation (Data)
For the DAY that tasks:group:ready telemetry was recorded…
Property
What it represents (I think)
min
The fewest number of tasks shown (out of all times the HH tasks page was shown)
max
The maximum number of tasks shown (out of all times the HH tasks page was shown)
sum
The sum total number of tasks shown (out of all times the HH tasks page was shown)
count
The number of times the HH tasks page was shown
Interpretation (Narrative)
Using the sample data provided above, user_3 saw the HH tasks page 15 times. The maximum number of tasks they saw on the HH tasks page at any one time was 2. The total number of tasks shown on the HH tasks page across all times they saw the HH tasks page was 8.
It seems strange that we would see a min = 0 (and a sum < count), I would have hoped we wouldn’t show the HH tasks page if there are no other tasks to show… or is it actually incrementing count even when the CHT determines that the HH tasks screen does not need to be shown?
Just from a quick code inspection, I suspect you’ve found a new measurement issue.
We record this.telemetryService.record('tasks:group:ready', this.tasks.length); but this.tasks isn’t assigned a value until after it is measured.
If this theory is correct, you can’t trust this data. An example of how it is wrong is that, it will measure 0 for all users the first time the experience is shown no matter how many tasks are visible. That aligns with the data you shared, and explains the min=0 for all users.
@michael Do you “know how”/“are willing” to test this to confirm the theory? Get the household tasks experience to trigger, view the user’s telemetry data, see if the value reflects the number of tasks you saw?
Thanks @kenn . I ran a test yesterday and looked at the telemetry data today. Here’s what I found…
Scenario
There were 3 tasks for Jennifer Mbae’s household (1 for her and 2 for her children)
I tapped on and completed the task for Jennifer Mbae
The “Other household tasks” screen showed up
When the screen first rendered, there were only 2 tasks (for Peter and Mary)
After a second or two, the task for Jennifer showed up, so there are 3 tasks on this screen
The task for Jennifer was generated as a result of the original task I just completed
Observations
Interestingly, tasks:group:ready → min did not=0, but it also didn’t include the newly generated (3rd) task that showed up after a few moments. It’s a tough call, but I think I’m OK with it not including the newly generated task because I imagine it is unlikely CHWs would want to complete a task that was a result of another task they just completed 5 seconds ago (though there may be some cases where they do).
Is there something else you think I should try that might recreate min = 0 ?
Looking at the data some more (see example below)… there are many entries for tasks:group:ready where the count is non-zero but all other entries are 0, so it must be incrementing the count even when the household tasks screen is not shown. The reason I don’t think the screen is being shown is that for the example below, there were no entries for tasks:group:modal:confirm or tasks:group:modal:reject(when you try to navigate away from the household tasks screen it will pop up a confirmation modal).
So I guess we can know how many times the household tasks screen was shown and then cancelled (using the tasks:group:modal:confirm property), but I don’t think I can know how many times it was shown, and we can’t know how many times it was shown and a task from it was completed directly from there. (By looking at submitted reports I can tell if other tasks were completed for that household around the same time, but I don’t think there’s a way to know if it was directly from the HH task screen - something like this might help).
