Should NA equals NA in comparison?

[zhangyingmath]

Here is what I see:

DT = dt.Frame({'f': [1,2,3, np.nan]})
df = pd.DataFrame({'f': [1,2,3, np.nan]})

df.loc[df.f == df.f].shape
(3, 1)

DT[dt.f.f == dt.f.f, :].shape
(4, 1)

I believe the issue is rooted in the python behavior that

None == None
True

np.nan == np.nan
False

I am curious if the datatable behavior above is a conscious choice, or not. Being a long-time pandas user, I personally think the pandas/numpy behavior (NA does not equal NA) acceptable. I am curious what the datatable team thinks of this.

[samukweku]

@zhangyingmath , this is just a guess, but it seems np.nan is turned to None (NA) within datatable. As such, None==None returns True.

[zhangyingmath]

Thanks @samukweku ! Yes, that seems to be what is going on, in terms of code logic. However, I am also wondering what the behavior "should/ought to" be, and I will comment on @st-pasha 's comments below.

[st-pasha]

Well, this is a rather philosophical question.

On one hand, the axiom of reflexivity postulates that every value is equal to itself. This implies that NA==NA is True. It is possible to reject this axiom and postulate that some values (such as NA) are not equal to themselves. However, if we do accept the axiom, it becomes easy to reason about the properties of the == relation. In particular, every column in a frame is equal to itself, and every frame is equal to itself too.

Now, as you correctly pointed out, some people reject full reflexivity and postulate that NA==NA is False. Why would that be a case? We'd have to look at what "NA" actually mean. It turns out there could be multiple different kinds of NAs:

• An NA could mean "not a number", i.e. a special value that signals an error in a calculation. For example, ln(-1) or arcsin(10) both produce NAs (alternatively, they could have thrown an exception instead). In this case "NA" is just an error object, and there is little reason to say that one error should be different from another.

• An NA could also mean "not applicable". For example, if you have a dataset with data on how many children there are in a family, and what is the ages of youngest and oldest of the children, then for a family with 0 children the last two fields will be NAs. In this case, again, the records of two families both having 0 children should compare equal. Thus, in this case NA==NA is again True.

• An NA could also mean "not available" (aka "missing"). Unlike the previous case, the data does exist somewhere, it's merely not known in the dataset. In this case NA value is akin to a black box, and it therefore can be argued that two NA values are not equal to each other: their underlying hidden values could be different. But they could also be the same, we just don't know. Thus, in this case the result of comparison NA==NA is itself NA, i.e. not known.

We may also mention another special number here: infinity. Well, technically it's not a number, it's a symbol. But in numpy it can indicate either the value that is truly infinite, or merely large enough to not fit into float64. Even though they are not numbers, they compare as equal:

>>> np.inf == np.inf
True

This is despite the fact that those infinities can represent demonstrably unequal quantities:

>>> (np.float64(10) ** np.float64(1000)) == (np.float64(10) ** np.float64(1000000))
True

We use the same approach with NAs as with infinities: each value is equal to itself. This just seems more consistent and easier to reason about.

[zhangyingmath]

Thanks @st-pasha ! Yes, I agree, you made some good points. There are certainly times when letting NA == NA makes sense, and I have no problem that this is the default behavior in h2o datatable when I write DT[dt.f.f == dt.f.g, :]. That being said, I'd still like to propose two suggestions for your consideration:

1. (If it is not already there, in which case I may have simply missed it when reading the doc), please consider adding a paragraph in the documentation (comparison with pandas/numpy). Chances are, users (data wranglers) may not always think from a mathematical perspective. So it may help them to let them know that they should expect a difference here if they used to work with pandas/numpy.
   In my personal opinion, NA handling in a data processing package is really important, and is also a place where many users like to make mistakes. So helping them wouldn't hurt.

2. I wonder if it is possible to add an additional API to the comparison operator that allows more flexibility with more key word arguments, sth like DT['f'].equals(DT['g'], match_na=True/False). An analogy to this arrangement would be this example: in pandas, df.get('column_that_doesnot_exist', default=0) would give me 0, while df['column_that_doesnot_exist'] would raise an error. This gives the user more control.

In data wrangling, sometimes the input source is so bad that nothing is guaranteed. (As a hypothetical example, say my rectangular data might be sourced from parsing some logs, but the upstream process made an update that suddenly give me strings when it used to be integers. Pretend I am doing a "regression test" in this case,

DT = dt.Frame({'val_yesterday': [1,2,3, None], 'val_today': ['1', '2', '3', None]})
DT[dt.f.val_yesterday == dt.f.val_today, :].shape[0]
1

Obviously, this "successful match" on the row of NA values isn't really super interesting.

[zhangyingmath]

Ah, I just find out that IEEE 754 says NA should not equal NA in comparison, https://en.wikipedia.org/wiki/NaN

[st-pasha]

Indeed, in IEEE-754 nan==nan is False. However, if you check the rationale for such decision, it turns out that it was done for the sole reason that at the time of the adoption of the standard there was no way to check if a number was nan, so they decided that x==x returning 0 would be such a test. Further, if you read comments here https://stackoverflow.com/questions/1565164, you'll see that such convention creates problems when numbers are used in containers (such as in a Frame).

Also, IEEE-754's nan has the meaning of "not a number", i.e. an error object. In modern data-science applications the meanings "not applicable" / "not available" are far more common.

[zhangyingmath]

Thanks for the link. I was checking how R data.table behaves regarding this issue. Interestingly, in R, NA can be part of a int, float or boolean vector. And NA comparison with NA gives a (boolean) NA. But:

1. If used in a boolean mask for a regular vector in R, NA will cause the corresponding element to be selected.
2. Unless, the boolean mask is used in a data.table, then NA will be treated as FALSE in selection.