Floats are so back, with Volker () picking up the torch both on his blog and the atproto IETF WG mailing list. Thank you to Volker for pushing on this, and to float proponents for their patience

The buried lede in this post is a new proposal for representing numbers in the data model. That proposal is at the bottom of this post if you just want to skip ahead.

But the float question has been kicking around for some time, and touches on some more general data model questions. I want to get down my thoughts on that now, in the moment, regardless of what we end up deciding in the working group.

This post presumes you are already familiar with the atproto data model, the concerns already discussed about adding floats, and how numbers work generally in CBOR and JSON.

Data Model Requirements

What properties do we need out of the atproto data model? We certainly need the ability to map back and forth between CBOR and JSON. But how strict the determinism around this needs to be hasn't really been nailed down. Here is one attempt to define and motivate the properties we need.

  1. 1.

    Everybody agrees whether a given data object (eg, record) is valid under the atproto data model rules. In other words, we need to specify the data model rules well enough that there is as little ambiguity around this as possible, ideally zero. This is philosophically against Postel's Law (permissive on what you accept). This is particularly important in a decentralized system because data passes around between many parties and services, but we want consistent processing and handling of that data. If we have gaps here, we get interoperability and reliability problems like content not showing up across hosts or client apps, which are very difficult to track down. And we create loop-holes that adversaries can exploit to sneak data past anti-abuse tooling.

    The network is not entirely consistent about this today, but we can improve things with additional     test vectors. There are also caveats around data size limits, which should maybe evolve over time.

  2. 2.

    Everybody computes the same hash (CID) when presented with a valid data object (JSON or CBOR). This is non-trivial to achieve, but is critical for referential integrity for hash links. A protocol-level example of strong references are moderation labels, which annotate a specific version (hash/CID) of a record. An application-specific example is microblogging replies and quote-posts, with versioned references between replies and quote-posts. If services or client software receive and index data records as JSON (eg, using the popular jetstream software), and they want to independently verify the integrity of those references, they need to be able to re-compute the hash of the original record and have it match the CID. And if it does not match, they want to have confidence that this means the reference was invalid and the content had changed.

    If you have unmodified CBOR data on-hand, you just re-hash those bytes to get the version. So this is mostly about services which receive or store data as JSON (eg, from jetstream). There are situations where CBOR objects get modified and re-encoded as CBOR for hashing (eg, when verifying label and commit signatures), but those are relatively simple data objects.

    One example of the way content references over JSON data is used today is storing lexicon schemas as JSON files in git, and then checking those schemas against published versions for differences. The     com.atproto.lexicon.schema records use unknown for the schema data, and exercise most features of the data model.

  3. 3.

    JSON data objects need to be tolerant of re-encoding by non-atproto software. This is an addendum to #2: data objects often pass through many software systems that parse JSON into an internal representation, and then re-encode it on output. This includes data stores (like PostgreSQL JSONB), web frameworks, message buses (like Kafka), and more. It is not safe to assume that JSON emitted by atproto-aware software will be passed through byte-identical through all these systems. As long as two representations of the same data object are semantically "equal" under the JSON data model, they should have the same atproto content hash (CID).

  4. 4.

    Everybody agrees whether a data object (eg, record) is valid against a given lexicon schema. This is about how the data model aligns with the lexicon schema language.

Are floats the right goal?

Before getting into the mechanics of how to shoe-horn IEEE 754 binary64 floating-point numbers into the data model in a way consistent with the requirements above, I want to step back and look at the bigger picture. Is this the right tool for the job? What are we actually trying to do here?

There are many ways to represent numbers on computers. Integers obviously don't cover all use-cases, but neither do 64-bit floats. Fixed-point representations are common for monetary values, bignums come up surprisingly often, etc. Time-series and scientific data transfer often involve arrays or matrices of numbers, and commonly use smaller precision (eg, float32) or encoding techniques for efficiency.

Because atproto data is commonly represented as JSON, floats would need to be transformed between binary and decimal string formats. Why not just put double quotes around them, and keep them as strings? Strings can represent a broader range of numeric types than IEEE 754 binary64. In many common cases (like "2" or "98.6"), the CBOR string representation would be similar or even smaller than a fixed 8-byte float. Numbers as strings are pretty easy to understand when looking at HTTP API responses, and easy to write by hand if needed. Only software aware of the specific data type being used would need to do string/binary conversion, and such software would usually have all the context it needs on-hand to do that conversion.

Not all machine float values can be represented in JSON or CBOR. Special values like NaN and Inf would need to be handled specially, or could cause bugs or errors. They could be represented simply in a string format.

With that off my chest, I'll admit that floats cover a wide number of use-cases that are good fits for atproto, including geolocation (eg, lat/long pairs), weather data, and basic sensor data.

How To Do It

There have been three main proposals to date for adding floats to the data model.

The first is to use string encoding, as discussed in the section above, and recommended in the current atproto.com data model specs. Floats and other number types would be regular strings in the data model, and the lexicon schema language would get a new "float" string format type that could be used by SDKs to help automate parsing and encoding them. I think this option has been a bit underexplored, with float proponents taking it for granted that using 64-bit CBOR floats is the best encoding. I think that even if we go with a different proposal, it might be worth adding additional numeric string formats to the lexicon language. For example, "decimal" (for things like monetary quantities), or "bigint". A "float" string format could also represent more (all?) of the IEEE 754 standard, including special values.

The second option is external tagging, something like { "$float": 1.23 }. Any un-tagged JSON number would be treated as an integer, and all tagged numbers would be treated as floats. Similar to CID links and $bytes, this would require atproto-specific processing when mapping data between JSON and CBOR, though the JSON representation would be robust to processing by other software. The type would be explicit and distinct: integers are integers and floats are floats. I think this meets all the data model requirements discussed above. But the big downside with this is that the JSON representation looks weird and would be annoying for new developers to work with.

Volker's proposal is a third option, to use JSON number formatting to distinguish between integers and floats. "1" would be an integer, while "1.0" would be a float. This happens to work out of the box with some JSON libraries, and can be achieved with configuration in others. But in my opinion the big problem is that JSON number formatting is quite flexible in the JSON specification, and there is no rule that it needs to be preserved when JSON data is processed. This means that intermediate software is free to re-encode it, removing contextual formatting details. This means it doesn't guarantee property #3 (and thus #2 either) from the requirements above.

And now a fourth proposal: adopt the JSON numeric model.

The number Proposal

This isn't an entirely new idea: has pushed in this direction for a while, and Volker mentioned something similar in an earlier blog post.

The basic idea is to flip things around and make number the core numeric type. The semantics and JSON representation would match JSON numbers: integer or floating point, in whatever formatting. "123" and "123.0" are both allowed and mean the same thing. The CBOR encoding would depend on the number itself: if it is an integer, it gets represented as int64, otherwise it gets represented as floats. When decoding CBOR, it would be mandatory to check that float values are not whole numbers (integers), and throw an invalid data error if they are. How numbers get represented in programming languages is an SDK choice. They could always be floats (like in JavaScript), or match the CBOR encoding, or some other convention. What matters is consistency when encoding and decoding CBOR.

The integer type would become a lexicon validation concept, and represented in the data model as number. SDKs can use methods like JavaScript's Number.isInteger() to check if a float is a whole number (integer). Existing data records and deployed atproto software that works with integers would continue to function just as it does today.

A nice property is that common small integer numbers (like "0" and "1") would be represented more compactly than they would be as full float64.

If specified and implemented carefully, I think this would introduce floats with an ergonomic developer experience, while meeting all the requirements described earlier. Here are a few corner-cases to consider:

  • whether isInteger() implementations exist and are perfectly consistent across languages and machine architectures

  • numbers beyond the "safe" 2^53 range where all integers can be represented as a 64-bit float

  • numbers beyond the 2^63 int64 range (do they encode as floats?)

  • being able to losslessly represent any int64 or any float64 is a bit weird

  • other issues we have not thought of

It would probably be best to modify CBOR implementations to handle this logic, though technically atproto SDKs could wrap existing libraries and tweak data objects before passing them for encoding, and then verify data after decoding.

We can discuss this proposal in the IETF working group, though it might also make sense to experiment with implementation in the DASL ecosystem.