This post is part of a series about the Halo Infinite Web API.
You can also explore the .NET wrapper for the API that makes endpoint interaction a bit easier.
As I was investigating the API, I’ve noticed something peculiar about the requests that were coming out of my machine and the responses I was getting.
Accept: application/x-bond-compact-binary header along with content that kind of looks like gibberish with the occasional nuggets of information embedded in them. I’ve never seen this before, but I worked a little bit with protocol buffers and this seemed oddly familiar. The
x-bond-compact-binary was a signal that the content is likely a binary format that is used to reduce the amount of data that is sent over the wire. Doing a bit more research, I found out that this is nothing else but Microsoft’s take on protocol buffers - the Bond framework.
Having an idea of what it is, I got curious - just how much is 343 saving in terms of content size if they do try to squeeze as much performance as possible without extra overhead? If it’s anything like protocol buffers, my assumption was that the Bond format strips out most of the “metadata” around the content and just serializes it in a way that the data models on either end of the wire can understand, but beyond that - the data would be almost useless (since we wouldn’t know what we’re looking at).
To validate my size hypothesis, let’s take an API endpoint such as this:
This endpoint is designed to get bot customization data and, just like many other API endpoints, accepts three types of formats in which data can be returned:
application/xml (not all API endpoints actually support it - bit of a hit or miss here), and
application/x-bond-compact-binary. It also returns quite a bit of information all at once, so it could be a good testing surface.
As a side-note - having all three formats available is incredibly convenient for reverse-engineering the API and understand what exactly it’s doing behind the scenes, but I digress. I first requested the JSON content for the response - a whopping 8,114 line response totaling to a rough 10.08KB with 21ms to response. Not terribly bad. There is no XML variant of this endpoint, so I didn’t have anything to compare here. The Bond-formatted content landed on my box in 17ms with a size of 9.23KB - an 8.432% savings in content volume. Not bad at all, especially since this adds up given just how many network requests Halo Infinite makes over the span of many multiplayer games - any savings in volume is good, and that also reduces the parsing overhead.
The challenge remained, though - how can I parse this out? I had no access to the underlying data model and it seemed that I needed it to be able to go from binary content to something a human can read. Apparently I was not the only one, because there is a GitHub issue requesting an implementation of a Fiddler inspector, which, by its nature, also requires the ability to deserialize Bond content without access to its data model.
ReadFieldBegin: this will tell you the field ordinal and its type
- if type is
BT_STOP, be done with this struct
- if type is
BT_STOP_BASE, continue reading fields but know that you’re one level lower in the hierarchy
- if type is a scalar, string, or blob, call the appropriate ReadFoo method
- if type is a container, call the appropriate ReadContainerBegin method: this will tell you the count of elements and the element type(s).
- read count container elements, using the container type to know how to read each item
- if type is a struct, recur
- Loop back to step 2 until you see
Additionally, Christopher called out two other important pieces of the process:
If you don’t read the field, you need to skip it.
ReadFieldBeginis “interpret the next bytes as if they were the preamble of the next field”, not “seek to next field and read it’s preamble”.
You need to make sure you initialize the reader with the right protocol and version. There’s no header or checksum in serialized data, so the reader can’t tell if you pointed a Fast Binary reader at Compact Binary stream. (Marshalled data has a header that indicates the protocol and version in the first four bytes.)
This all sounds like a reasonable way to start tackling the problem. Lucky for me, there is a Bond implementation in C# so I don’t need to do everything from scratch. My current implementation, following the algorithm above, is this (keep in mind that I do not read all field types - some are skipped):
One thing I’ll call out is that this was done using a bit of trial-and-error - pay attention to this line:
Recall earlier that the content type that we worked with was
application/x-bond-compact-binary. This was the first clue that told me that I likely should be using
CompactBinaryReader (refer to compact binary encoding reference) as opposed to something else. The
2 parameter is the protocol version - again, this is just coming from my assumption that the compact binary format comes with the second version of the protocol based on the aforementioned compact binary encoding reference:
With this implementation I was able to take a response I got that was encoded in Bond format, and try and parse it out locally. How do you save a Bond response? Whatever tool you use for experimentation, such as Fiddler or Postman (or maybe you’re even writing your own) generally allows you to save the binary content by exporting it to a file.
Running the code on the sample response from the bot customization endpoint I called out earlier I seemed to get the right result that looks a bit less like gibberish:
To compare, I looked at the JSON response and selected a small chunk:
Then I tried to find the same chunk in the decoded response:
The latter looks a bit more verbose purely because I’ve added quite a bit of logging to my code to make sure that I am able to diagnose any parsing issues. So far, though, it looks good, and I am able to read Bond responses. As you can tell, it would be much harder to interpret those without having the data model handy. There is no indication what any of the data means and you would just need to infer it from context.
A lot of it can likely be done from looking at content groups (e.g., you’re looking at lists of bot appearance configurations), but things like integer values that are flagged in the JSON response as
ConfigurationId would be impossible to know without prior knowledge.
All this being said, I hope this can give you a better idea of how to look at Bond content coming through the Halo API, and combined with JSON, and in select cases - XML responses, you can make more sense of what goes on between your computer and the Halo servers.