Record and Event Stream File Formats

Overview

The Hedera network produces record stream files and event stream files that captures chronological information about transactions that took place on the network. A record stream file is a file that contains a series of transactions in chronological order that occurred within a two second interval. The record stream file (.rcd) contains a transaction record for each transaction in that file. For each record stream file there is a corresponding signature file (rcd_sig) that includes the signature generated by the node. The event stream file(.evts) contains information about an event. Event information includes event hash data, event unhashed data, and consensus data. Each event stream files are created in 5 second intervals. Each event stream file has a corresponding signature file (.evts_sig) that includes the node signature.

The record stream file and event stream file formats will both be updated to version 5 in a future mainnet release.

Version Migration

File Type

Current Version

Next Version

Record Stream File

2

5

Record Stream Signature File

No version number

5

Event Stream File

3

5

Event Stream Signature File

No version number

5

Note: Current record signature files (rcd_sig) and event signature (evts_sig) file formats do not have a version number. The first byte's value is 4, which denotes a marker. For maintaining backwards compatibility, we made the first byte in the new version stream signature file be 5, which denotes the version. Thus, we use version 5 as the next version number.

Version 2 Record Stream File Format

Record Stream Files Names

A record stream file name is made up of a string representation of the Instant of consensus timestamp of the first transaction in the file using ISO-8601 representation, with colons converted to underscores for Windows compatibility. The nano-of-second outputs zero, three, six or nine digits as necessary.

Examples of a record stream file name with corresponding signature file:

Example 1:

Record Stream File: 2020-10-19T21_35_39Z.rcd

Record Stream Signature File: 2020-10-19T21_35_39Z.rcd_sig

Example 2:

Record Stream File: 2020-10-19T21_35_39.454265Z.rcd

Record Stream Signature File: 2020-10-19T21_35_39.454265Z.rcd_sig

Record Stream File Format (.rcd)

The below table describes the content that can be parsed from a record file.

Name

Type (Bytes)

Description

Record Stream File Format Version

int (4)

Record stream file format version,

Value: 2

Hapi Version

int (4)

HAPI protocol version,

Value: 3

Prev File Hash Marker

byte

Value: 1

Prev File Hash

byte[48]

SHA384 hash of previous file, if not exist then all zeroes

Record Marker

byte

Value: 2

Length of Transaction

int (4)

Byte size of the following Transaction message

Transaction

byte[]

Serialized Transaction message bytes

Length of TransactionRecord

int (4)

Byte size of the following TransactionRecord message

TransactionRecord

byte[]

Serialized TransactionRecord message bytes

Record Marker

byte

Value: 2

...

...

...

The signature file *.rcd_sig signs the hash of corresponding *.rcd file;

Record Stream Signature File Format (.rcd_sig)

Note: Event stream signature file format does not have a version number

The below table describes the content that can be parsed from a record signature file.

Name

Type (Bytes)

Description

File Hash Marker

byte

Value: 4

File Hash

byte[48]

SHA384 hash of corresponding *.rcd file

Signature Marker

byte

Value: 3

Length of Signature

int (4)

Byte size of the following signature bytes

Signature

byte[]

Signature bytes

File Hash Calculation

The record file hash is calculated using the following formula.

h[i] = hash(p[i-1] || h[i-1] || hash(c[i-1]))

  • || denotes concatenation

  • h[i] is the hash of the file i

  • p[i-1] is the contents in the file before the PREV_FILE_HASH

  • h[i-1] is the hash of file i-1, i.e., PREV_FILE_HASH

  • c[i-1] is the contents of the file after PREV_FILE_HASH

Version 5 Record Stream File Format

Generated using functionality provided by swirlds-common in the swirlds-platform sdk for serializing objects to record files (.rcd), and deserializing objects from record file (.rcd). In the next release, swirlds-common package will contain functionalities for parsing and verifying version 5 record stream files, which can be used by both services and mirror node.

Record Stream Files Names

A record file name is created using a string representation of the Instant of consensus timestamp of the first transaction in the file using ISO-8601 representation, with colons converted to underscores for Windows compatibility.

The nano-of-second always outputs nine digits with padding when necessary, to ensure same length filenames and proper sorting.

Examples of a record stream file name with corresponding signature file:

Example 1:

Record File (.rcd): 2020-10-19T21_35_39.000000000Z.rcd

Record Signature File(.rcd_sig): 2020-10-19T21_35_39.000000000Z.rcd_sig

Example 2:

Record File (.rcd): 2020-10-19T21_35_39.454265000Z.rcd

Record Signature File(.rcd_sig): 2020-10-19T21_35_39.454265000Z.rcd_sig

Record Stream File Format (.rcd)

MD – Meta Data

The below tables describe the content that can be parsed from a record file.

Name

Type (Bytes)

Description

MD

Record Stream File Format Version

int (4)

Record stream file format version,

Value: 5

y

Hapi Proto Major Version

int (4)

Matches NetworkGetVersionInfo.hapiProtoVersion

Major.Minor.Patch

Major version: 0;

Minor version: 9;

Patch version: 0

y

Hapi Proto Minor Version

int (4)

y

Hapi Proto Patch Version

int (4)

y

Object stream version

int (4)

Value: 1

This defines the format of the remainder of the file. This version number is used when parsing a stream file with methods defined in swirlds-common package

y

Start Object Running Hash

byte[]

Running Hash of all RecordStreamObject before writing this file.

See `Hash object` table below for details

y

1st RecordStreamObject

byte[]

Serialized RecordStreamObject bytes.

See ` RecordStreamObject ` table below for details

n

2nd RecordStreamObject

byte[]

...

n

...

...

...

n

End Object Running Hash

byte[]

Running Hash of all RecordStreamObject before closing this file.

y

Hash Object

Name

Type (Bytes)

Description

Class ID

long (8)

Value: 0xf422da83a251741e

Class Version

int (4)

Value: 1

This version number will be updated each time when the serialization format of a Hash object is changed

Digest Type

int (4)

Value: 0x58ff811b

denotes SHA-384

Length of Hash

int (4)

Value: 48 for SHA-384

Hash Bytes

byte[]

Serialized Hash bytes

Record Stream Object

Name

Type (Bytes)

Description

Class ID

long (8)

Value: 0xe370929ba5429d8b

Class Version

int (4)

Value: 1

Length of TransactionRecord

int (4)

Byte size of the following TransactionRecord message

TransactionRecord

byte[]

Serialized TransactionRecord message bytes

Length of Transaction

int (4)

Byte size of the following Transaction message

Transaction

byte[]

Serialized Transaction message bytes

Record Stream Signature File Format (.rcd_sig)

In version 5, the record stream signature file format is the same as event stream signature file format. The below tables describe the content that can be parsed from a record signature file.

Name

Type (Bytes)

Description

Signature File Format Version

byte

Value: 5

Object Stream Signature Version

int (4)

Value: 1

This defines the format of the remainder of the signature file. This version number is used when parsing a signature file with methods defined in swirlds-common package

Entire Hash of the corresponding stream file

byte[]

Hash of the entire corresponding stream file

Signature on hash bytes of Entire Hash

byte[]

A signature object generated by signing the hash bytes of Entire Hash. See ` Signature ` table below for details

Metadata Hash of the corresponding stream file

byte[]

Metadata Hash of the corresponding stream file

Signature on hash bytes of Metadata Hash

byte[]

A signature object generated by signing the hash bytes of Metadata Hash

Signature Object

Name

Type (Bytes)

Description

Class ID

long (8)

Value: 0x13dc4b399b245c69

Class Version

int (4)

Value: 1

SignatureType

int (4)

Value: 1

Denotes SHA384withRSA

Length of Signature

int (4)

Byte size of the signature bytes

CheckSum

int (4)

101 - length of signature bytes

Signature bytes

byte[]

Serialized Signature bytes

File Hash Calculation

In version 5 there are 3 different hashes being calculated:

Object Running Hash

  • Object Running hash is calculated based on hash of each object, which allows us to remove contents of an object from a stream file while maintaining an unbroken chain of hashes.

  • The object running hash will be saved in state, so that reconnect nodes will be able to continue generating stream files identical to those generated by other nodes after reconnecting.

  • Object running hash is calculated as the following: hash(Object Running Hash || hash(OBJECT))

    • || denotes concatenation

    • In the event stream file OBJECT will be each event;

    • In the record stream file OBJECT will be the Record Stream Object;

Entire .rcd Hash

  • This hash is calculated on all the bytes of a .rcd file.

  • With the entire hash, mirror node can download valid .rcd files whose entire hash are agreed on by valid signatures of at least 1/3 of nodes.

  • Suppose the content of a .rcd file with version 5’s content is as following: f[i] = head[i] || startHash[i] || contents[i] || endHash[i]

    • || denotes concatenation

    • f[i] denotes the whole content of a .rcd file;

    • head[i] denotes the bytes before the start Object Running Hash;

    • startHash[i] denotes the Object Running Hash before writing this file;

    • contents[i] denotes the content between the start Object Running Hash and the end Object Running Hash;

    • endHash [i] denotes the Object Running Hash before closing this file;

  • Entire hash is calculated as following:

entireHash[i] = hash(head[i] || startHash[i] || contents[i] || endHash[i])

Metadata .rcd Hash

  • This hash is calculated on metadata bytes in the file, denoted in the table.

  • If some contents of an object were removed from a file, the metadata hash would still be valid.

  • Metadata hash is calculated as following:

metaHash[i] = hash(head[i] || startHash[i] || endHash[i])

Migration fo the record stream from v2 to v5

Hash Bytes in Start Object Running Hash in the first v5 .rcd file is the same as File Hash in the last v2 .rcd_sig file, which is also the hash of the last v2 .rcd file.

Version 3 Event Stream File Format

Event Stream Files Names

A string representation of the Instant of consensus timestamp of the first transaction in the file using ISO-8601 representation, with colons converted to underscores for Windows compatibility. The nano-of-second outputs zero, three, six or nine digits as necessary.

Example of an event stream file name with corresponding signature file:

Event Stream File (.evts): 2020-10-19T21_35_39.454265Z.evts

Event Stream Signature File (.evts_sig): 2020-10-19T21_35_39.454265Z.evts_sig

Event Stream File Format (.evts)

The below table describes the content that can be parsed from an event stream file.

Name

Type (Bytes)

Description

Event Stream File Format Version

int (4)

Value: 3

Prev File Hash Marker

byte

Value: 1

Prev File Hash

byte[48]

SHA384 hash of previous .evts file, if not exist then all zeroes

Event Marker

byte

Value: 90

Event Format Version

int (4)

Value: 3

Event

byte[]

serialized Event bytes

see ` ConsensusEvent` table for details

Event's Hash

byte[]

This Event’s Hash

Event Marker

byte

Value: 90

...

Event Stream Signature File Format (.evts_sig)

Note: Event stream signature file format does not have a version number

The below table describes the content that can be parsed from an event stream signature file.

Name

Type (Bytes)

Description

File Hash Marker

byte

Value: 4

File Hash

byte[48]

SHA384 hash of corresponding *.evts file

Signature Marker

byte

Value: 3

Length of Signature

int (4)

Byte size of the following signature bytes

Signature

byte[]

Signature bytes

Version 5 Event Stream File Format

Generated using functionality provided by swirlds-common for serializing objects to .evts file, and deserializing objects from .evts file. In the next release, swirlds-common will contain functionalities for parsing and verifying version 5 event stream files, which can be used by both services and mirror node.

Event Stream Files Names

A string representation of the Instant of consensus timestamp of the first transaction in the file using ISO-8601 representation, with colons converted to underscores for Windows compatibility.

The nano-of-second always outputs nine digits with padding when necessary, to ensure same length filenames and proper sorting.

Example of an event stream file name with corresponding signature file.

Example 1:

Event Stream File (.evts): 2020-10-19T21_35_39.000000000Z.evts

Event Stream Signature File (.evts_sig): 2020-10-19T21_35_39.000000000Z.evts_sig

Example 2:

Event Stream File (.evts): 2020-10-19T21_35_39.454265000Z.evts

Event Stream Signature File (.evts_sig): 2020-10-19T21_35_39.454265000Z.evts_sig

Event Stream File Format (.evts)

The below tables describe the information that can be parsed from an event stream file.

MD – Meta Data

Name

Type (Bytes)

Description

MD

Event Stream File Format Version

int (4)

Event stream file format version,

Value: 5

y

Object Stream Version

int (4)

Value: 1

This defines the format of the remainder of the file. This version number is used when parsing a stream file with methods defined in swirlds-common package

y

Start Object Running Hash

byte[]

Running Hash of all Events reached consensus before writing this file

y

1st ConsensusEvent

byte[]

Serialized ConsensusEvent bytes.

The following ` ConsensusEvent ` table contains detail of a ConsensusEvent

n

2nd ConsensusEvent

byte[]

...

n

...

...

...

n

End Object Running Hash

byte[]

Running Hash of all Events reached consensus before closing this file

y

ConsensusEvent

Name

Type (Bytes)

Description

Class ID

long (8)

Value: 0xe250a9fbdcc4b1ba

Class version

Int

Value: 1

BaseEventHashedData

byte[]

Serialized BaseEventHashedData bytes

BaseEventUnhashedData

byte[]

Serialized BaseEventUnhashedData bytes

ConsensusData

byte[]

Serialized ConsensusData bytes

BaseEventHashedData

Name

Type (Bytes)

Description

Class Version

int (4)

Value: 1

Creator’s Id

long (8)

ID of this event's creator

SelfParent's Generation

long (8)

the generation for the event’s self parent

OtherParent's Generation

long (8)

the generation for the event’s other parent

SelfParent's Hash

Class version

int (4)

Value: 1

SelfParent's Hash

Digest Type

int (4)

Value: 0x58ff811b

denotes SHA-384

SelfParent's Hash

length

int (4)

Value: 48 for SHA-384

SelfParent's

Hash bytes

byte[]

SelfParent's Hash bytes

OtherParent's Hash

Class version

int (4)

Value: 1

OtherParent's Hash

Digest Type

int (4)

Value: 0x58ff811b

denotes SHA-384

OtherParent's Hash

length

int (4)

Value: 48 for SHA-384

OtherParent's Hash

bytes

byte[]

OtherParent's Hash bytes

Time Created

byte[]

Creation time of this event

See `Instant` table for details

Transactions

byte[]

A serialized array of Transactions,

Details is shown in the following table

Transactions: serialized form of an array of transactions

Name

Type (Bytes)

Description

Size of the Transaction Array

int (4)

Number of transactions in this array.

-1 denotes null.

0 denotes empty array.

Stop deserialization when size is –1 or 0.

All Same Class

boolean (1 bit)

Value: true

1st

Transaction

isNull

boolean (1 bit)

Whether this transaction is null

Transaction Class Version

int (4)

Value: 1

Length of Contents

int (4)

Length of contents in this transaction

Checksum

int (4)

277 – contents.length

System

boolean (1 bit)

Whether this transaction was originated by the application or the platform

Contents

byte[]

Contents bytes

Size of Signatures List

int (4)

Value: 0

signatures are not serialized, so the size is 0

CheckSum

int (4)

Value: 353

2nd

Transaction

...

...

...

...

...

...

...

Instant

Name

Type (Bytes)

Description

Epoch Second

long (8)

If this instant is null, Epoch Second will be Long.MIN_VALUE

Nanos

long (8)

If the instant is null, Nanos will not be written

BaseEventUnhashedData

Name

Type (Bytes)

Description

Class Version

int (4)

Value: 1

Creator Sequence

long (8)

Sequence number for this by its creator (0 is first)

OtherParent’s ID

long (8)

ID of otherParent

OtherParent’s Sequence

long (8)

Sequence number for otherParent event

Length of Signature

int (4)

Length of signature bytes

Signature bytes

byte[]

Creator's signature bytes for this event

ConsensusData

Name

Type (Bytes)

Description

Class Version

int (4)

Value: 2

Generation

long (8)

Generation (which is 1 plus max of parents' generations)

Round Created

long (8)

The created round of this event

isStale

boolean (1 bit)

Is there a consensus that this event is stale (no order, transactions ignored)

Last Event in Round Received

boolean (1 bit)

Is this event the last in consensus order of all those with the same received round

Consensus Timestamp

byte[]

The community's consensus timestamp for this event

See `Instant` table for details

Round Received

long (8)

Round where >=1/2 famous see this event

Consensus Order

long (8)

The event’s consensus order in history (0 first)

Event Stream Signature File Format (.evts_sig)

In version 5, the event stream signature file format is the same as record stream signature file format. The below table describes the content that can be parsed from an event stream signature file.

Name

Type (Bytes)

Description

Signature File Format Version

byte

Value: 5

Object Stream Signature Version

int (4)

Value: 1

This defines the format of the remainder of the signature file. This version number is used when parsing a signature file with methods defined in swirlds-common package

Entire Hash of the corresponding stream file

byte[]

Hash of the entire corresponding stream file

Signature on hash bytes of Entire Hash

byte[]

A signature object generated by signing the hash bytes of Entire Hash. See ` Signature ` table below for details

Metadata Hash of the corresponding stream file

byte[]

Metadata Hash of the corresponding stream file

Signature on hash bytes of Metadata Hash

byte[]

A signature object generated by signing the hash bytes of Metadata Hash