Record and Event Stream File Formats

Search…
Overview
The Hedera network produces record stream files and event stream files that captures chronological information about transactions that took place on the network. A record stream file is a file that contains a series of transactions in chronological order that occurred within a two second interval. The record stream file (.rcd) contains a transaction record for each transaction in that file. For each record stream file there is a corresponding signature file (rcd_sig) that includes the signature generated by the node. The event stream file(.evts) contains information about an event. Event information includes event hash data, event unhashed data, and consensus data. Each event stream files are created in 5 second intervals. Each event stream file has a corresponding signature file (.evts_sig) that includes the node signature.
The record stream file and event stream file formats will both be updated to version 5 in a future mainnet release.
Version Migration
File Type
Current Version
Next Version
Record Stream File
​2​
​5​
Record Stream Signature File
​No version number​
​5​
Event Stream File
​3​
​5​
Event Stream Signature File
​No version number​
​5​
Note: Current record signature files (rcd_sig) and event signature (evts_sig) file formats do not have a version number. The first byte's value is 4, which denotes a marker. For maintaining backwards compatibility, we made the first byte in the new version stream signature file be 5, which denotes the version. Thus, we use version 5 as the next version number.
Version 2 Record Stream File Format
Record Stream Files Names
A record stream file name is made up of a string representation of the Instant of consensus timestamp of the first transaction in the file using ISO-8601 representation, with colons converted to underscores for Windows compatibility. The nano-of-second outputs zero, three, six or nine digits as necessary.
Examples of a record stream file name with corresponding signature file:
Example 1:
Record Stream File: 2020-10-19T21_35_39Z.rcd
Record Stream Signature File: 2020-10-19T21_35_39Z.rcd_sig
Example 2:
Record Stream File: 2020-10-19T21_35_39.454265Z.rcd
Record Stream Signature File: 2020-10-19T21_35_39.454265Z.rcd_sig
Record Stream File Format (.rcd)
The below table describes the content that can be parsed from a record file.
Name
Type (Bytes)
Description
Record Stream File Format Version
int (4)
Record stream file format version,
Value: 2
Hapi Version
int (4)
HAPI protocol version,
Value: 3
Prev File Hash Marker
byte
Value: 1
Prev File Hash
byte[48]
SHA384 hash of previous file, if not exist then all zeroes
Record Marker
byte
Value: 2
Length of Transaction
int (4)
Byte size of the following Transaction message
Transaction
byte[]
Serialized Transaction message bytes
Length of TransactionRecord
int (4)
Byte size of the following TransactionRecord message
TransactionRecord
byte[]
Serialized TransactionRecord message bytes
Record Marker
byte
Value: 2
...
...
...
The signature file *.rcd_sig signs the hash of corresponding *.rcd file;
Record Stream Signature File Format (.rcd_sig)
Note: Event stream signature file format does not have a version number
The below table describes the content that can be parsed from a record signature file.
Name
Type (Bytes)
Description
File Hash Marker
byte
Value: 4
File Hash
byte[48]
SHA384 hash of corresponding *.rcd file
Signature Marker
byte
Value: 3
Length of Signature
int (4)
Byte size of the following signature bytes
Signature
byte[]
Signature bytes
File Hash Calculation
The record file hash is calculated using the following formula.
h[i] = hash(p[i-1] || h[i-1] || hash(c[i-1]))
|| denotes concatenation
h[i] is the hash of the file i
p[i-1] is the contents in the file before the PREV_FILE_HASH
h[i-1] is the hash of file i-1, i.e., PREV_FILE_HASH
c[i-1] is the contents of the file after PREV_FILE_HASH
Version 5 Record Stream File Format
Generated using functionality provided by swirlds-common in the swirlds-platform sdk for serializing objects to record files (.rcd), and deserializing objects from record file (.rcd). In the next release, swirlds-common package will contain functionalities for parsing and verifying version 5 record stream files, which can be used by both services and mirror node.
Record Stream Files Names
A record file name is created using a string representation of the Instant of consensus timestamp of the first transaction in the file using ISO-8601 representation, with colons converted to underscores for Windows compatibility.
The nano-of-second always outputs nine digits with padding when necessary, to ensure same length filenames and proper sorting.
Examples of a record stream file name with corresponding signature file:
Example 1:
Record File (.rcd): 2020-10-19T21_35_39.000000000Z.rcd
Record Signature File(.rcd_sig): 2020-10-19T21_35_39.000000000Z.rcd_sig
Example 2:
Record File (.rcd): 2020-10-19T21_35_39.454265000Z.rcd
Record Signature File(.rcd_sig): 2020-10-19T21_35_39.454265000Z.rcd_sig
Record Stream File Format (.rcd)
MD – Meta Data
The below tables describe the content that can be parsed from a record file.
Name
Type (Bytes)
Description
MD
Record Stream File Format Version
int (4)
Record stream file format version,
Value: 5
y
Hapi Proto Major Version
int (4)
Matches NetworkGetVersionInfo.hapiProtoVersion
Major.Minor.Patch
Major version: 0;
Minor version: 9;
Patch version: 0
y
Hapi Proto Minor Version
int (4)
​
y
Hapi Proto Patch Version
int (4)
​
y
Object stream version
int (4)
Value: 1
This defines the format of the remainder of the file. This version number is used when parsing a stream file with methods defined in swirlds-common package
y
Start Object Running Hash
byte[]
Running Hash of all RecordStreamObject before writing this file.
See Hash object table below for details
y
1st RecordStreamObject
byte[]
Serialized RecordStreamObject bytes.
See RecordStreamObject table below for details
n
2nd RecordStreamObject
byte[]
...
n
...
...
...
n
End Object Running Hash
byte[]
Running Hash of all RecordStreamObject before closing this file.
y
Hash Object
Name
Type (Bytes)
Description
Class ID
long (8)
Value: 0xf422da83a251741e
Class Version
int (4)
Value: 1
This version number will be updated each time when the serialization format of a Hash object is changed
Digest Type
int (4)
Value: 0x58ff811b
denotes SHA-384
Length of Hash
int (4)
Value: 48 for SHA-384
Hash Bytes
byte[]
Serialized Hash bytes
Record Stream Object
Name
Type (Bytes)
Description
Class ID
long (8)
Value: 0xe370929ba5429d8b
Class Version
int (4)
Value: 1
Length of TransactionRecord
int (4)
Byte size of the following TransactionRecord message
TransactionRecord
byte[]
Serialized TransactionRecord message bytes
Length of Transaction
int (4)
Byte size of the following Transaction message
Transaction
byte[]
Serialized Transaction message bytes
Record Stream Signature File Format (.rcd_sig)
In version 5, the record stream signature file format is the same as event stream signature file format. The below tables describe the content that can be parsed from a record signature file.
Name
Type (Bytes)
Description
Signature File Format Version
byte
Value: 5
Object Stream Signature Version
int (4)
Value: 1
This defines the format of the remainder of the signature file. This version number is used when parsing a signature file with methods defined in swirlds-common package
Entire Hash of the corresponding stream file
byte[]
Hash of the entire corresponding stream file
Signature on hash bytes of Entire Hash
byte[]
A signature object generated by signing the hash bytes of Entire Hash. See ` Signature ` table below for details
Metadata Hash of the corresponding stream file
byte[]
Metadata Hash of the corresponding stream file
Signature on hash bytes of Metadata Hash
byte[]
A signature object generated by signing the hash bytes of Metadata Hash
Signature Object
Name
Type (Bytes)
Description
Class ID
long (8)
Value: 0x13dc4b399b245c69
Class Version
int (4)
Value: 1
SignatureType
int (4)
Value: 1
Denotes SHA384withRSA
Length of Signature
int (4)
Byte size of the signature bytes
CheckSum
int (4)
101 - length of signature bytes
Signature bytes
byte[]
Serialized Signature bytes
File Hash Calculation
In version 5 there are 3 different hashes being calculated:
Object Running Hash
Object Running hash is calculated based on hash of each object, which allows us to remove contents of an object from a stream file while maintaining an unbroken chain of hashes.
The object running hash will be saved in state, so that reconnect nodes will be able to continue generating stream files identical to those generated by other nodes after reconnecting.
Object running hash is calculated as the following:

hash(Object Running Hash || hash(OBJECT)) \
|| denotes concatenation
In the event stream file OBJECT will be each event;
In the record stream file OBJECT will be the Record Stream Object;
Entire .rcd Hash
This hash is calculated on all the bytes of a .rcd file.
With the entire hash, mirror node can download valid .rcd files whose entire hash are agreed on by valid signatures of at least 1/3 of nodes.
Suppose the content of a .rcd file with version 5’s content is as following:

f[i] = head[i] || startHash[i] || contents[i] || endHash[i] \
|| denotes concatenation
f[i] denotes the whole content of a .rcd file;
head[i] denotes the bytes before the start Object Running Hash;
startHash[i] denotes the Object Running Hash before writing this file;
contents[i] denotes the content between the start Object Running Hash and the end Object Running Hash;
endHash [i] denotes the Object Running Hash before closing this file;
Entire hash is calculated as following:
entireHash[i] = hash(head[i] || startHash[i] || contents[i] || endHash[i])
Metadata .rcd Hash
This hash is calculated on metadata bytes in the file, denoted in the table.
If some contents of an object were removed from a file, the metadata hash would still be valid.
Metadata hash is calculated as following:
metaHash[i] = hash(head[i] || startHash[i] || endHash[i]) \
Migration fo the record stream from v2 to v5
Hash Bytes in Start Object Running Hash in the first v5 .rcd file is the same as File Hash in the last v2 .rcd_sig file, which is also the hash of the last v2 .rcd file.
Version 3 Event Stream File Format
Event Stream Files Names
A string representation of the Instant of consensus timestamp of the first transaction in the file using ISO-8601 representation, with colons converted to underscores for Windows compatibility. The nano-of-second outputs zero, three, six or nine digits as necessary.
Example of an event stream file name with corresponding signature file:
Event Stream File (.evts): 2020-10-19T21_35_39.454265Z.evts
Event Stream Signature File (.evts_sig): 2020-10-19T21_35_39.454265Z.evts_sig
Event Stream File Format (.evts)
The below table describes the content that can be parsed from an event stream file.
Name
Type (Bytes)
Description
Event Stream File Format Version
int (4)
Value: 3
Prev File Hash Marker
byte
Value: 1
Prev File Hash
byte[48]
SHA384 hash of previous .evts file, if not exist then all zeroes
Event Marker
byte
Value: 90
Event Format Version ****
int (4)
Value: 3
Event
byte[]
serialized Event bytes
see ConsensusEvent table for details
Event's Hash
byte[]
This Event’s Hash
Event Marker
byte
Value: 90
...
…
…
Event Stream Signature File Format (.evts_sig)
Note: Event stream signature file format does not have a version number
The below table describes the content that can be parsed from an event stream signature file.
Name
Type (Bytes)
Description
File Hash Marker
byte
Value: 4
File Hash
byte[48]
SHA384 hash of corresponding *.evts file
Signature Marker
byte
Value: 3
Length of Signature
int (4)
Byte size of the following signature bytes
Signature
byte[]
Signature bytes
Version 5 Event Stream File Format
Generated using functionality provided by swirlds-common for serializing objects to .evts file, and deserializing objects from .evts file. In the next release, swirlds-common will contain functionalities for parsing and verifying version 5 event stream files, which can be used by both services and mirror node.
Event Stream Files Names
A string representation of the Instant of consensus timestamp of the first transaction in the file using ISO-8601 representation, with colons converted to underscores for Windows compatibility.
The nano-of-second always outputs nine digits with padding when necessary, to ensure same length filenames and proper sorting.
Example of an event stream file name with corresponding signature file.
Example 1:
Event Stream File (.evts): 2020-10-19T21_35_39.000000000Z.evts
Event Stream Signature File (.evts_sig): 2020-10-19T21_35_39.000000000Z.evts_sig
Example 2:
Event Stream File (.evts): 2020-10-19T21_35_39.454265000Z.evts
Event Stream Signature File (.evts_sig): 2020-10-19T21_35_39.454265000Z.evts_sig
Event Stream File Format (.evts)
The below tables describe the information that can be parsed from an event stream file.
MD – Meta Data
Name
Type (Bytes)
Description
MD
Event Stream File Format Version
int (4)
Event stream file format version,
Value: 5
y
Object Stream Version
int (4)
Value: 1
This defines the format of the remainder of the file. This version number is used when parsing a stream file with methods defined in swirlds-common package
y
Start Object Running Hash
byte[]
Running Hash of all Events reached consensus before writing this file
y
1st ConsensusEvent
byte[]
Serialized ConsensusEvent bytes.
The following ConsensusEvent table contains detail of a ConsensusEvent
n
2nd ConsensusEvent
byte[]
...
n
...
...
...
n
End Object Running Hash
byte[]
Running Hash of all Events reached consensus before closing this file
y
ConsensusEvent
Name
Type (Bytes)
Description
Class ID
long (8)
Value: 0xe250a9fbdcc4b1ba
Class version
Int
Value: 1
BaseEventHashedData
byte[]
Serialized BaseEventHashedData bytes
BaseEventUnhashedData
byte[]
Serialized BaseEventUnhashedData bytes
ConsensusData
byte[]
Serialized ConsensusData bytes
BaseEventHashedData
Name
Type (Bytes)
Description
Class Version
int (4)
Value: 1
Creator’s Id
long (8)
ID of this event's creator
SelfParent's Generation
long (8)
the generation for the event’s self parent
OtherParent's Generation
long (8)
the generation for the event’s other parent
SelfParent's Hash
Class version
int (4)
Value: 1
SelfParent's Hash
Digest Type
int (4)
Value: 0x58ff811b
denotes SHA-384
SelfParent's Hash
length
int (4)
Value: 48 for SHA-384
SelfParent's
Hash bytes
byte[]
SelfParent's Hash bytes
OtherParent's Hash
Class version
int (4)
Value: 1
OtherParent's Hash
Digest Type
int (4)
Value: 0x58ff811b
denotes SHA-384
OtherParent's Hash
length
int (4)
Value: 48 for SHA-384
OtherParent's Hash
bytes
byte[]
OtherParent's Hash bytes
Time Created
byte[]
Creation time of this event
See Instant table for details
Transactions
byte[]
A serialized array of Transactions,
Details is shown in the following table
Transactions: serialized form of an array of transactions
Text
Name
Type (Bytes)
Description
​
Size of the Transaction Array
int (4)
Number of transactions in this array.
-1 denotes null.
0 denotes empty array.
Stop deserialization when size is –1 or 0.
​
All Same Class
boolean (1 bit)
Value: true
1st
Transaction
isNull
boolean (1 bit)
Whether this transaction is null
Transaction Class Version
int (4)
Value: 1
​
Length of Contents
int (4)
Length of contents in this transaction
​
Checksum
int (4)
277 – contents.length
​
System
boolean (1 bit)
Whether this transaction was originated by the application or the platform
​
Contents
byte[]
Contents bytes
​
Size of Signatures List
int (4)
Value: 0
signatures are not serialized, so the size is 0
​
CheckSum
int (4)
Value: 353
​
2nd
Transaction
...
...
...
...
...
...
...
Instant
Name
Type (Bytes)
Description
Epoch Second
long (8)
If this instant is null, Epoch Second will be Long.MIN_VALUE
Nanos
long (8)
If the instant is null, Nanos will not be written
BaseEventUnhashedData
Name
Type (Bytes)
Description
Class Version
int (4)
Value: 1
Creator Sequence
long (8)
Sequence number for this by its creator (0 is first)
OtherParent’s ID
long (8)
ID of otherParent
OtherParent’s Sequence
long (8)
Sequence number for otherParent event
Length of Signature
int (4)
Length of signature bytes
Signature bytes
byte[]
Creator's signature bytes for this event
ConsensusData
Name
Type (Bytes)
Description
Class Version
int (4)
Value: 2
Generation
long (8)
Generation (which is 1 plus max of parents' generations)
Round Created
long (8)
The created round of this event
isStale
boolean (1 bit)
Is there a consensus that this event is stale (no order, transactions ignored)
Last Event in Round Received
boolean (1 bit)
Is this event the last in consensus order of all those with the same received round
Consensus Timestamp
byte[]
The community's consensus timestamp for this event
See Instant table for details
Round Received
long (8)
Round where >=1/2 famous see this event
Consensus Order
long (8)
The event’s consensus order in history (0 first)
Event Stream Signature File Format (.evts_sig)
In version 5, the event stream signature file format is the same as record stream signature file format. The below table describes the content that can be parsed from an event stream signature file.
Name
Type (Bytes)
Description
Signature File Format Version
byte
Value: 5
Object Stream Signature Version
int (4)
Value: 1
This defines the format of the remainder of the signature file. This version number is used when parsing a signature file with methods defined in swirlds-common package
Entire Hash of the corresponding stream file
byte[]
Hash of the entire corresponding stream file
Signature on hash bytes of Entire Hash
byte[]
A signature object generated by signing the hash bytes of Entire Hash. See ` Signature ` table below for details
Metadata Hash of the corresponding stream file
byte[]
Metadata Hash of the corresponding stream file
Signature on hash bytes of Metadata Hash
byte[]
A signature object generated by signing the hash bytes of Metadata Hash