Hedera Mirror Node
Hedera mirror node release notes
This release comes with a number of improvements to support the JSON-RPC Relay. We now support alias and EVM address lookups for the
account.id parameter on the balances endpoint. We optimized the transaction nonce filtering in /api/v1/contracts/{id}/results by denormalizing the data. Finally, an issue with empty function_parameters in /api/v1/contracts/results/{id} response was addressed.The other big item we worked on was support for volume level compression with Citus. We know that the type of time series data the mirror node stores would be highly compressible and we wanted to use that to our advantage to both reduce increasing storage costs and improve read/write performance. PostgreSQL supports a basic form of compression at the column level called TOAST, but it only takes effect for very large columns. Citus has compression when using their columnar storage access method, but we found it to be too slow for our needs. Since with Citus we knew we wouldn't be using a SaaS service we had more control over the database deployment, so we decided to experiment with Kubernetes volume compression. By creating custom Kubernetes node pools exclusively for Citus, we could install ZFS via the zfs-localpv and enable Zstandard compression on the database's underlying volume. The results were a 3.6x compression ratio with zero loss in performance. To put into perspective, that means the current mainnet database size of 17TB could be reduced to 4.7TB.
Other areas of improvement include improving documentation around disaster recovery efforts. This includes a runbook on restoring a mirror node from backup. There's also a runbook on how to perform local stream file verification. Acceptance tests have been previously integrated into the automated deployment process but suffered from a long execution time mainly due to using Gradle to download dependencies at runtime. We containerized the acceptance tests so the dependencies are downloaded at build time reducing runtime by 3-4x.
Work continues on HIP-584 with this release the first to support non-static contract state reads for non-precompile functions. Please see the Swagger UI table for
/api/v1/contracts/call for a breakdown of which functionality is supported in what release. More estimate gas functionality was copied from services code to make progress on estimation. A new stacked state frame functionality was added to be used in the future to support contract writes and cached reads.The Spotless code formatting tool was used to format the entire codebase to be consistent. A git commit hook was added to ensure any new changes stays consistent and developers can focus on what matters.
Finally, there were a large number of bug fixes and performance improvements. See below for the full details.
MAINNET UPDATE COMPLETED: MAY 9, 2023
TESTNET UPDATE COMPLETED: MAY 2, 2023
HIP-584 EVM archive node saw further progress this release with a focus on testing and establishing the foundation for estimate gas functionality in the next release. While consensus nodes undergo a modularization effort that will pay dividends down the road, the archive node needs functionality for estimate gas before that process could be completed. To make progress on HIP-584, the necessary EVM logic was temporarily copied from consensus nodes into the mirror node web3 module. A large focus was placed on increasing acceptance test coverage for contract call with precompiles.
Users writing dApps want to be able to monitor for token approval and transfer events. HAPI transactions like
CryptoTransfer, CryptoApproveAllowance, CryptoDeleteAllowance, TokenMint, TokenWipe, and TokenBurn do not emit events that could be captured by monitoring tools like The Graph since they’re executed outside the EVM. To address, the mirror node now generates synthetic contract log events for these non-EVM HAPI transactions.A new subscription API was designed for HIP-668 GraphQL API. Once it's implemented in a future release, the new contract log subscription will stream contract events to clients via a WebSocket connection.
For our Citus database transition, PostgreSQL 15 compatibility was verified and made the default for this v2 schema. The lookup of historical balance information via
/api/v1/balances?timestamp= was optimized for sharded databases so it stays performant. Performance testing showed a decrease in shard count could greatly improve performance so we lowered the number of shards from 32 to 16. This testing also allowed us to provide an initial recommended resource configuration for the Citus deployment.There was a large focus on test improvements in this release. In addition to the aforementioned HIP-584 test coverage, we also optimized the acceptance tests to reduce the overall test duration in Kubernetes by half without reducing coverage. The acceptance test logs were cleaned up to reduce unnecessary log statements and standardize its output. The hbar balance used by the tests now is logged at the end of test execution. Acceptance tests for hollow account creation were added. We now generate multi-platform snapshot images from the
main branch for testing with local node. Testkube configuration was enhanced to make it more configurable. Finally, all Java test compiler warnings were fixed and will now fail the build if any future warnings occur.There is a bug introduced by #5776 that causes the importer to fail on startup. It's recommended to hold off on upgrading to v0.79.0 until we can address this in a v0.79.1. Alternatively, it can be worked around by disabling the faulty migration by setting
hedera.mirror.importer.migration.syntheticTokenAllowanceOwnerMigration.enabled=false.MAINNET UPDATE COMPLETED: APRIL 21, 2023
TESTNET UPDATE COMPLETED: APRIL 13, 2023
HIP-584 Mirror Evm Archive Node now has token precompile support. This was the last major piece of functionality needed for the
/api/v1/contracts/call to be considered eth_call equivalent. The new API was added to the REST API's OpenAPI documentation so that it appears on our Swagger UI. A number of performance optimizations were worked on to make it scalable as well as various test improvements to verify its correctness. Various bugs were addressed including the proper handling of reverts. In the next few releases, we plan to fine tune contract call and implement contract gas estimation.A large focus was put on performance and resiliency this release. On the performance front, we've optimized the list schedules REST API to be scalable on Citus. Performance tests can now trigger automatically via TestKube once the helm tests complete. Those same k6 performance tests were enhanced to automatically pick appropriate configuration values specific to the environment. The transaction hash table was partitioned and the ingest process made to insert hashes in parallel. This change dramatically speeds up the time to insert the optional transaction hashes. Similarly, an option was added to control which transaction types should cause a hash insertion.
On the resiliency front, the importer component was analyzed for any code paths that may cause record file processing to halt due to bad input from consensus nodes. Any such code was made to handle the error, log/notify, and move on to the next transaction. This change makes the mirror node ingestion more resilient and moves toward preferring availability over correctness. Partial mirror nodes that might become stuck due to having an incomplete address book can now continue to ingest with a new
consensusMode property and logic. Partial mirror nodes will now also be able to have a corrected account and token balance even if the entity was missing a deleted flag. Finally, we were able to complete a longstanding refactoring effort to move all transaction specific logic to individual transaction handlers and fixed a number of bugs in the process.There were a few important bugs fixed in this release that are worth noting. A fix was put in place to correct inaccurate fungible token total supply. Additionally, NFTs for deleted tokens no longer appear as active in the REST API.
MAINNET UPDATE COMPLETED: APRIL 4, 2023
TESTNET UPDATE COMPLETED: MARCH 29, 2023
This release fixes the tracking of NFT balances. Historically, these came from the balance file sent by the consensus nodes every 15 minutes. When we started tracking the live fungible token balances and moved away from using this balance file we unfortunately broke the NFT balance calculation. We not only fixed the issue but went ahead and took the time to track the up-to-date NFT balance as well.
The
/api/v1/contracts/{id}/state REST API shows the current state of a contract's slot values. Users requested the ability to query for the key/value pairs for their contract at an arbitrary point in the past. To address, we now expose a timestamp query parameter that will get the historical contract state. This allows the JSON-RPC relay to offer a proper eth_getStorageAt with support for historical blocks.HIP-584 continues to make progress. Quite a few bugs were squashed including handling reverts and populating the revert reason and raw data. Performance tests were added to k6 to load test contract calls with token precompiles.
MAINNET UPDATE COMPLETED: MARCH 23, 2023
TESTNET UPDATE COMPLETED: MARCH 13, 2023
The new
/api/v1/contracts/call REST API as specified in HIP-584 is finally ready for initial production use. This release adds support for rate limiting the API with an initial value of 100 requests per second per instance. Tags were added to the gas per second metric to indicate if the request was a call, an estimate, or resulted in an error for increased observability. Various bug fixes were also addressed.HIP-668 GraphQL API was added to our deployment with the addition of a new helm chart for this API. This will allow for initial use of the API in all environments with the understanding that it's still very much alpha and subject to change.
We made a lot of headway on our Citus integration. Citus was upgraded to 11.2 which showed a nice 15-20% performance boost for a number of query patterns. We optimized the contract results APIs performance by distributing based upon contract ID instead of payer account. Search for a transaction by its hash on Citus was improved by adding the distribution column to the query and limiting the results to a timestamp range. The search for an account by its ID also saw improvements on Citus.
The Helm charts contain some breaking changes to be aware of. The
hedera-mirror wrapper chart enables the new hedera-mirror-graphql sub-chart by default. The GraphQL deployment requires a new mirror_graphql database user to exist for it to successfully start up. You can create the user by logging into the database as owner and executing the following SQL query:create user mirror_graphql with login password 'password' in role readonly;
If you're using the embedded PostgreSQL sub-chart (which we don't recommend for production use), then you'll have to delete its StatefulSet before upgrading due to a major bump in its chart version.
The
hedera-mirror-common chart had all of its components upgraded to new major versions that include breaking changes. If you're using this chart, run the following commands before upgrading:kubectl delete daemonset mirror-prometheus-node-exporter
kubectl delete daemonset mirror-traefik
kubectl delete statefulset mirror-loki
kubectl delete ingressroute mirror-traefik-dashboard
kubectl apply --server-side --force-conflicts=true -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.63.0/example/prometheus-operator-crd/monitoring.coreos.com_alertmanagerconfigs.yaml
kubectl apply --server-side --force-conflicts=true -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.63.0/example/prometheus-operator-crd/monitoring.coreos.com_alertmanagers.yaml
kubectl apply --server-side --force-conflicts=true -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.63.0/example/prometheus-operator-crd/monitoring.coreos.com_podmonitors.yaml
kubectl apply --server-side --force-conflicts=true -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.63.0/example/prometheus-operator-crd/monitoring.coreos.com_probes.yaml
kubectl apply --server-side --force-conflicts=true -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.63.0/example/prometheus-operator-crd/monitoring.coreos.com_prometheuses.yaml
kubectl apply --server-side --force-conflicts=true -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.63.0/example/prometheus-operator-crd/monitoring.coreos.com_prometheusrules.yaml
kubectl apply --server-side --force-conflicts=true -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.63.0/example/prometheus-operator-crd/monitoring.coreos.com_servicemonitors.yaml
kubectl apply --server-side --force-conflicts=true -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.63.0/example/prometheus-operator-crd/monitoring.coreos.com_thanosrulers.yaml
MAINNET UPDATE COMPLETED: MARCH 2, 2023
Work continues on HIP-584 to get it closer to production ready for simple contract calls. Caching logic was added to the repository layer to optimize its capability along with performance tests to verify those improvements. A metric was added to track the gas per second being used along with various other bug fixes.
The monitor API and dashboard used internally for observing our production systems was containerized. Additionally, it was integrated into the Helm chart and invoked as part of the Helm tests to ensure the deployment is verified.
Finally, there were a number of query optimizations as part of our Citus effort.
MAINNET UPDATE COMPLETED: FEBRUARY 18, 2023
MAINNET UPDATE COMPLETED: FEBRUARY 14, 2023
This release switches the testnet bucket to the new one created for the testnet reset that occurred on January 26, 2023. It also updates the address book to reflect the additional nodes added to testnet since the last reset. If you're running a testnet mirror node, please see the reset instructions for help getting your node updated.
In HIP-668, we propose adding a new mirror node GraphQL API and would greatly appreciate your feedback. In this release, a new GraphQL module with a simple account lookup query was added to provide the basis for future work on this HIP. In the next release, we will add the automated deployment of this module to all environments. It is considered an alpha API subject to breaking changes at any time, so it's not recommended to depend upon for production use. This has been made explicit by using
/graphql/alpha in its URL.Finally, a number of query optimizations were implemented for Citus while ensuring it doesn't cause regressions with the existing database. This will continue to be the focus of the next few releases.
MAINNET UPDATE COMPLETED: FEBRUARY 10, 2023
TESTNET UPDATE COMPLETED: FEBRUARY 3, 2022
This release is the first one with support for HIP-584 EVM Archive Node. HIP-584 allows mirror nodes to act as a read only EVM for free execution of smart contracts. This new feature is considered alpha with much of the work still to be implemented like support for precompiled contracts, gas estimation, etc. This functionality requires the mirror node to be configured to ingest the optional traceability sidecar files and it requires a network where those files are generated. Currently only previewnet has contract traceability enabled.
The testnet bucket name has been updated to the new bucket name after its recent quarterly reset. Likewise the bootstrap testnet address book was updated to reflect the additional testnet nodes that have been added since the previous reset. Mirror node operators running a testnet node should either manually populate the new bucket name or update to this release.
The remaining work targeted significant testing improvements and bug fixes. Our performance tests were expanded to all endpoints to catch issues earlier in the lifecycle. Additional acceptance test coverage was added along with a number of fixes. CI stability has greatly improved with a focus on fixing flaky tests. Code smells as reported by Sonar were reduced to only a handful and in the next release reduced all the way down to zero. Finally, we merged work that enables nightly performance testing in our integration and mainnet staging environments via TestKube.
MAINNET UPDATE COMPLETED: JANUARY 25, 2022
TESTNET UPDATE COMPLETED: JANUARY 18, 2022
This release is a smaller release as most of the team took time off for the holidays. Still, we managed to implement HIP-583 Expand alias support in CryptoCreate & CryptoTransfer Transactions. We now allow hollow accounts to be later finalized into a contract when it is fully created.
We also worked on adding support for Testkube. Testkube allows us to automate our testing in Kubernetes environments by triggering tests based upon various conditions. Specifically, it will be used to run nightly performance regression tests against a mainnet staging environment to ensure our API performance doesn't regress. We'll continue to expand on this automated testing in future releases.
There were also a number of bug fixes in this release, mainly focused on fixing our release process after the switch from Maven to Gradle in the last release.
MAINNET UPDATE COMPLETED: JANUARY 19, 2023
As of this release, all account and token balances in the REST API will reflect their real time balance information. Historically, the mirror node has relied upon the balance file uploaded by consensus nodes every 15 minutes for its balance information. We've been working towards this milestone for many releases gradually rolling out real-time balance tracking to more entities and more APIs. This release completes this migration with the addition of real time token balances to both the accounts and the balances REST APIs.
The mirror node now implements support for HIP-583 alias on
CryptoCreate transactions. With this, clients can directly set an alias during account creation instead of relying upon the implicit auto-account creation during transfers. The mirror node respects this explicit alias along with the new explicit EVM address in both CryptoCreate or the TransactionRecord. This avoids the brittle EVM address calculation on the mirror node that has caused us some trouble in the past.This release completes the migration from Maven to Gradle for our build process. A lot of work has been put into the new build to improve its performance and stability both locally and in continuous integration (CI). GitHub Actions workflows have been consolidated from one workflow per module to a single Gradle build workflow with a matrix strategy running them in parallel for each module and database schema. This greatly simplifies the workflow configuration making it easier to maintain and debug.
We continue to make progress on our Citus exploration. The v2 schema for Citus now does timestamp based partitioning of data and automates this process via pg_cron. A Citus specific environment was created and we're currently conducting performance tests against it at scale to verify it meets our requirements.
This release adds automation to keep our GCP Marketplace application up to date with each release. While not fully automatic due to the manual nature of Marketplace version submission, now any new production tag will trigger the generation and verification of the marketplace images.
This was a big release and there were a lot of other various improvements and fixes. See the full release note below.
MAINNET UPDATE COMPLETED: DECEMBER 29, 2022
TESTNET UPDATE COMPLETED: DECEMBER 14, 2022
As part of HIP-406, the mirror node is adding a new account staking rewards REST API. This API will show the staking rewards paid to an account over time. The mirror node now also shows staking reward transfers in the transaction REST APIs (e.g.
/api/v1/transactions, /api/v1/transactions/{id}, and the list of transactions in /api/v1/accounts/{id}). This can be useful to show which transaction involving your account after the staking period ended triggered the lazy reward payout.GET /api/v1/accounts/{id}/rewards{
"rewards": [{
"account_id": "0.0.1000",
"amount": 10,
"timestamp": "123456789.000000001"
}],
"links": {
"next": null
}
}
The REST API saw further improvements outside of staking. The accounts REST APIs now show a calculated expiration timestamp to mirror the HAPI
CryptoGetInfo query. Previously expiration timestamp only shows up if explicitly sent via a transaction that supports it (mainly update transactions). Now if it's null we calculate it as created_timestamp.seconds + auto_renew_period. Every contract results endpoint was updated to include an address field for the EVM address of the created contract.This release makes progress on being able to execute contract calls on the mirror node as outlined in HIP-584. A lot of the groundwork is being laid that will be further refined in upcoming releases.\
MAINNET UPDATE COMPLETED: DECEMBER 5, 2022
TESTNET UPDATE COMPLETED: NOVEMBER 29, 2022
As noted in previous releases, HIP-367 is deprecating the token relationship information returned from HAPI queries. In this release, its mirror node replacement is now feature complete. We now track and show the current fungible token balance in the token relationships API instead of relying upon the 15 minute balance export from consensus nodes. In a future release, the accounts and balances REST APIs will be updated to show the current fungible token balance.
The importer component now supports a local file stream provider. This allows it to read stream files from a local directory instead of just the S3-compatible providers it supported previously. This mode is useful for debugging stream files received out of band or for reducing complexity and latency in a local node setup. To try it out, set
hedera.mirror.importer.downloader.cloudProvider=LOCAL and populate the hedera.mirror.importer.dataPath/streams folder with the same file structure as the cloud buckets.We now show a contract's
CREATE2 EVM address in the contract logs REST APIs. Previously, we would convert the Hedera shard.realm.num to a 20-byte EVM address but this did not always reflect the true EVM address of the contract. Using the CREATE2 form of the EVM address provides increased Ethereum compatibility.We continue to make progress on converting our build process to Gradle. This release adds a Golang Gradle plugin to download the Go SDK and use it to build and test the Rosetta module.
MAINNET UPDATE COMPLETED: NOVEMBER 18, 2022
TESTNET UPDATE COMPLETED: NOVEMBER 18, 2022
Besides the usual round of bug fixes, this release focuses on some internal enhancements to lay the groundwork for some upcoming features. We now track and persist the current fungible token balance in the database. This information is not yet exposed on any API but will be rolled out to the token relationships, accounts and balances REST APIs in the near future.
We're continuing our work towards CitusDB as a possible database replacement in this release by adding distribution columns and fixing our v2 schema tests.
Finally, we implemented initial Gradle support to improve build times and provide a better developer experience. Initial testing shows build and test times reduced from 8 minutes overall down to 2 minutes. The Gradle and Maven build scripts will be maintained concurrently for a few releases until we can ensure the Gradle build reaches feature parity with Maven.
MAINNET UPDATE COMPLETED: NOVEMBER 10, 2022
TESTNET UPDATE COMPLETED: NOVEMBER 10, 2022
HIP-367 deprecated the list of token balances for an account returned via HAPI. The mirror node has been working on its replacement for a few releases by storing the current account balance, combining contract and entity tables, and adding a history table for
token_account. This work has paved the way for a new token relationships REST API that will list all fungible and non-fungible tokens associated with a particular account. This API also returns some metadata like balance, KYC status, freeze status, and whether it's an automatic association or not. Currently the fungible token balance being returned is from the 15 minute account balance file. We are actively working towards tracking the real-time fungible token balance and it will be updated to reflect that in a future release.GET /api/v1/accounts/{id}/tokens{
"tokens": [{
"automatic_association": true,
"balance": 15,
"created_timestamp": "1234567890.000000002",
"freeze_status": "UNFROZEN",
"kyc_status": "GRANTED",
"token_id": "0.0.1135"
}],
"links": {
"next": null
}
}
HIP-513 details how smart contract traceability information from consensus nodes is made available via sidecar files. The contract state changes within the sidecar are persisted by mirror nodes and made available on the contract results APIs. However, these state changes do not always reflect the full list of smart contract storage values since not all slots are modified during any particular contract invocation. We now persist a rolled up view of state changes to track the latest slot key/value pairs. This contract state information is now exposed via a new
/api/v1/contracts/{id}/state REST API where id is either the shard.realm.num, realm.num, num, or a hex encoded EVM address of the smart contract.GET /api/v1/contracts/{id}/state{
"state": [{
"address": "0x0000000000000000000000000000000000001f41",
"contract_id": "0.0.100",
"slot": "0x0000000000000000000000000000000000000000000000000000000000000001",
"timestamp": "1676540001.234390005",
"value": "0x0000000000000000000000000000000000000000000000000000000000000010"
}],
"links": {
"next": null
}
}
In a push for further decentralization, we now randomize node used to download data files after reaching consensus. Previously, the data structure we used generally caused us to use the first node returned in the verified list, which was usually
0.0.3. We now randomly pick a node until we can successfully download the stream file. We also internally changed all tables that used node account ID to use its node ID instead.On the testing front, we enhanced various test and monitoring tools to add support for new APIs. We also added an acceptance test startup probe to delay the start of the tests until the network as a whole was healthy. This avoids the mirror node acceptance tests reporting a false positive when a long migration or startup process on the consensus or mirror nodes causes a delay.
MAINNET UPDATE COMPLETED: OCTOBER 24, 2022
TESTNET UPDATE COMPLETED: OCTOBER 17, 2022
Continuing our goal of having up to date balances everywhere, this release now shows the current hbar balance on the balances REST API. If you provide a
timestamp parameter, it will fallback to the previous behavior and use the 15 minute balance file. This allows us to continue to provide a historical view of your balance over time, while also showing the latest balance if no specific time range is requested. Live fungible token balance is actively being worked on for an upcoming release.Another big feature this release is support for cloud storage failover. The importer can now be configured with multiple S3 download sources and will iterate over each until one is successful. This makes the mirror node more decentralized and provides more resiliency in the face of cloud failure. The existing
hedera.mirror.importer.downloader properties used to configure the cloudProvider, accessKey, secretKey, etc. will continue to be supported and inserted as the first entry in the sources list, but it's recommended to migrate your configuration to the newer format. Also in the downloader, we increased the downloader batch size to 100 to improve historical synchronization speed. A hedera.mirror.importer.downloader.sources.connectionTimeout property was added to avoid occasional connection errors.hedera:
mirror:
importer:
downloader:
sources:
- backoff: 30s
connectionTimeout: 10s
credentials:
accessKey: <redacted>
secretKey: <redacted>
maxConcurrency: 50
projectId: myapp
region: us-east-2
type: GCP
uri: https://storage.googleapis.com
- credentials:
accessKey: <redacted>
secretKey: <redacted>
type: S3
HIP-573 gives token creators the option to exempt all of their token’s fee collectors from a custom fee. This mirror node release adds support to persist this new
all_collectors_are_exempt HAPI field and expose it via the /api/v1/tokens/{id} REST API.An important characteristic of a mirror node is that anyone can run one and store only the data they care about. The mirror node has supported such capability for a few years now but the configuration syntax was a bit tricky to get correct. To address this shortcoming, we add some examples to the configuration documentation to clarify things. This entity filtering was historically limited to just create, update and delete operations on entities. We've now expanded this filtering to include payer account IDs and accounts or tokens involved in transfers.
As part of the S3 failover work, we made a number of changes to existing properties to streamline things and only support one property for all stream types. The
hedera.mirror.importer.downloader.(balance|event|record).batchSize properties were removed in favor of a single, generic hedera.mirror.importer.downloader.batchSize. Likewise, the hedera.mirror.importer.downloader.(balance|event|record).threads properties were removed in favor of hedera.mirror.importer.downloader.threads. The hedera.mirror.importer.downloader.(balance|event|record).prefix properties were removed in favor of hardcoded configuration since there's never been a need to adjust these. If you're using any of these properties, please adjust your config accordingly.If you're writing stream files to disk after downloading by enabling the
writeFiles or writeSignatures properties, there is one other breaking change to be aware of. As part of our migration away from node account IDs, we changed the paths on disk to use the node ID as well. If you'd like to avoid having two directories for the same node please rename your local directories manually. For example, change ${dataDir}/recordstreams/record0.0.3 to ${dataDir}/recordstreams/record0.MAINNET UPDATE COMPLETED: OCTOBER 11, 2022
TESTNET UPDATE COMPLETED: OCTOBER 6, 2022
The mirror node now calculates consensus using the staking weight of all the nodes as outlined in HIP-406. If staking is not yet activated, it falls back to the previous behavior of counting each node as
1/N weight where N is the number of consensus nodes running on the network. The /api/v1/accounts and /api/v1/accounts/{id}REST APIs now expose a pending_reward field that provides an estimate of the staking reward payout in tinybars as of the last staking period. The /api/v1/network/supply REST API updated its configured list of unreleased supply accounts to accurately reflect the separation of accounts done for staking purposes by Hedera.This release implements the contract actions REST API detailed in HIP-513. An example of an actions payload is shown below. We added
transaction_hash, transaction_index, block_hash and block_number fields to the contract logs REST APIs. This work was done to optimize the performance for the eth_getLogs JSON-RPC method used by the relay. Also for the relay, we now support lookup of non-Ethereum contract results by its 32 byte transaction hash.GET /api/v1/contracts/results/0.0.5001-1676540001-234390005/actions{
"actions": [{
"call_depth": 1,
"call_type": "CALL",
"call_operation_type": "CALL",
"caller": "0.0.5001",
"caller_type": "CONTRACT",
"from": "0x0000000000000000000000000000000000001389",
"timestamp": "1676540001.234390005",
"gas": 1000,
"gas_used": 900,
"index": 1,
"input": "0xabcd",
"to": "0x70f2b2914a2a4b783faefb75f459a580616fcb5e",
"recipient": "0.0.8001",
"recipient_type": "CONTRACT",
"result_data": "0x1234",
"result_data_type": "OUTPUT",
"value": 100
}],
"links": {
"next": null
}
}
To support mirror node explorers being able to display the transaction that created an entity, we added a
created_timestamp field to the accounts REST API.The Rosetta module saw a large number of improvements this release. Rosetta now supports a transaction memo be returned in its metadata response. To improve performance of large blocks, Rosetta now limits the number of transactions in its block response. The importer balance reconciliation job was disabled in the Rosetta Docker image since Rosetta performs its own reconciliation process. There were other various fixes to the reconciliation process balance offset, charts testing failing for PRs from forked repos, and fixing a slow search block by hash query in rosetta.
MAINNET UPDATE COMPLETED: SEPTEMBER 26 2022
TESTNET UPDATE COMPLETED: SEPTEMBER 12, 2022
In the last release, we began keeping track of the current balance of every account and contract instead of solely relying upon the balance file written every 15 minutes by consensus nodes. In this release, we now show this up-to-date balance on the
/api/v1/accounts and /api/v1/accounts/{id} REST APIs in the existing balance field. Token balances and the balances REST API still shows the balance information from the 15 minute balance file. In a future release, we'll look at changing those to track the current balance.As part of HIP-406, it details a pending reward calculation that can be used to estimate the reward payout between your last payout event and the staking period that just ended. The mirror node now does a similar calculation daily and will in a future release show this pending reward amount on the REST API.
The reconciliation job periodically runs and reconciles the balance files with the crypto transfers that occurred in the record files. This job allowed us to catch an issue with missing transactions for
FAIL_INVALID crypto transfers that was fixed in hedera-services v0.27.7. This release contains the errata for the missing transactions that allows reconciliation to proceed successfully once again. It also saw performance improvements including a delay property to throttle its speed and added job status persistence so it doesn't restart from the beginning every time. A new remediationStrategy property provides a mechanism to continue after failure to aid in debugging multiple reconciliation errors.MAINNET UPDATE COMPLETED: SEPTEMBER 7 2022
TESTNET UPDATE COMPLETED: SEPTEMBER 2, 2022
This release adds a highly requested feature: the mirror node now tracks the current account balance. Previously, the mirror node would store balance information whose source was a balance file that consensus nodes generate and upload every 15 minutes. As a result, balance information was always behind by up to 15 minutes for active accounts. We were able to figure out a way to track this information at scale with SQL in this release. The next release will actually expose this up to date account balance information on both
/api/v1/accounts or /api/v1/accounts/{id}. In future releases, will look at adding live balances to /api/v1/balances when no timestamp parameter is provided and track up to date token balances.Work continues on HIP-513 Contract Traceability, with this release adding a few important items. Consensus nodes will, when first activating the sidecar mechanism, send migration records that includes all smart contract runtime bytecode and current storage values. The mirror node now supports receiving these special migration sidecars and updating its database with the migrated data. This paves the way for the mirror node to have the necessary information to execute smart contracts without modifying state in a future release. Also in this release we now show the contract initcode that was used to unsuccessfully create a smart contract in a new
failed_initcode field in the contract result REST API.The network supply REST API saw an update to adjust the unreleased supply accounts used to calculate the unreleased supply. This change was necessary as Hedera adjusts the treasury accounts for use with staking.
MAINNET UPDATE COMPLETED: AUGUST 29, 2022
TESTNET UPDATE COMPLETED: AUGUST 22, 2022
Mirror Node 0.62 saw HIP-406 staking related improvements to its REST API and partial support for HIP-513 contract traceability.
The
/api/v1/network/nodes will now use the address book stake as a fallback when it has not seen any NodeStakeUpdate transactions on the network. This release also contains a new network stake REST API /api/v1/network/stake to show aggregate stake information common to all nodes:{
"max_staking_reward_rate_per_hbar": 17808,
"node_reward_fee_fraction": 0.0,
"stake_total": 35000000000000000,
"staking_period": {
"from": "1658774045.000000000",
"to": "1658860445.000000000"
},
"staking_period_duration": 1440,
"staking_periods_stored": 365,
"staking_reward_fee_fraction": 1.0,
"staking_reward_rate": 100000000000,
"staking_start_threshold": 25000000000000000
}
HIP-513 Smart Contract Traceability adds support for an optional sidecar to contain contract traceability information. In this release, the mirror node supports downloading and persisting contract state changes, contract initcode, contract runtime bytecode, and contract actions (AKA traces). The
/api/v1/contracts/{id} REST API now shows the runtime bytecode for newly created contracts. The next release will support a sidecar migration that will populate contract state changes and bytecode for all existing contracts.HIP-435 Record Stream V6 required changes to the state proof REST API in order to not break when V6 was enabled. With this release, the API was updated to support record files in the new v6 format.
The Rosetta API saw a few minor fixes and improvements. It now uses the Hedera network alias everywhere in the Rosetta server . It also fixes the issue that Rosetta did not support alias as the from address for crypto transfers. Additionally, the Rosetta
sub_network_identifier was disabled since it was not needed.There were a surprising number of technical debt improvements this release. The REST API and monitor API were both converted from CommonJS to ES6 modules, allowing us to finally upgrade some of our dependencies to the latest version. The REST API spec tests were organization into folders by endpoint and changed to use a single database container for the entire suite. On the importer, mutable contract information was merged into the
entity table. The RecordItem constructor was removed everywhere in favor its builder method. Finally, we added parser performance tests to be able to generate large record files and stress test record file ingestion.In a recent release, we added the
stake_total field to the /api/v1/network/nodes API to show the aggregate stake of the network. With the addition of the new /api/v1/network/stake API, we now have a separate API to return aggregate staking information associated with the network. As such, we made the decision in this release to remove the stake_total field from the response of the /api/v1/network/nodes API to stay consistent. If you're using this field, please update your code to use the stake_total field in the /api/v1/network/stake API.MAINNET UPDATE COMPLETED: AUGUST 2, 2022
TESTNET UPDATE COMPLETED: JULY 27, 2022
This release adds initial support for HIP-513 Smart Contract Traceability Extension. Contract traceability information is now available inside an optional sidecar file uploaded separately to cloud storage. Mirror node operators can choose whether to download this extra information by configuring the
hedera.mirror.importer.parser.record.sidecar properties on the importer. By default, sidecar files will not be downloaded. Enabling it will permit contract state, actions, and bytecode data to be persisted by the mirror node. HIP-513 support is incomplete in this release and the next release will enable full persistence of all sidecar types.The transactions REST API now supports multiple
transactiontype query filters to simplify searches across types.The version of the mirror node in GCP Marketplace was updated to v0.60.0. This required migration to the new GCP Producer Portal which should help streamline future version updates.
The monitor components saw an option added to retrieve the address book on startup. This avoids having to configure the list of nodes to monitor manually in pre-production environments and ensure the list of nodes is up to date. The monitor now uses OpenAPI generated models to dog food our OpenAPI schema. We also added an option to the monitor to set the max memo length property for published transactions.
MAINNET UPDATE COMPLETED: JULY 18, 2022
TESTNET UPDATE COMPLETED: JULY 14, 2022
The two big features of this release are support for a data retention period and HIP-351 pseudorandom number generation.
On public networks, mirror nodes can generate tens of gigabytes worth of data every day and this rate is only projected to increase. Mirror nodes now support an optional data retention period that is disabled by default. When enabled, the retention job purges historical data beyond a configured time period. By reducing the overall amount of data in the database it will reduce operational costs and improve read/write performance. Only insert-only data associated with balance or transaction data is deleted. Cumulative entity information like accounts, contracts, etc. are not deleted.
HIP-351 adds a pseudorandom number generator transaction. The mirror node now persists this
PrngTransaction type including the pseudorandom number or the bytes it generates. A future release will expose this information on the REST API.There were various other improvements in this release. Block numbers are now migrated to be consistent with other mirror nodes regardless of their configured start date when it receives the first v6 record file with the canonical block number from consensus nodes. We added the reward rate at the start of the staking period to the nodes REST API. Rosetta now shows fee crypto transfers operation type as
FEE. Rosetta also shows account aliases as account addresses in Rosetta DATA API response.TESTNET UPDATE COMPLETED: JUNE 29, 2022
The previous release saw support for the persistence of HIP-406 staking-related data. Staking persistence saw further fine-tuning in this release to adapt to changes in the
NodeStakeUpdateTransaction protobuf. The decline_reward, staked_account_id, staked_node_id fields were added to /api/v1/accounts and /api/v1/accounts/{id} to show account-level staking properties. We also added staking related fields to the existing /api/v1/network/nodes REST API (see example below).GET /api/v1/network/nodes{
"nodes": [
{
"description": "address book 1",
"file_id": "0.0.102",
"max_stake": 50000,
"memo": "0.0.4",
"min_stake": 1000,
"node_account_id": "0.0.4",
"node_cert_hash": "0x01d...",
"node_id": 1,
"public_key": "0x4a...",
"service_endpoints": [],
"stake": 20000,
"stake_not_rewarded": 19900,
"stake_rewarded": 100,
"stake_total": 100000,
"staking_period": {
"from": "1655164800.000000000",
"to": "1655251200.000000000"
},
"timestamp": {
"from": "16552512001.000000000",
"to": null
}
}
],
"links": {
"next": null
}
}
Support for the new record file v6 format as defined in HIP-435 was added in this release. Record file v6 adds block number as well as support for the new sidecar record files that carry detailed contract traceability information that mirror nodes can optionally choose to download. The record and signature files are now in a more maintainable protobuf format that should make them easier to enhance with new fields in the future without requiring breaking changes. Also, the v6 record files will now be compressed which should translate into reduced network and storage costs while potentially improving performance. Once v6 is enabled in a future hedera-service's release, mirror node operators will be required to update to a version that supports the new v6 format to avoid downtime.
Rosetta saw a number of improvements this release to better align it with the Rosetta specification. A configurable valid duration seconds option was added to the transaction construction API to support customization of this value. Support for a consistent block number regardless of
startDate was added in Rosetta now that Hedera has a consistent block as defined in HIP-415. A 0x prefix was added to alias addresses returned via the API to denote that the data is hex-encoded.MAINNET UPDATE COMPLETED: JUNE 22, 2022
TESTNET UPDATE COMPLETED: JUNE 16, 2022
This release contains support for HIP-406 Staking, HIP-410 Wrapped Ethereum Transaction, and HIP-482 JSON-RPC Relay as well as a long overdue upgrade to Java 17.
HIP-406 Staking is coming and the mirror node is getting ready for it. This release we added persistence support to store staking information. In the next release, we'll expose this information via our APIs.
HIP-410 and HIP-482 are both intended to improve the onramp for existing Ethereum developers. Towards that end, we added pagination support to both of the contract logs REST APIs. You can now page through logs via a combination of a consensus timestamp and log index parameters. The new blocks REST APIs also saw new
gas_used and logs_bloom fields added that show the aggregated values for all transactions within the block. Finally, we added a new network fee schedule REST API. Currently, it only exposes the gas price for ContractCall, ContractCreate, and EthereumTransaction types in tinybars.GET /api/v1/network/fees{
"fees": [
{
"gas": 35561,
"transaction_type": "ContractCall"
},
{
"gas": 481934,
"transaction_type": "ContractCreate"
},
{
"gas": 35561,
"transaction_type": "EthereumTransaction"
}
],
"timestamp": "1633392000.387357562"
}
Since mirror node's inception in 2019, it has used Java 11 to build and run due to it being the most recent LTS release. After Java 17 LTS was released in September 2021 we knew we wanted to upgrade. With this release we upgraded to 17 after validating that the mirror node was still functional and performant. If you're using our official container images, they are also on Java 17 so there will be no migration necessary besides updating the image. If you're running outside of a container, you'll need to either upgrade your JRE to 17 or rebuild the jar from source with
-Djava.version=11.MAINNET UPDATE COMPLETED: MAY 25, 2022
TESTNET UPDATE COMPLETED: MAY 25, 2022
This release is focused on adding the necessary data and APIs needed for the JSON-RPC Relay defined in HIP-482. The JSON-RPC Relay implements the Ethereum JSON-RPC standard and relays HIP-410 Ethereum transactions to consensus nodes. Since the concept of a block is crucial for JSON-RPC APIs, this release also contains the implementation of HIP-415 Introduction of Blocks.
The mirror node now exposes the concept of blocks as introduced in HIP-415. We now calculate and store the cumulative gas used and the contract log bloom filter for the block as a whole. This HIP defines three new REST APIs and this release includes all three: a list blocks REST API, a get blocks REST API, and a list contract results REST API. The new
/api/v1/blocks API supports the usual limit and order query parameters along with timestamp and block.number to support equality and range operators for consensus timestamps and block numbers, respectively. The /api/v1/blocks/{hashOrNumber} is identical to the list blocks but only returns a single block by either its block hash or its block number. Finally, a /api/v1/contracts/results REST API was added that is identical to the existing /api/v1/contracts/{id}/results but able to search across contracts.GET /api/v1/blocks{
"blocks": [{
"count": 4,
"gas_limit": 150000000,
"gas_used": 50000000,
"hapi_version": "0.24.0",
"hash": "0xa4ef824cd63a325586bfe1a66396424cd33499f895db2ce2292996e2fc5667a69d83a48f3883f2acab0edfb6bfeb23c4",
"logs_bloom": "0x549358c4c2e573e02410ef7b5a5ffa5f36dd7398",
"name": "2022-04-07T16_59_23.159846673Z.rcd",
"number": 19533336,
"previous_hash": "0x4fbcefec4d07c60364ac42286d5dd989bc09c57acc7370b46fa8860de4b8721e63a5ed46addf1564e4f8cd7b956a5afa",
"size": 8489,
"timestamp": {
"from": "1649350763.159846673",
"to": "1649350763.382130000"
}
}],
"links": {
"next": null
}
}
GET /api/v1/blocks/{hashOrNumber}{
"count": 4,
"gas_limit": 150000000,
"gas_used": 50000000,
"hapi_version": "0.24.0",
"hash": "0xa4ef824cd63a325586bfe1a66396424cd33499f895db2ce2292996e2fc5667a69d83a48f3883f2acab0edfb6bfeb23c4",
"logs_bloom": "0x549358c4c2e573e02410ef7b5a5ffa5f36dd7398",
"name": "2022-04-07T16_59_23.159846673Z.rcd",
"number": 19533336,
"previous_hash": "0x4fbcefec4d07c60364ac42286d5dd989bc09c57acc7370b46fa8860de4b8721e63a5ed46addf1564e4f8cd7b956a5afa",
"size": 8489,
"timestamp": {
"from": "1649350763.159846673"
"to": "1649350763.382130000"
}
}
A number of changes were made in support of HIP-410 Ethereum Transactions. The
/api/v1/accounts/{idOrAlias} REST API was updated to accept an EVM address as a path parameter in lieu of an ID or alias. An ethereum_nonce and evm_address was added to the response of /api/v1/accounts/{idOrAliasOrAddress} and /api/v1/accounts. The existing /api/v1/contracts/results/{transactionId} was updated to accept the 32 byte Ethereum transaction hash as a path parameter in addition to the transaction ID that it supports now. Its response, as well as the similar /api/v1/contracts/{idOrAddress}/results/{timestamp}, was updated to add the following new Ethereum transaction fields:{
"access_list": "0xabcd...",
"block_gas_used": 564684,
"chain_id": "0x0127",
"gas_price": "0xabcd...",
"max_fee_per_gas": "0xabcd...",
"max_priority_fee_per_gas": "0xabcd...",
"nonce": 1,
"r": "0x84f0...",
"s": "0x5e03...",
"transaction_index": 1,
"type": 2,
"v": 0
}
Note: Existing fields omitted for brevity.
A new exchange rate REST API
/api/v1/network/exchangerate was added that returns the exchange rate network file stored in 0.0.112. It supports a timestamp parameter to retrieve the exchange rate at a certain time in the past.{
"current_rate": {
"cent_equivalent": 596987
"expiration_time": 1649689200
"hbar_equivalent": 30000
},
"next_rate": {
"cent_equivalent": 594920
"expiration_time": 1649692800
"hbar_equivalent": 30000
},
"timestamp": "1649689200.123456789"
}
A new
/api/v1/contracts/results/logs API was added with the same query parameters and response as /api/v1/contracts/{address}/results/logs but with the ability to search across contracts. It does not support address as a query parameter as it’s expected users use the existing API if they need logs for a specific address. The same rules around not exceeding maxTimestampRange still applies and allows it to stay performant. Pagination is possible using a combination of the timestamp and index query parameters.Finally, this releases completes our implementation of HIP-423 Long Term Scheduled Transactions. Two new fields
wait_for_expiry and expiration_time were added to /api/v1/schedules and /api/v1/schedules/{id}MAINNET UPDATE COMPLETED: MAY 18, 2022
TESTNET UPDATE COMPLETED: MAY 17, 2022
This is a big release with support for six different Hedera Improvement Proposals. Most of these changes are on the ingest side of things and future releases will work on adding them to our APIs.
HIP-16 will enable contract expiry and brings a new auto renew account field that contains the account responsible for any renewal fees associated with the contract. The contract REST APIs now show this
auto_renew_account field along with a new permanent_removal flag that is set to true when the system expires a contract.We missed a requirement in our implementation of HIP-329 CREATE2 opcode to allow a HAPI client to do a native transfer of assets to a contract known only by its CREATE2 address. We fixed this logic gap and now support EVM addresses in a
CryptoTransferTransaction.HIP-336 saw further polish to our allowance support. Support for an optional spender ID parameter on our
/api/v1/accounts/{id}/nfts?spender.id={id} REST API is now included.HIP-410 brings with it the ability to wrap an Ethereum native transaction and submit it to Hedera. The mirror node can now parse this new
EthereumTransaction along with the results from its execution. Any contracts created or results and logs generated by its execution will automatically show up on the relevant contract REST APIs. Support for specifying the contract initcode directly on a contract create was also added. To support Externally Owned Accounts (EOA) being able to submit Ethereum transactions, we now calculate and store the EVM address of the account's ECDSA secp256k1 alias. Finally, we added support for repeated topics in contract logs REST API to bring it more inline with the eth_getLogs JSON-RPC method. If you supply multiple different topic parameters (e.g. topic0 and topic1) it is considered an AND operation as before, but if pass repeated parameters like topic0=0x01&topic0=0x02&topic1=0x03 it means “(topic 0 is 0x01 or 0x02) and (topic 1 is 0x03)”.HIP-415 is defining a block as the number of records files since stream start (AKA genesis). Since only mirror nodes have a full history, they will be used to provide consensus nodes the current block number to update their state. Since partial mirror nodes with an effective start date after stream start won't have all record files, they may contain an inconsistent value for their block number in contrast to other mirror nodes with all data. This release attempts to correct that with a migration to bring them inline with full mirror nodes so everyone has a consistent block number value.
HIP-423 Long term scheduled transactions enhances the existing scheduled transactions to allow time-based scheduling of transactions. This release adds ingest support for the new schedule-related fields. Next release will expose these fields via our existing schedule REST APIs.
As part of this release, we have finished upgrading our PostgreSQL databases to version 14 and have updated all tests to use that version as well. We recommend mirror node operators plan their migration to PostgreSQL 14 at their earliest convenience. More details on the upgrade process can be found in our database guide.
The migrations in this release are estimated to take up to 2 hours against a mainnet database. However, the migration that takes up the bulk of this time will only run if it needs to correct block numbers. This is only necessary if your mirror node is a partial mirror node and has an effective start date that occurs after the stream start.
MAINNET UPDATE COMPLETED: MAY 4, 2022
This release is mainly focused on finishing out our support for HIP-336 Approval and Allowance API for Tokens. We added support for the new
CryptoDeleteAllowance transaction and removed support for the CryptoAdjustAllowance transaction that didn't make it into the final design. NFT allowances are tracked at the NFT transfer granularity allowing for up to date allowance information on the mirror node. Current spender information will show up in both /api/v1/accounts/{id}/nfts and /api/v1/tokens/{id}/nfts REST APIs. We also added the is_approval flag to APIs that show transfers.With more developers using computers using Apple's M-series CPUs, it become clear the mirror node needed to support ARM-based architectures to accommodate them. In this release we added multi-architecture Docker images using docker buildx. We now push
linux/amd64 and linux/arm64 variants to our Google Container Registry. If there's a need for additional operating systems or architectures in the future it can easily be expanded upon.MAINNET UPDATE COMPLETED: APRIL 25, 2022
TESTNET UPDATE COMPLETED: APRIL 19, 2022
This release adds support for three new REST APIs and four HIPs.
HIP-21 describes the need for a free network info query to enable SDKs and other clients to be able to retrieve the current list of nodes. In v0.49.1, we added a new
NetworkService.getNodes() gRPC API. In this release, we're adding an equivalent address book API to our REST API. In addition to the standard order and limit parameters, it supports a file.id query parameter to filter by the two address books 0.0.101 or 0.0.102 and a node.id query parameter to filter nodes and provide pagination.GET /api/v1/network/nodes{
"nodes": [
{
"description": "",
"file_id": "0.0.102",
"memo": "0.0.3",
"node_account_id": "0.0.3",
"node_cert_hash": "0x3334...",
"node_id": 0,
"public_key": "0x308201...",
"service_endpoints": [
{
"ip_address_v4": "13.124.142.126",
"port": 50211
}
],
"timestamp": {
"from": "1636052707.740848001",
"to": null
}
}
],
"links": {
"next": null
}
}
HIP-336 describes new Hedera APIs to approve and exercise allowances to a delegate account. An allowance grants a spender the right to transfer a predetermined amount of the payer's hbars or tokens to another account of the spender's choice. In v0.50.0 we added database support to store the new allowance transactions. In this release, two new REST APIs were created to expose the hbar and fungible token allowances. Full allowance support won't be available until a future release when consensus nodes enable it on mainnet.
GET /api/v1/accounts/{accountId}/allowances/crypto{
"allowances": [
{
"amount_granted": 10,
"owner": "0.0.1000",
"spender": "0.0.8488",
"timestamp": {
"from": "1633466229.96874612",
"to": "1633466568.31556926"
}
},
{
"amount_granted": 5,
"owner": "0.0.1000",
"spender": "0.0.9857",
"timestamp": {
"from": "1633466229.96874612",
"to": null
}
}
],
"links": {}
}
GET /api/v1/accounts/{accountId}/allowances/tokens{
"allowances": [
{
"amount_granted": 10,
"owner": "0.0.1000",
"spender": "0.0.8488",
"token_id": "0.0.1032",
"timestamp": {
"from": "1633466229.96874612",
"to": "1633466568.31556926"
}
},
{
"amount_granted": 5,
"owner": "0.0.1000",
"spender": "0.0.9857",
"token_id": "0.0.1032",
"timestamp": {
"from": "1633466229.96874612",
"to": null
}
}
],
"links": {}
}
Also on the REST API, we added support for HIP-329 CREATE2 addresses. Now any API that accepts a contract ID will also accept the 20-byte EVM address as a hex-encoded string. We improved the performance of the REST API by adding cache control headers to enable distributed caching via a CDN. The performance of the list transactions by type REST API saw a fix to improve its performance.
As part of HIP-260, contract precompile call data now populates new fields
amount, gas, and function_parameter inside ContractFunctionResult within the TransactionRecord. Mirror node now stores these fields and exposes them via its existing contract results REST APIs.There were a number of security improvements made to containerized mirror nodes. All Docker images now run as non-root regardless of running in Kubernetes or Docker Compose. The helm charts saw changes to conform to the Kubernetes restricted pod security standard. This ensures the mirror node runs with security best practices and reduces its overall attack surface. The Kubernetes Pod Security Standard replaces the deprecated PodSecurityPolicy and as such we've removed all configuration related to the latter.
This release has a long migration that is expected to take around 75 minutes to complete, depending upon your database hardware and configuration. As always, we recommend a red/black deployment to eliminate downtime during migrations. If you're using the
hedera-mirror-common chart, please check the kube-prometheus-stack upgrade notes to ensure Prometheus Operator can update successfully.MAINNET UPDATE COMPLETED: APRIL 7, 2022
TESTNET UPDATE COMPLETED: MARCH 29, 2022
This release is mainly focused around the area of data integrity and ensuring the data in the mirror node is consistent with consensus nodes. To this end, we added an errata database migration that only runs for mainnet and corrects three known issues that impacted the stream files. The state of the consensus nodes was never impacted, only the externalization of these changes to the stream files that the mirror node consumes.
To find the inconsistent data and ensure it stays consistent going forward, we added a new balance reconciliation job. This job runs nightly and compares the balance file information against the record file information to ensure they are in sync. It does three checks for each balance file: verifies the balance files add up to 50 billion hbars, verifies the aggregated hbar transfers match the balance file, and verifies the aggregated token transfers match the balance file. It can be disabled if not needed via
hedera.mirror.importer.reconciliation.enabled=false.We also fixed a bug that caused transfers with a zero amount to show up for crypto create transactions with a zero initial balance. This was due entirely to our code inserting the extra transfers, not because of any problem in the stream files. We also fixed an REST API bug that caused the contract byte code to show up as double encoded to hex.
For the Rosetta API, we added account alias support to various endpoints. And we now support parsing contract results for precompiled contract functions like HTS functions. This capability is disabled by a feature flag and will be enabled in a future release.
This release contains a couple medium sized database migrations to correct the erroneous data in the database. It is expected to take about 45 minutes against a full mainnet database.
MAINNET UPDATE COMPLETED: MARCH 18, 2022
TESTNET UPDATE COMPLETED: MARCH 15, 2022
HIP-331 is a community contributed improvement proposal requesting the addition of a new REST API to retrieve an account's list of owned non-fungible tokens (NFTs). The mirror node has an existing
/api/v1/tokens/{tokenId}/nfts API to retrieve all NFTs for a given token, but it didn't satisfy the requirement to show NFTs across token classes. This release adds the new /api/v1/accounts/{accountId}/nfts API to satisfy this need. It is our first API with multiple query parameters required for paging and as such has a few restrictions around their use. Please see the OpenAPI description for this API for further details.GET /api/v1/accounts/0.0.1001/nfts?token.id=gte:1500&serialnumber=gte:2&order=asc&limit=2 {
"nfts": [
{
"account_id": "0.0.1001",
"created_timestamp": "1234567890.000000006",
"deleted": false,
"metadata": "bTI=",
"modified_timestamp": "1234567890.000000006",
"serial_number": 2,
"token_id": "0.0.1500"
},
{
"account_id": "0.0.1001",
"created_timestamp": "1234567890.000000008",
"deleted": false,
"metadata": "bTM=",
"modified_timestamp": "1234567890.000000008",
"serial_number": 3,
"token_id": "0.0.1500"
}
],
"links": {
"next": "/api/v1/accounts/0.0.1001/nfts?order=asc&limit=2&token.id=gte:0.0.1500&serialnumber=gt:3"
}
}
The mirror node now has performance tests written using k6 for all of our APIs. These tests can be used to verify the performance doesn't regress from release to release. In the future, we plan to integrate these into a nightly regression test suite to improve our current approach of testing each release.
A number of deployment issues were addressed in this release. We now disable leader election by default until we can fix the issues with its implementation. Likewise we changed the importer Kubernetes deployment strategy from a rolling update to recreate to avoid ever having multiple importer pods running concurrently. A migration readiness probe was added to the importer. This will mark importer pods as unready until it completes all database migrations. Doing this will ensure Helm doesn't finish its release and run its tests before the migrations are completed.
We continue to fine tune our Rosetta implementation with a number of performance improvements and bug fixes. The performance of the Rosetta get genesis balance script was improved to reduce initial startup time. The embedded PostgreSQL container was upgraded to PostgreSQL 14. The Rosetta unified Docker image was updated to comply with the Rosetta persistence requirements.
MAINNET UPDATE COMPLETED: FEBRUARY 28, 2022
TESTNET UPDATE COMPLETED: FEBRUARY 25, 2022
This is a smaller release focusing on observability improvements and Rosetta API fixes.
On the observability front, we've reduced the volume of log information the REST API produces in half. We also change the REST API to generate a consistent trace log for all responses that includes accurate client IPs, the elapsed time, and a status code. We reduced the number of time series by about 50% that the mirror node produces to reduce monitoring costs.
For the Rosetta API, we added a workaround for the missing disappearing token transfer issue that allows the check data reconciliation to pass. Overall reconciliation time was improved by tweaking configuration parameters and improving NFT balance tracking performance. We worked around slow genesis account balance file loading by Rosetta CLI by switching to a dynamic account balance loading approach. A number of other Rosetta issues were also addressed.
MAINNET UPDATE COMPLETED: FEBRUARY 23, 2022
TESTNET UPDATE COMPLETED: FEBRUARY 15, 2022
This release adds support for three new improvement proposals: HIP-260 Smart Contract Traceability, HIP-329 CREATE2 Opcode, and HIP-336 Allowance APIs. It also updates the REST API to reflect the latest phases of HIP-226 and HIP-227 and updates the Rosetta API for HIP-31.
HIP-260 describes a need for improving the traceability of smart contracts by providing a verifiable trail of contract state changes in the transaction record. The mirror node can now store these state changes and expose them via the contract results REST API to show the values read and written for each slot. This information was added to both
/api/v1/contracts/results/{transactionId} and /api/v1/contracts/{id}/results/{timestamp}. Below is an example, with other fields omitted for brevity:{
"state_changes": [{
"address": "0x0000000000000000000000000000000000001f41",
"contract_id": "0.0.8001",
"slot": "0x0000000000000000000000000000000000000000000000000000000000000002",
"value_read": "0xaf846d22986843e3d25981b94ce181adc556b334ccfdd8225762d7f709841df0",
"value_written": "0x000000000000000000000000000000000000000000c2a8c408d0e29d623347c5"
}, {
"address": "0x0000000000000000000000000000000000001f42",
"contract_id": "0.0.8002",
"slot": "0xe1b094dec1b7d360498fa8130bf1944104b7b5d8a48f9ca88c3fc0f96c2d7225",
"value_read": "0x000000000000000000000000000000000000000000000001eafa3aaed1d27246",
"value_written": null
}]
}
HIP-329 adds support for EIP-1014 generated contract addresses via the CREATE2 opcode. As part of this, a new
evm_address is added to the transaction record that will be present for contract create transactions. Additionally, this evm_address can be populated in any ContractID that appears in the transaction body. The mirror node was updated to be able to map this evm_address to its corresponding contract number and to expose this property on the contracts REST API. We also store full contract information for child contracts since they now appear as separate internal transactions in the record stream, filling a long-standing gap in missing smart contract data.HIP-336 allowance functionality allows an account owner to delegate another account to spend hbars or tokens on his or her behalf. This feature provides an implementation of ERC20, IERC20, and ERC721 on the Hedera network. The mirror node was updated to support these new transaction types and store the absolute or relative crypto, fungible or non-fungible allowances. In a later release we will expose this information via a REST API as detailed in the design document.
Multiple new fields were added to the contract REST APIs as outlined in HIP-226 and HIP-227. The fields
bloom, result, and status were added to the contract results API response. result and status show similar information with the former being the HAPI response enum while the latter returning 0x1 or 0x0 to show if the transaction was successful or not, as is common in web3 APIs. We also added bloom to the contract logs API response. Finally, we now return a partial response for contract calls without a result.The importer component added a new
hedera.mirror.importer.parser.record.entity.persist.topics property to control the persistence of topic messages. This can be set to false for mirror node operators if topic message data is not being used. On mainnet alone, this data currently takes up to 2TB worth of storage.The Monitor component gained support for parallel node validation to improve startup performance. Now all validation is done in a background thread, adding and removing nodes as necessary while the publisher thread continues publishing transactions without any interruptions. This re-work also fixed issues with subscription halting during node validation and taking too long to validate a down node.
Rosetta saw a few important improvements including adding support for HIP-31 expected token decimals. The Rosetta unified Docker image saw functionality added to automatically restore the database using a database snapshot on initial startup.
As part of HIP-329 CREATE2, we renamed the existing
solidity_address in the contract REST API to evm_address. This new name accurately reflects the naming in the HIP and protobuf and avoids tying the address to Solidity when Hedera supports more than just Solidity contracts.MAINNET UPDATE COMPLETED: FEBRUARY 15, 2022
TESTNET UPDATE COMPLETED: FEBRUARY 4, 2022
This release implements three Hedera Improvement Proposals (HIPs) and upgrades the mirror node database to the latest version.
HIP-21 describes the need for a free network info query to enable SDKs and other clients to be able to retrieve the current list of nodes. To satisfy this need we added a new
NetworkService.getNodes() streaming gRPC API to get the list of current nodes from the address book network file. By making it a streaming API we avoid the client having to handle paging themselves, while still allowing us to split the large address book into smaller chunks. Since there are two address book files, we provide an option to choose which FileID to return.message AddressBookQuery {
.proto.FileID file_id = 1; // The ID of the address book file on the network. Can be either 0.0.101 or 0.0.102.
int32 limit = 2; // The maximum number of node addresses to receive before stopping. If not set or set to zero it will return all node addresses in the database.
}
service NetworkService {
rpc getNodes (AddressBookQuery) returns (stream .proto.NodeAddress);
}
HIP-171 describes the need for returning the payer account in the topic message REST API response. This release does just that while also adding in the topic message chunk information that was present in the gRPC API but missing from the REST API.
{
+ "chunk_info": {
+ "initial_transaction_id": {
+ "account_id": "0.0.1000",
+ "nonce": 0,
+ "scheduled": false,
+ "transaction_valid_start": "1234567890-000000006"
+ },
+ "number": 2,
+ "total": 5
+ },
"consensus_timestamp": "1234567890.000000001",
"topic_id": "0.0.7",
"message": "bWVzc2FnZQ==",
+ "payer_account_id": "0.0.1000",
"running_hash": "cnVubmluZ19oYXNo",
"running_hash_version": 2,
"sequence_number": 1
}
Continuing our support for HIP-32 auto account creation, we added alias support to our accounts REST API. Now when you query
/v1/api/accounts/:id the id can be either a Hedera entity in the 0.0.x form or a hex-encoded alias. An account's alias will now also show up in all of the accounts REST API output.A lot of testing was done to ensure that PostgreSQL 14 functions correctly with the mirror node and provides as good as or better performance to older versions. We are now in the process of migrating our Hedera managed mirror nodes to PostgreSQL 14. We recommend other mirror node operators consider upgrading to the latest database version at their earliest convenience and have provided upgrade instructions to aid in that process.
MAINNET UPDATE COMPLETED: JANUARY 26, 2022
TESTNET UPDATE COMPLETED: JANUARY 18, 2022
HIP-206 adds a parent consensus timestamp to the transaction record for internal transactions that occur like HTS precompiles invoked from a smart contract. To round out the
nonce support in the last release we added parent_consensus_timestamp to /api/v1/accounts/{id} and /api/v1/transactions. This field helps define the parent/child relationships between transactions.HIP-226 describes the recently added contract results REST API. Each release we've been iterating and adding more functionality to the API until it matches the description in the HIP. This release adds the list of logs generated by a smart contract execution. Here's a sample of the new JSON response:
{
"amount": 30,
...
"logs": [
{
"address": "0x0000000000000000000000000000000000001389",
"contract_id": "0.0.5001",
"data": "0x0123",
"index": 0,
"topics": [
"0x97c1fc0a6ed5551bc831571325e9bdb365d06803100dc20648640ba24ce69750",
"0x8c5be1e5ebec7d5bd14f71427d1e84f3dd0314c0f7b2291e5b200ac8c7c3b925",
"0xddf252ad1be2c89b69c2b068fc378daa952ba7f163c4a11628f55a4df523b3ef",
"0xe8d47b56e8cdfa95f871b19d4f50a857217c44a95502b0811a350fec1500dd67"
]
},
{
"address": "0x000000000000000000000000000000000000138a",
"contract_id": "0.0.5002",
"data": "0x0123",
"index": 1,
"topics": [
"0x97c1fc0a6ed5551bc831571325e9bdb365d06803100dc20648640ba24ce69750",
"0x8c5be1e5ebec7d5bd14f71427d1e84f3dd0314c0f7b2291e5b200ac8c7c3b925"
]
}
]
}
In the last release, we added a new Web3 JSON-RPC API. With this release, we've added a
hedera-mirror-web3 Helm chart that can be used to deploy it on Kubernetes. Additional metrics were added to the Java module and a new Grafana dashboard was created to visualize them. The Web3 API was also added to the docker-compose file.If you're upgrading an existing deployment of our Helm chart, there are a few breaking changes to keep in mind. First, we deploy the new Web3 API chart by default and it requires a
mirror_web3 database user exist with read permission. Please create the new database user before upgrading or you can disable the hedera-mirror-web3 sub-chart.We've upgraded the
PodDisruptionBudget resources from policy/v1beta1 to policy/v1 and as a result now require Kubernetes 1.21 or greater. For the hedera-mirror chart, if you're using the optional postgresql sub-chart you must scale the PostgreSQL replicas down to one before initiating an upgrade in order to upgrade repmgr.If you're using the
hedera-mirror-common chart, there are a number of breaking changes in the community sub-charts it uses. Before upgrading, you will need to delete the prometheus-adapter and kube-state-metrics deployments. You'll also need to reinstall a few custom resource definitions. Run the below commands to do so:kubectl delete deployment -l app.kubernetes.io/name=kube-state-metrics --cascade=orphan
kubectl delete deployment -l app=prometheus-adapter
kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.53.1/example/prometheus-operator-crd/monitoring.coreos.com_alertmanagerconfigs.yaml
kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.53.1/example/prometheus-operator-crd/monitoring.coreos.com_alertmanagers.yaml
kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.53.1/example/prometheus-operator-crd/monitoring.coreos.com_podmonitors.yaml
kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.53.1/example/prometheus-operator-crd/monitoring.coreos.com_probes.yaml
kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.53.1/example/prometheus-operator-crd/monitoring.coreos.com_prometheuses.yaml
kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.53.1/example/prometheus-operator-crd/monitoring.coreos.com_prometheusrules.yaml
kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.53.1/example/prometheus-operator-crd/monitoring.coreos.com_servicemonitors.yaml
kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.53.1/example/prometheus-operator-crd/monitoring.coreos.com_thanosrulers.yaml
MAINNET UPDATE COMPLETED: JANUARY 3, 2022
TESTNET UPDATE COMPLETED: DECEMBER 30, 2021
This release continues the focus on Smart Contracts 2.0. The mirror node is useful for debugging a smart contract execution and our focus has been on providing APIs to make developers' lives easier. To that end, we added support for transaction ID nonce, a new contract logs REST API, and a new web3 API component.
The new Web3 API module provides an implementation of existing JSON-RPC APIs for the Hedera network. JSON-RPC API is a widely used standard for interacting with distributed ledgers. The aim in providing a Hedera implementation of these APIs is to ease the migration of existing dApps to Hedera and simplify the developer on-ramp. Currently, the Web3 module only provides a partial implementation of the Ethereum JSON-RPC API. Specifically, only the
eth_blockNumber method has been implemented in this release as we focused on putting the groundwork in place first.As part of HIP-32 and HIP-206 a
nonce field was added to the TransactionID protobuf to guarantee uniqueness for platform generated transactions. This nonce field was added to any REST API that returns transaction data. A nonce query parameter was added to /api/v1/transactions/:transactionId, /api/v1/transactions/:transactionId/stateproof, and /api/v1/contracts/results/:transactionId to be able to distinguish between a user-submitted transaction and an internal transaction generated as a result of that transaction. Note that /api/v1/transactions/:transactionId without a nonce parameter will default to returning all transactions regardless of nonce while the other APIs will default nonce to 0.The new
/api/v1/contracts/{id}/results/logs REST API provides a search API to query for logs across contract executions for a particular contract. Searching by consensus timestamp and topics is supported. Note that for performance reasons it doesn't currently support pagination and requires a timestamp or timestamp range be provided to search by topic.