The best way to run Apache Hive in production
We develop an execution engine MR3 for big data processing and maintain it main application - Hive on MR3. Our execution engine MR3 provides native support for both Hadoop and Kubernetes.
| #1. You want to install Hive on Hadoop or upgrade an old version of Hive. |
Installing Hive on Hadoop or upgrading to a higher version of Hive is not a simple task. It usually requires upgrades of a few important dependencies of Hive and may even require an upgrade of Hadoop itself. In contrast, Hive on MR3 is very easy to install and requires no change to an existing installation of Hadoop (such as HDP and CDH). |
| #2. You want to run Hive directly on Kubernetes. |
As the enterprise environment gravitates towards Kubernetes at an accelerating pace, the industry is looking for a solution that enables Hive to run directly on Kubernetes. Unfortunately only an expedient solution exists today which first operates Hadoop on Kubernetes and then runs Hive on Hadoop. For this problem, Hive on MR3 is a perfect solution ready for you. |
| #3. You want to run Hive without installing Kubernetes or Hadoop. |
MR3 supports standalone mode which does not require a resource manager such as Hadoop and Kubernetes. By exploiting standalone mode, you can run Hive virtually in any type of cluster. Installing Hive on MR3 in standalone mode is also very easy. |
| #1. Hive on MR3 is stable with 700+ security and critical patches backported. |
Anyone who manually builds Apache Hive 3 (such as version 3.1.3) soon discovers that it is not really ready for production use because many important patches have not been merged into the source repository. We have backported over 700 important patches to Hive on MR3 and keep backporting more patches. |
| #2. Hive on MR3 achieves the speed of LLAP and runs as fast as Trino. |
LLAP (Low-Latency Analytical Processing) is a major component of Hive which allows it to far outperform competing technologies. Enabling LLAP, however, is excruciatingly difficult because of its complex architecture. Hive on MR3 automatically achieves the speed of LLAP with no additional configuration. From our latest performance evaluation, Hive on MR3 (with fault tolerance) runs as fast as Trino (without fault tolerance). |
| #3. Hive on MR3 achieves a much higher throughput than Hive on Tez. |
A common use case of Hive on Tez is to run ETL (Extract-Transform-Load) jobs. By virtue of its advanced resource sharing model, Hive on MR3 can deliver significant cost savings, especially if many ETL jobs are run concurrently. |
| #4. Hive on MR3 supports Remote Shuffle Service. |
Remote Shuffle Service is being adopted by a growing number of technologies because of its numerous potentials. Hive on MR3 is also evolving fast to support Remote Shuffle Service. Currently Hive on MR3 supports Apache Celeborn as Remote Shuffle Service and can eliminate over 95% of local disk writes. |
| #5. No vendor lock-in. |
Running Hive on MR3 means that there is no risk of vendor lock-in. Since Hive on MR3 runs with Hive Metastore, the user can switch back to Apache Hive or an alternative technology at any time. |
| Intrusion, USA |
Intrusion, a leader in cybersecurity, has adopted Hive on MR3 on Kubernetes for its data warehouse. Previously Intrusion was running HDP (Hortonworks Data Platform). All the existing code, including SQL queries, Java UDFs, and Python transform scripts, has been successfully migrated. |
| Shuyun, China |
Shuyun, a provider of data services for marketing enterprises, uses Hive on MR3 on Hadoop to power its data warehouse of 1 petabyte. After switching from HDP (Hortonworks Data Platform) to Hive on MR3, the data analysis department has seen a significant increase in productivity. |
| Grain Whale, China |
Grain Whale, a Chinese game publishing company, switched from HDP (Hortonworks Data Platform) to Hive on MR3 to achieve a significant improvement in analytical capabilities on the same hardware resources. |
If you are interested in our solution, you can try it yourself or request a demo. For any question about our solution and licenses, please contact us.