01.Impala
components
impalad的部署
Impala daemons can be deployed in one of the following ways:
- HDFS and Impala are co-located, and each Impala daemon runs on the same host as a DataNode.
- Impala is deployed separately in a compute cluster and reads remotely from HDFS, S3, ADLS, etc.
hive连接表需要刷新impalad
The catalog service avoids the need to issue REFRESH and INVALIDATE METADATA statements when the metadata changes are performed by statements issued through Impala. When you create a table, load data, and so on through Hive, you do need to issue REFRESH or INVALIDATE METADATA on an Impala daemon before executing a query there.
impala 自己执行修改元数据的请求时，不需要刷新和重载元数据，当通过hive创建表和加载数据时，在执行查询之前，需要在Impala守护进程上发出REFRESH或INVALIDATE元数据。
The REFRESH and INVALIDATE METADATA statements are not needed when the CREATE TABLE, INSERT, or other table-changing or data-changing operation is performed through Impala. These statements are still needed if such operations are done through Hive or by manipulating data files directly in HDFS, but in those cases the statements only need to be issued on one Impala daemon rather than on all daemons.
当通过Impala执行创建表、插入或其他表更改或数据更改操作时，不需要刷新和无效元数据语句。
如果通过Hive或直接在HDFS中操作数据文件，仍然需要，但是在这种情况下，只需要在一个Impala守护进程上发出这些语句，而不是在所有守护进程上。
元数据加载与对查询的影响
‑‑load_catalog_in_background option to control when the metadata of a table is loaded.
If set to false, the metadata of a table is loaded when it is referenced for the first time. This means that the first run of a particular query can be slower than subsequent runs. Starting in Impala 2.2, the default for ‑‑load_catalog_in_background is false.
表的元数据在第一次引用时加载。这意味着特定查询的第一次运行可能比后续运行慢
If set to true, the catalog service attempts to load metadata for a table even if no query needed that metadata. So metadata will possibly be already loaded when the first query that would need it is run. However, for the following reasons, we recommend not to set the option to true.
catalogd尝试加载表的元数据，即使没有查询需要该元数据。因此，在运行第一个需要元数据的查询时，元数据可能已经被加载
Background load can interfere with query-specific metadata loading. This can happen on startup or after invalidating metadata, with a duration depending on the amount of metadata, and can lead to a seemingly random long running queries that are difficult to diagnose.
后台加载可能会干扰特定查询的元数据加载。这种情况可能在启动时发生，也可能在元数据失效后发生，持续时间取决于元数据的数量，并可能导致看似随机的长时间运行查询，而这些查询很难诊断。
Impala may load metadata for tables that are possibly never used, potentially increasing catalog size and consequently memory usage for both catalog service and Impala Daemon.
Impala可以为可能从未使用过的表加载元数据，这可能会增加目录大小，从而增加目录服务和Impala守护进程的内存使用量。
元数据刷新耗时
For tables with a large volume of data and/or many partitions, retrieving all the metadata for a table can be time-consuming, taking minutes in some cases. Thus, each Impala node caches all of this metadata to reuse for future queries against the same table.
对于具有大量数据和/或许多分区的表，检索表的所有元数据可能很耗时，在某些情况下会花费几分钟。
因此，每个Impala节点都会缓存所有这些元数据，以供将来针对同一表的查询重用。
If the table definition or the data in the table is updated, all other Impala daemons in the cluster must receive the latest metadata, replacing the obsolete cached metadata, before issuing a query against that table
For DDL and DML issued through Hive, or changes made manually to files in HDFS, you still use the REFRESH statement (when new data files are added to existing tables) or the INVALIDATE METADATA statement (for entirely new tables, or after dropping a table, performing an HDFS rebalance operation, or deleting data files). Issuing INVALIDATE METADATA by itself retrieves metadata for all the tables tracked by the metastore. If you know that only specific tables have been changed outside of Impala, you can issue REFRESH table_name for each affected table to only retrieve the latest metadata for those tables.