22 lines
403 B
Markdown
22 lines
403 B
Markdown
## 1. 问题
|
|
|
|
在hive中使用了过多的窗口函数导致了内存OOM
|
|
|
|
## 2. 原因
|
|
|
|
每一个窗口函数，hive都会在内存中执行一次窗口划分，消耗大量的内存，最终导致OOM；
|
|
|
|
## 3. 解决方法
|
|
|
|
使用window算子，复用窗口，如：
|
|
```sql
|
|
select
|
|
id,
|
|
ts,
|
|
v,
|
|
lead(v) over w as lead_v,
|
|
lag(v) over w as lag_v
|
|
from
|
|
test
|
|
WINDOW w as (partition by id order by ts)
|
|
``` |