1. 从零掌握Presto/Trino集合运算第一次接触Presto/Trino的集合运算时我完全被UNION、INTERSECT、EXCEPT这些操作符搞晕了。直到在电商用户行为分析项目中踩过几次坑后才发现它们其实是处理数据集的瑞士军刀。想象你手上有两份销售数据线上商城和线下门店UNION就像把两个Excel表格上下拼接INTERSECT则是找出两个渠道都购买过的VIP客户而EXCEPT能筛选出仅在线下消费的银发族群体。UNION ALL是最直接的合并方式它保留所有记录包括重复项。记得去年双十一大促时我们需要合并来自MySQL和Hive的订单数据当时用下面这段代码快速生成了总销售报表-- 合并两个数据源的订单记录保留重复 SELECT order_id, customer_id, amount FROM mysql.orders_online UNION ALL SELECT order_id, customer_id, amount FROM hive.orders_offline但要注意性能陷阱当处理千万级数据时UNION ALL会比UNION快3-5倍因为后者需要额外去重。有次我忘记这个区别导致周报生成时间从2分钟暴增到15分钟。INTERSECT的实战价值在于用户画像交叉分析。比如要找出同时满足月消费1万和最近登录7天的高价值用户-- 交叉分析高净值用户 SELECT user_id FROM dw.user_consumption WHERE monthly_spend 10000 INTERSECT SELECT user_id FROM dw.user_activity WHERE last_login_date CURRENT_DATE - INTERVAL 7 DAYEXCEPT特别适合做数据清洗。去年做RFM模型时我用它排除测试账号的影响-- 排除测试账号后的有效用户 SELECT user_id FROM production.users EXCEPT SELECT user_id FROM test.test_accounts实际工作中集合运算有三大黄金法则所有SELECT语句的列数和类型必须严格匹配大数据量操作时优先考虑分区字段过滤混合使用ALL/DISTINCT时要评估性能损耗2. 多维分析的秘密武器GROUPING SETS家族在零售行业做经营分析时最头疼的就是要同时出不同维度的汇总报表。直到我发现GROUPING SETS这套组合拳原来需要写5个SQL的报表现在1个就能搞定。比如分析全国连锁店的销售数据时-- 多维度销售分析 SELECT region, city, store_type, SUM(sales) AS total_sales, GROUPING(region, city, store_type) AS group_id FROM sales_data GROUP BY GROUPING SETS ( (region, city, store_type), -- 门店粒度 (region, store_type), -- 区域类型 (region), -- 大区汇总 () -- 全国总计 ) ORDER BY group_id;ROLLUP的层次化聚合简直是制作年报的利器。它自动生成从细到粗的所有分组组合比如时间维度从年月日一直汇总到全年-- 时间维度层级汇总 SELECT EXTRACT(YEAR FROM order_date) AS year, EXTRACT(MONTH FROM order_date) AS month, EXTRACT(DAY FROM order_date) AS day, SUM(amount) AS daily_sales FROM orders GROUP BY ROLLUP( EXTRACT(YEAR FROM order_date), EXTRACT(MONTH FROM order_date), EXTRACT(DAY FROM order_date) )CUBE更适合做探索性分析。有次做商品关联分析用CUBE发现了意想不到的品类组合-- 商品品类全组合分析 SELECT category1, category2, payment_method, COUNT(DISTINCT order_id) AS order_count FROM order_details GROUP BY CUBE(category1, category2, payment_method)GROUPING函数是理解这些结果的钥匙。它返回的二进制掩码能准确告诉我们当前行是哪个分组组合的汇总结果。这里有个实用技巧在BI工具中可以用CASE语句转换这些数字为可读标签SELECT CASE GROUPING(region) WHEN 1 THEN ALL ELSE region END AS region_label -- 其他列... FROM sales_data GROUP BY ROLLUP(region)3. WITH子句SQL的模块化编程重构复杂SQL就像整理一团乱麻直到我学会WITH子句这种乐高式编程。去年做用户生命周期分析时原本300行的嵌套查询被拆解成清晰的模块WITH -- 第一步识别新用户 new_users AS ( SELECT user_id, MIN(order_date) AS first_order_date FROM orders GROUP BY user_id ), -- 第二步计算复购行为 repeat_purchases AS ( SELECT user_id, COUNT(DISTINCT order_id) AS order_count FROM orders WHERE order_date (SELECT first_order_date FROM new_users nu WHERE nu.user_id orders.user_id) GROUP BY user_id ), -- 第三步关联用户属性 user_segments AS ( SELECT u.user_id, CASE WHEN r.order_count 5 THEN 高价值 WHEN r.order_count 1 THEN 潜力 ELSE 流失风险 END AS segment FROM new_users u LEFT JOIN repeat_purchases r ON u.user_id r.user_id ) -- 最终输出 SELECT segment, COUNT(*) AS user_count FROM user_segments GROUP BY segment;WITH RECURSIVE更是处理层级数据的核武器。处理组织架构数据时用它查询部门层级关系比写存储过程优雅多了-- 递归查询部门树 WITH RECURSIVE org_tree AS ( -- 基础查询获取根部门 SELECT dept_id, dept_name, parent_id, 1 AS level FROM department WHERE parent_id IS NULL UNION ALL -- 递归查询关联子部门 SELECT d.dept_id, d.dept_name, d.parent_id, t.level 1 FROM department d JOIN org_tree t ON d.parent_id t.dept_id ) SELECT * FROM org_tree ORDER BY level, dept_id;性能优化方面有个血的教训WITH子句虽然是临时视图但Presto/Trino不保证只执行一次。有次我误以为WITH子句会被缓存导致一个10亿级表被扫描了三次。正确的做法是对大表查询先用CREATE TABLE AS物化中间结果。4. 窗口函数让数据分析飞起来第一次用窗口函数分析用户购买路径时我仿佛打开了新世界的大门。原来需要Java代码实现的复杂分析现在几句SQL就能搞定。比如计算每个用户的消费累计占比-- 用户消费累计分析 SELECT user_id, order_date, amount, SUM(amount) OVER (PARTITION BY user_id ORDER BY order_date) AS running_total, ROUND(amount * 100.0 / SUM(amount) OVER (PARTITION BY user_id), 2) AS percent_of_total FROM orders WHERE user_id IN (1001, 1002, 1003);LAG/LEAD这对兄弟函数是时间序列分析的标配。去年做零售库存预警时用它们实现了自动化的周环比分析-- 销售周环比分析 WITH weekly_sales AS ( SELECT product_id, DATE_TRUNC(week, sale_date) AS week_start, SUM(quantity) AS weekly_quantity FROM sales GROUP BY 1, 2 ) SELECT product_id, week_start, weekly_quantity, LAG(weekly_quantity, 1) OVER (PARTITION BY product_id ORDER BY week_start) AS prev_week_quantity, ROUND((weekly_quantity - LAG(weekly_quantity, 1) OVER (PARTITION BY product_id ORDER BY week_start)) * 100.0 / NULLIF(LAG(weekly_quantity, 1) OVER (PARTITION BY product_id ORDER BY week_start), 0), 2) AS week_over_week_pct FROM weekly_sales ORDER BY product_id, week_start;窗口帧的灵活定义是高级分析的杀手锏。做移动平均分析时发现三种帧类型各有妙用-- 三种移动平均计算方式 SELECT date, sales, -- 固定窗口最近7天 AVG(sales) OVER (ORDER BY date ROWS BETWEEN 6 PRECEDING AND CURRENT ROW) AS ma_7days, -- 动态窗口当月累计 AVG(sales) OVER (PARTITION BY EXTRACT(YEAR_MONTH FROM date) ORDER BY date ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) AS mtd_avg, -- 对称窗口前后3天 AVG(sales) OVER (ORDER BY date ROWS BETWEEN 3 PRECEDING AND 3 FOLLOWING) AS centered_ma FROM daily_sales;性能调优方面有个关键发现窗口函数的PARTITION BY子句应该尽量使用分区键。有次在十亿级用户表上添加正确的分区字段后查询从15分钟降到47秒。另外多个窗口函数尽量合并到同一个OVER子句中能减少数据扫描次数。