spark写相同的HIVE表或者相同的HDFS路径时抛出异常

L13763338360

已于 2022-08-23 17:39:40 修改

阅读量1.4k

点赞数 2

CC 4.0 BY-SA版权

分类专栏： spark 文章标签： spark hive hdfs

于 2022-08-23 11:11:22 首次发布

本文链接：https://blog.csdn.net/L13763338360/article/details/126480629

1. 写相同的hive表

代码示例


import os
import time
import logging
from pyspark.sql import SparkSession

spark = SparkSession \
            .builder \
            .appName("mytest") \
            .enableHiveSupport() \
            .getOrCreate()

df = spark.sql("select * from xxx.xxx")
print(df.show())
df.write.mode("overwrite").saveAsTable("xxx.xxx")

抛出异常

---------------------------------------------------------------------------
AnalysisException                         Traceback (most recent call last)
/tmp/ipykernel_36930/1113601968.py in <module>
     12 df = spark.sql("select * from dev_sztoc_crm.leon_test")
     13 print(df.show())
---> 14 df.write.mode("overwrite").saveAsTable("dev_sztoc_crm.leon_test")

/usr/share/spark3/python/pyspark/sql/readwriter.py in saveAsTable(self, name, format, mode, partitionBy, **options)
   1158         if format is not None:
   1159             self.format(format)
-> 1160