数据湖DeltaLake编译scalastyle检查错误Failing because of negative scalastyle result

delta编译报错很明显是scalastyle检查的问题,但是sbt不像maven有-skip可以不执行检查, 用scalastyle:on /scalastyle:off 太麻烦,只好把scalastyle-config.xml文件的内容全部清空

$ build/sbt compile
Using E:\dev_tools\jdk1.8.0_202\jdk as default JAVA_HOME.
Note, this will be overridden by -java-home if it is set.
Attempting to fetch sbt
Launching sbt from build/sbt-launch-0.13.18.jar
[info] Loading global plugins from C:\Users\Administrator\.sbt\0.13\plugins
[info] Loading project definition from E:\code\github\delta\project\project
[info] Loading project definition from E:\code\github\delta\project
Missing bintray credentials C:\Users\Administrator\.bintray\.credentials. Some intray features depend on this.
[info] Set current project to delta-core (in build file:/E:/code/github/delta/)
[info] Executing in batch mode. For better performance use sbt's shell
[info] scalastyle using config E:\code\github\delta\scalastyle-config.xml
[error] E:\code\github\delta\src\main\scala\org\apache\spark\sql\delta\DeltaTimTravelSpec.scala:59:6:
[error]       Are you sure that you want to throw Error? In most cases, you shold use appropriate Exception instead.
[error]       If you must throw Error, wrap the code block with
[error]       // scalastyle:off throwerror
[error]       throw new XXXError(...)
[error]       // scalastyle:on throwerror
[error]
[error] E:\code\github\delta\src\main\scala\org\apache\spark\sql\delta\schema\ShemaUtils.scala:179:22:
[error]       Are you sure that you want to use toUpperCase or toLowerCase withut the root locale? In most cases, you
[error]       should use toUpperCase(Locale.ROOT) or toLowerCase(Locale.ROOT) istead.
[error]       If you must use toUpperCase or toLowerCase without the root local, wrap the code block with
[error]       // scalastyle:off caselocale
[error]       .toUpperCase
[error]       .toLowerCase
[error]       // scalastyle:on caselocale
[error]
[error] E:\code\github\delta\src\main\scala\org\apache\spark\sql\delta\schema\ShemaUtils.scala:252:26:
[error]       Are you sure that you want to use toUpperCase or toLowerCase withut the root locale? In most cases, you
[error]       should use toUpperCase(Locale.ROOT) or toLowerCase(Locale.ROOT) istead.
[error]       If you must use toUpperCase or toLowerCase without the root local, wrap the code block with
[error]       // scalastyle:off caselocale
[error]       .toUpperCase
[error]       .toLowerCase
[error]       // scalastyle:on caselocale
[error]
[error] E:\code\github\delta\src\main\scala\org\apache\spark\sql\delta\schema\ShemaUtils.scala:916:98:
[error]       Are you sure that you want to use toUpperCase or toLowerCase withut the root locale? In most cases, you
[error]       should use toUpperCase(Locale.ROOT) or toLowerCase(Locale.ROOT) istead.
[error]       If you must use toUpperCase or toLowerCase without the root local, wrap the code block with
[error]       // scalastyle:off caselocale
[error]       .toUpperCase
[error]       .toLowerCase
[error]       // scalastyle:on caselocale
[error]
[error] E:\code\github\delta\src\main\scala\org\apache\spark\sql\delta\schema\ShemaUtils.scala:921:4:
[error]       Are you sure that you want to use toUpperCase or toLowerCase withut the root locale? In most cases, you
[error]       should use toUpperCase(Locale.ROOT) or toLowerCase(Locale.ROOT) istead.
[error]       If you must use toUpperCase or toLowerCase without the root local, wrap the code block with
[error]       // scalastyle:off caselocale
[error]       .toUpperCase
[error]       .toLowerCase
[error]       // scalastyle:on caselocale
[error]
[error] E:\code\github\delta\src\main\scala\org\apache\spark\sql\delta\Snapshotscala:305:58:
[error]       Are you sure that you want to throw Error? In most cases, you shold use appropriate Exception instead.
[error]       If you must throw Error, wrap the code block with
[error]       // scalastyle:off throwerror
[error]       throw new XXXError(...)
[error]       // scalastyle:on throwerror
[error]
[error] E:\code\github\delta\src\main\scala\org\apache\spark\sql\delta\util\FilNames.scala:100:22:
[error]       Are you sure that you want to throw Error? In most cases, you shold use appropriate Exception instead.
[error]       If you must throw Error, wrap the code block with
[error]       // scalastyle:off throwerror
[error]       throw new XXXError(...)
[error]       // scalastyle:on throwerror
[error]
[error] E:\code\github\delta\src\main\scala\org\apache\spark\sql\delta\util\ParitionUtils.scala:703:47:
[error]       Are you sure that you want to use toUpperCase or toLowerCase withut the root locale? In most cases, you
[error]       should use toUpperCase(Locale.ROOT) or toLowerCase(Locale.ROOT) istead.
[error]       If you must use toUpperCase or toLowerCase without the root local, wrap the code block with
[error]       // scalastyle:off caselocale
[error]       .toUpperCase
[error]       .toLowerCase
[error]       // scalastyle:on caselocale
[error]
[info] scalastyle Processed 95 file(s)
[info] scalastyle Found 8 errors
[info] scalastyle Found 0 warnings
[info] scalastyle Found 0 infos
[info] scalastyle Finished in 9 ms
[success] created output: E:\code\github\delta\target
java.lang.RuntimeException: Failing because of negative scalastyle result
        at scala.sys.package$.error(package.scala:27)
        at org.scalastyle.sbt.Tasks$.handleResult$1(Plugin.scala:132)
        at org.scalastyle.sbt.Tasks$.doScalastyleWithConfig$1(Plugin.scala:187)
        at org.scalastyle.sbt.Tasks$.doScalastyle(Plugin.scala:192)
        at org.scalastyle.sbt.ScalastylePlugin$$anonfun$rawScalastyleSettings$1$anonfun$apply$1$$anonfun$apply$2.apply(Plugin.scala:81)
        at org.scalastyle.sbt.ScalastylePlugin$$anonfun$rawScalastyleSettings$1$anonfun$apply$1$$anonfun$apply$2.apply(Plugin.scala:68)
        at scala.Function1$$anonfun$compose$1.apply(Function1.scala:47)
        at sbt.$tilde$greater$$anonfun$$u2219$1.apply(TypeFunctions.scala:40)
        at sbt.std.Transform$$anon$4.work(System.scala:63)
        at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:28)
        at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:28)
        at sbt.ErrorHandling$.wideConvert(ErrorHandling.scala:17)
        at sbt.Execute.work(Execute.scala:237)
        at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:228)
        at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:228)
        at sbt.ConcurrentRestrictions$$anon$4$$anonfun$1.apply(ConcurrentRestritions.scala:159)
        at sbt.CompletionService$$anon$2.call(CompletionService.scala:28)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:51)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutorjava:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecuto.java:624)
        at java.lang.Thread.run(Thread.java:748)
[error] (*:compileScalastyle) Failing because of negative scalastyle result
[error] Total time: 2 s, completed 2020-6-30 11:21:56

 

### 解决HDFS路径写入失败的方法 当遇到HDFS路径写入失败的情况时,可以从多个角度排查和解决问题。以下是详细的解决方案: #### 1. 权限设置不当 权限配置错误可能导致无法成功创建目录或写入文件。确保目标路径具有正确的所有权和权限设置[^1]。 ```bash sudo -u hdfs hdfs dfs -mkdir /ranger-test sudo -u hdfs hdfs dfs -chown example-user-1:example-group /ranger-test sudo -u hdfs hdfs dfs -chmod 700 /ranger-test ``` 这些命令用于创建测试目录`/ranger-test`,将其所有者设为`example-user-1`,并将权限设置为仅允许该用户访问。 #### 2. 数据节点状态异常 检查是否有数据节点处于不健康状态或者离线。可以通过NameNode Web UI查看集群的状态,并确认各个DataNode是否正常工作。如果有任何节点不可用,则需要进一步诊断其日志来找出原因并修复。 #### 3. 容量不足 如果磁盘空间已满,也会导致写操作失败。监控各节点上的可用存储资源,必要时清理不必要的文件或将部分数据迁移到其他位置释放空间。 #### 4. 配置参数调整 某些情况下,默认配置可能不适合当前的工作负载模式。考虑优化以下几项重要参数: - `dfs.replication`: 控制副本数量;适当减少可以在一定程度上缓解压力。 - `dfs.block.size`: 调整块大小有助于提高I/O效率。 修改完成后记得重启服务使更改生效。 #### 5. 日志分析 深入研究相关组件的日志记录(如NameNode、DataNode),寻找潜在的报错信息或其他线索帮助定位具体问题所在。 通过以上措施通常能够有效应对大多数常见的HDFS写入障碍情况。当然,在实际环境中还可能存在更复杂的情形,这时建议联系专业的技术支持团队获取更多协助。
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值