cpu飙高问题排查

1.top命令查看cpu情况

发现进程29851一直居高不下

2.先找出占用 CPU 最高的线程 ID

top  -H  -p  29851

发现前10个线程cpu都很高

3. 将线程 ID 转换为 16 进制

先看线程14401

（1）printf "%x\n" [线程ID]   得到16进制数据

4. 执行 jstack

 jstack 29851 > stack.txt 

5. 查找对应的高 CPU线程堆栈

grep -A 30   [16进制线程ID]   stack.txt

"http-nio-18080-exec-3" #106 daemon prio=5 os_prio=0 tid=0x00007ff3d7268800 nid=0x3843 waiting for monitor entry [0x00007ff322ae1000]
   java.lang.Thread.State: BLOCKED (on object monitor)
        at org.springframework.boot.loader.data.RandomAccessDataFile$FileAccess.read(RandomAccessDataFile.java:219)
        - locked <0x0000000640096fc8> (a java.lang.Object)
        at org.springframework.boot.loader.data.RandomAccessDataFile$FileAccess.access$400(RandomAccessDataFile.java:205)
        at org.springframework.boot.loader.data.RandomAccessDataFile.read(RandomAccessDataFile.java:117)
        at org.springframework.boot.loader.data.RandomAccessDataFile.access$700(RandomAccessDataFile.java:33)
        at org.springframework.boot.loader.data.RandomAccessDataFile$DataInputStream.doRead(RandomAccessDataFile.java:175)
        at org.springframework.boot.loader.data.RandomAccessDataFile$DataInputStream.read(RandomAccessDataFile.java:155)
        at java.util.zip.InflaterInputStream.fill(InflaterInputStream.java:238)
        at org.springframework.boot.loader.jar.ZipInflaterInputStream.fill(ZipInflaterInputStream.java:68)
        at java.util.zip.InflaterInputStream.read(InflaterInputStream.java:158)
        at org.springframework.boot.loader.jar.ZipInflaterInputStream.read(ZipInflaterInputStream.java:52)
        at java.io.FilterInputStream.read(FilterInputStream.java:107)
        at com.maxmind.db.BufferHolder.<init>(BufferHolder.java:63)
        at com.maxmind.db.Reader.<init>(Reader.java:89)
        at com.maxmind.geoip2.DatabaseReader.<init>(DatabaseReader.java:33)
        at com.maxmind.geoip2.DatabaseReader.<init>(DatabaseReader.java:23)
        at com.maxmind.geoip2.DatabaseReader$Builder.build(DatabaseReader.java:129)
        at cn.com.antwalletbot.test.GeoIpTest.getCountry(GeoIpTest.java:43)
        at cn.com.antwalletbot.common.config.interceptor.IpRateLimiterFilter.doFilterInternal(IpRateLimiterFilter.java:74)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
        at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
        at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)

从这个线程堆栈来看，我发现了一个比较严重的性能问题：

（1）问题定位：

• 线程状态是 BLOCKED，等待获取监视器锁
• 问题出现在 IP 限流过滤器中 (IpRateLimiterFilter)
• 具体是在读取 GeoIP 数据库时发生的阻塞

（2）具体调用链：

IpRateLimiterFilter.doFilterInternal -> GeoIpTest.getCountry -> DatabaseReader.Builder.build -> DatabaseReader.<init> -> Reader.<init> -> BufferHolder.<init>

（3）核心问题：

• 每次请求都在尝试重新初始化 GeoIP 数据库读取器
• 这是一个重量级操作，不应该在请求过滤器中反复进行
• 多个线程在竞争同一个锁，导致大量阻塞

问题代码：

public class GeoIpTest {
    private static final Logger log = LoggerFactory.getLogger(GeoIpTest.class);

    public static void main(String[] args) {
        String ip = "2400:c600:3360:b864:1:0:de91:c6b5";
        getCountry(ip);
    }

    public static String getCountry(String ip) {
        try {
            if ("127.0.0.1".equals(ip)) {
                log.info("getCountry-本地IP，不进行查询");
                return StringUtils.EMPTY;
            }
            // 读取当前工程下的IP库文件
            InputStream inputStream = IpUtil.class.getClassLoader().getResourceAsStream("GeoLite2-Country.mmdb");


            // 使用 InputStream 创建 DatabaseReader
            DatabaseReader countryReader = new DatabaseReader.Builder(inputStream)
                    .withCache(new CHMCache())
                    .build();
            // 读取IP库文件

            CountryResponse countryResponse = countryReader.country(InetAddress.getByName(ip));
            String country = countryResponse.getCountry().getNames().get("zh-CN");
            log.info("从country IP库读取国家结果： {}", country);
            return country;
        } catch (Exception e) {
            //log.error("getCountry 从country IP库读取国家失败，ip:,错误信息是:{}", ip);
            return StringUtils.EMPTY;
        }
    }
} 

主要问题：

（1）主要性能问题： 

// 每次调用都重新创建 DatabaseReader，这是最大的性能瓶颈
DatabaseReader countryReader = new DatabaseReader.Builder(inputStream)
        .withCache(new CHMCache())
        .build();

（2）资源泄露问题：

// 没有关闭 InputStream 和 DatabaseReader
InputStream inputStream = IpUtil.class.getClassLoader().getResourceAsStream("GeoLite2-Country.mmdb");
// 应该使用 try-with-resources 或在 finally 中关闭

（3）改进方案：

@Component
public class GeoIpService {
    private static final Logger log = LoggerFactory.getLogger(GeoIpService.class);
    private DatabaseReader databaseReader;
    
    @PostConstruct
    public void init() {
        try (InputStream inputStream = GeoIpService.class.getClassLoader()
                .getResourceAsStream("GeoLite2-Country.mmdb")) {
            if (inputStream == null) {
                throw new IllegalStateException("Cannot find GeoLite2-Country.mmdb");
            }
            databaseReader = new DatabaseReader.Builder(inputStream)
                    .withCache(new CHMCache())
                    .build();
        } catch (Exception e) {
            log.error("Failed to initialize GeoIP database", e);
            throw new RuntimeException("Failed to initialize GeoIP database", e);
        }
    }
    
    @PreDestroy
    public void destroy() {
        if (databaseReader != null) {
            try {
                databaseReader.close();
            } catch (Exception e) {
                log.error("Error closing DatabaseReader", e);
            }
        }
    }

    public String getCountry(String ip) {
        if (StringUtils.isBlank(ip) || "127.0.0.1".equals(ip)) {
            return StringUtils.EMPTY;
        }

        try {
            CountryResponse response = databaseReader.country(InetAddress.getByName(ip));
            return response.getCountry().getNames().getOrDefault("zh-CN", StringUtils.EMPTY);
        } catch (Exception e) {
            log.error("Failed to get country for IP: {}", ip, e);
            return StringUtils.EMPTY;
        }
    }
}