Data_web(九)mongodb增量同步到mongodb

原创已于 2022-08-18 11:41:31 修改 · 1k 阅读

3 ·

CC 4.0 BY-SA版权

文章标签：

#mongodb #数据库

于 2022-07-30 16:23:19 首次发布

liunx 同时被 3 个专栏收录

14 篇文章

订阅专栏

数据库

11 篇文章

订阅专栏

shell

3 篇文章

订阅专栏

本文详细讲解了如何通过JSON配置连接MongoDB，包括设置MongoDB Reader和Writer的参数，如速度限制、查询条件，并重点介绍了如何使用时间戳进行增量数据同步。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

1.mongodb连接、字段等
查看上一篇
2.设置辅助增量要求如下
在这里插入图片描述
3.json格式如下

{
  "job": {
    "setting": {
      "speed": {
        "channel": 3,
        "byte": 1048576
      },
      "errorLimit": {
        "record": 0,
        "percentage": 0.02
      }
    },
    "content": [
      {
        "reader": {
          "name": "mongodbreader",
          "parameter": {
            "address": [
              "192.168.1.100:27017"
            ],
            "userName": "username",
            "userPassword": "password",
            "dbName": "DBname",
            "collectionName": "collectionname",
            "column": [
              {
                "name": "sourceSystem",
                "type": "String"
              },
              {
                "name": "configId",
                "type": "String"
              },
              {
                "name": "target",
                "type": "String"
              },
              {
                "name": "userName",
                "type": "String"
              },
              {
                "name": "paramJson",
                "type": "String"
              },
              {
                "name": "sendTime",
                "type": "Date"
              },
              {
                "name": "detailId",
                "type": "Long"
              }
            ],
            "query": "{\"sendTime\":{\"$gte\":ISODate(${lastTime}),\"$lt\":ISODate(${currentTime})},\"detailType\":\"app\",\"sourceSystem\":\"qfcoss\",\"success\": {\"$eq\": true}}"
          }
        },
        "writer": {
          "name": "mongodbwriter",
          "parameter": {
            "address": [
              "192.168.1.101:27017"
            ],
            "userName": "username",
            "userPassword": "password",
            "dbName": "DBname",
            "collectionName": "collectionname",
            "column": [
              {
                "name": "sourceSystem",
                "type": "String"
              },
              {
                "name": "app_code",
                "type": "String"
              },
              {
                "name": "target",
                "type": "String"
              },
              {
                "name": "user_name",
                "type": "String"
              },
              {
                "name": "content",
                "type": "String"
              },
              {
                "name": "send_time",
                "type": "Date"
              },
              {
                "name": "outer_id",
                "type": "Long"
              }
            ],
            "upsertInfo": {
              "isUpsert": false,
              "upsertKey": ""
            }
          }
        }
      }
    ]
  }
}

query筛选基本和mongodb的find方法类似 sendTime是时间格式筛选， success是布尔值筛选，detailType是字符或者int筛选可以参考

如果有时间戳字段，增量时间段修改成 %s注意不要有单引号
在这里插入图片描述
query修改为

    "query": "{\"updateTimestamp\":{\"$gte\":${lastTime},\"$lt\":${currentTime}},\"detailType\":\"app\",\"sourceSystem\":\"qfcoss\",\"success\": {\"$eq\": true}}"