============================================================

New run started at 2024-05-15.01:56:14.418879

sys.argv: "/usr/local/lib/python3.10/dist-packages/colab_kernel_launcher.py -f
/root/.local/share/jupyter/runtime/kernel-834206b9-e330-4cf2-9320-
9ff36282a7b4.json"
============================================================
The console stream is logged into /root/sg_logs/console.log

Indexing dataset annotations: 0%| | 0/101 [00:00<?, ?it/s]

Indexing dataset annotations: 100%|██████████| 101/101 [00:00<00:00, 3103.34it/s]

Indexing dataset annotations: 0%| | 0/21 [00:00<?, ?it/s]

Indexing dataset annotations: 100%|██████████| 21/21 [00:00<00:00, 3325.04it/s]

Indexing dataset annotations: 0%| | 0/23 [00:00<?, ?it/s]

Indexing dataset annotations: 100%|██████████| 23/23 [00:00<00:00, 3274.80it/s]
Downloading: "https://sghub.deci.ai/models/yolo_nas_s_coco.pth" to
/root/.cache/torch/hub/checkpoints/yolo_nas_s_coco.pth

0%| | 0.00/73.1M [00:00<?, ?B/s]

0%| | 8.00k/73.1M [00:00<36:39, 34.9kB/s]
0%| | 48.0k/73.1M [00:00<11:03, 115kB/s]
0%| | 96.0k/73.1M [00:00<07:59, 160kB/s]
0%| | 216k/73.1M [00:00<04:15, 300kB/s]
1%| | 456k/73.1M [00:01<02:13, 569kB/s]
1%| | 912k/73.1M [00:01<01:12, 1.04MB/s]
2%|▏ | 1.80M/73.1M [00:01<00:36, 2.06MB/s]
5%|▍ | 3.64M/73.1M [00:01<00:18, 4.03MB/s]
10%|▉ | 7.20M/73.1M [00:02<00:08, 7.71MB/s]
14%|█▎ | 9.94M/73.1M [00:02<00:07, 9.14MB/s]
19%|█▉ | 13.8M/73.1M [00:02<00:05, 11.6MB/s]
24%|██▍ | 17.7M/73.1M [00:02<00:04, 13.3MB/s]
30%|██▉ | 21.6M/73.1M [00:03<00:03, 14.4MB/s]
35%|███▍ | 25.5M/73.1M [00:03<00:03, 15.4MB/s]
40%|████ | 29.4M/73.1M [00:03<00:02, 15.8MB/s]
45%|████▍ | 32.8M/73.1M [00:03<00:02, 15.7MB/s]
50%|█████ | 36.7M/73.1M [00:04<00:02, 16.1MB/s]
55%|█████▌ | 40.5M/73.1M [00:04<00:02, 16.3MB/s]
61%|██████ | 44.3M/73.1M [00:04<00:01, 16.3MB/s]
66%|██████▌ | 48.2M/73.1M [00:04<00:01, 16.6MB/s]
70%|███████ | 51.4M/73.1M [00:04<00:01, 15.8MB/s]
76%|███████▌ | 55.3M/73.1M [00:05<00:01, 16.3MB/s]
80%|████████ | 58.8M/73.1M [00:05<00:00, 16.0MB/s]
86%|████████▌ | 62.7M/73.1M [00:05<00:00, 16.3MB/s]
89%|████████▉ | 65.3M/73.1M [00:05<00:00, 17.8MB/s]
92%|█████████▏| 67.1M/73.1M [00:05<00:00, 16.5MB/s]
94%|█████████▍| 68.7M/73.1M [00:06<00:00, 16.2MB/s]
97%|█████████▋| 71.3M/73.1M [00:06<00:00, 18.4MB/s]
100%|██████████| 73.1M/73.1M [00:06<00:00, 12.4MB/s]
The console stream is now moved to
checkpoints/potholes_yolonas_run/RUN_20240515_015743_420372/console_May15_01_57_43.
txt
[2024-05-15 01:57:43] INFO - sg_trainer.py - Using EMA with params {'decay': 0.99,
'decay_type': 'threshold'}
/usr/lib/python3.10/multiprocessing/popen_fork.py:66: RuntimeWarning: os.fork() was
called. os.fork() is incompatible with multithreaded code, and JAX is
multithreaded, so this will likely lead to a deadlock.
self.pid = os.fork()
[2024-05-15 01:57:49] INFO - sg_trainer_utils.py - TRAINING PARAMETERS:
 - Mode: Single GPU
- Number of GPUs: 1 (1 available on the machine)
- Full dataset size: 90 (len(train_set))
- Batch size per GPU: 16 (batch_size)
- Batch Accumulate: 1 (batch_accumulate)
- Total batch size: 16 (num_gpus * batch_size)
- Effective Batch size: 16 (num_gpus * batch_size *
batch_accumulate)
- Iterations per epoch: 5 (len(train_loader))
- Gradient updates per epoch: 5 (len(train_loader) /
batch_accumulate)
- Model: YoloNAS_S (19.02M parameters, 19.02M optimized)
- Learning Rates and Weight Decays:
- default: (19.02M parameters). LR: 0.00015 (19.02M parameters) WD: 0.0,
(42.13K parameters), WD: 0.04, (18.98M parameters)

[2024-05-15 01:58:02] INFO - base_sg_logger.py - Checkpoint saved in

checkpoints/potholes_yolonas_run/RUN_20240515_015743_420372/ckpt_best.pth
[2024-05-15 01:58:02] INFO - sg_trainer.py - Best checkpoint overriden: validation
[email protected]: 0.0
[2024-05-15 01:58:33] INFO - base_sg_logger.py - Checkpoint saved in
checkpoints/potholes_yolonas_run/RUN_20240515_015743_420372/ckpt_best.pth
[2024-05-15 01:58:33] INFO - sg_trainer.py - Best checkpoint overriden: validation
[email protected]: 0.003975240048021078
[2024-05-15 01:59:37] INFO - base_sg_logger.py - Checkpoint saved in
checkpoints/potholes_yolonas_run/RUN_20240515_015743_420372/ckpt_best.pth
[2024-05-15 01:59:37] INFO - sg_trainer.py - Best checkpoint overriden: validation
[email protected]: 0.004846301395446062
[2024-05-15 01:59:53] INFO - base_sg_logger.py - Checkpoint saved in
checkpoints/potholes_yolonas_run/RUN_20240515_015743_420372/ckpt_best.pth
[2024-05-15 01:59:53] INFO - sg_trainer.py - Best checkpoint overriden: validation
[email protected]: 0.0103761563077569
[2024-05-15 02:00:08] INFO - base_sg_logger.py - Checkpoint saved in
checkpoints/potholes_yolonas_run/RUN_20240515_015743_420372/ckpt_best.pth
[2024-05-15 02:00:08] INFO - sg_trainer.py - Best checkpoint overriden: validation
[email protected]: 0.030371809378266335
[2024-05-15 02:00:24] INFO - base_sg_logger.py - Checkpoint saved in
checkpoints/potholes_yolonas_run/RUN_20240515_015743_420372/ckpt_best.pth
[2024-05-15 02:00:24] INFO - sg_trainer.py - Best checkpoint overriden: validation
[email protected]: 0.036327213048934937
[2024-05-15 02:01:19] INFO - base_sg_logger.py - Checkpoint saved in
checkpoints/potholes_yolonas_run/RUN_20240515_015743_420372/ckpt_best.pth
[2024-05-15 02:01:19] INFO - sg_trainer.py - Best checkpoint overriden: validation
[email protected]: 0.03877224028110504
[2024-05-15 02:01:34] INFO - base_sg_logger.py - Checkpoint saved in
checkpoints/potholes_yolonas_run/RUN_20240515_015743_420372/ckpt_best.pth
[2024-05-15 02:01:34] INFO - sg_trainer.py - Best checkpoint overriden: validation
[email protected]: 0.05426982045173645
[2024-05-15 02:01:51] INFO - base_sg_logger.py - Checkpoint saved in
checkpoints/potholes_yolonas_run/RUN_20240515_015743_420372/ckpt_best.pth
[2024-05-15 02:01:51] INFO - sg_trainer.py - Best checkpoint overriden: validation
[email protected]: 0.06179603189229965
[2024-05-15 02:02:06] INFO - base_sg_logger.py - Checkpoint saved in
checkpoints/potholes_yolonas_run/RUN_20240515_015743_420372/ckpt_best.pth
[2024-05-15 02:02:06] INFO - sg_trainer.py - Best checkpoint overriden: validation
[email protected]: 0.07158444076776505
[2024-05-15 02:02:49] INFO - base_sg_logger.py - Checkpoint saved in
checkpoints/potholes_yolonas_run/RUN_20240515_015743_420372/ckpt_best.pth
[2024-05-15 02:02:49] INFO - sg_trainer.py - Best checkpoint overriden: validation
[email protected]: 0.08547002077102661
[2024-05-15 02:03:09] INFO - base_sg_logger.py - Checkpoint saved in
checkpoints/potholes_yolonas_run/RUN_20240515_015743_420372/ckpt_best.pth
[2024-05-15 02:03:09] INFO - sg_trainer.py - Best checkpoint overriden: validation
[email protected]: 0.09145709127187729
[2024-05-15 02:03:29] INFO - base_sg_logger.py - Checkpoint saved in
checkpoints/potholes_yolonas_run/RUN_20240515_015743_420372/ckpt_best.pth
[2024-05-15 02:03:29] INFO - sg_trainer.py - Best checkpoint overriden: validation
[email protected]: 0.09474649280309677
[2024-05-15 02:03:48] INFO - base_sg_logger.py - Checkpoint saved in
checkpoints/potholes_yolonas_run/RUN_20240515_015743_420372/ckpt_best.pth
[2024-05-15 02:03:48] INFO - sg_trainer.py - Best checkpoint overriden: validation
[email protected]: 0.10865537822246552
[2024-05-15 02:04:08] INFO - base_sg_logger.py - Checkpoint saved in
checkpoints/potholes_yolonas_run/RUN_20240515_015743_420372/ckpt_best.pth
[2024-05-15 02:04:08] INFO - sg_trainer.py - Best checkpoint overriden: validation
[email protected]: 0.12823665142059326
[2024-05-15 02:04:29] INFO - base_sg_logger.py - Checkpoint saved in
checkpoints/potholes_yolonas_run/RUN_20240515_015743_420372/ckpt_best.pth
[2024-05-15 02:04:29] INFO - sg_trainer.py - Best checkpoint overriden: validation
[email protected]: 0.13176535069942474
[2024-05-15 02:05:10] INFO - base_sg_logger.py - Checkpoint saved in
checkpoints/potholes_yolonas_run/RUN_20240515_015743_420372/ckpt_best.pth
[2024-05-15 02:05:10] INFO - sg_trainer.py - Best checkpoint overriden: validation
[email protected]: 0.13201041519641876
[2024-05-15 02:05:33] INFO - base_sg_logger.py - Checkpoint saved in
checkpoints/potholes_yolonas_run/RUN_20240515_015743_420372/ckpt_best.pth
[2024-05-15 02:05:33] INFO - sg_trainer.py - Best checkpoint overriden: validation
[email protected]: 0.13320530951023102
[2024-05-15 02:06:27] INFO - base_sg_logger.py - Checkpoint saved in
checkpoints/potholes_yolonas_run/RUN_20240515_015743_420372/ckpt_best.pth
[2024-05-15 02:06:27] INFO - sg_trainer.py - Best checkpoint overriden: validation
[email protected]: 0.14553937315940857
[2024-05-15 02:06:51] INFO - base_sg_logger.py - Checkpoint saved in
checkpoints/potholes_yolonas_run/RUN_20240515_015743_420372/ckpt_best.pth
[2024-05-15 02:06:51] INFO - sg_trainer.py - Best checkpoint overriden: validation
[email protected]: 0.15361304581165314
[2024-05-15 02:07:17] INFO - base_sg_logger.py - Checkpoint saved in
checkpoints/potholes_yolonas_run/RUN_20240515_015743_420372/ckpt_best.pth
[2024-05-15 02:07:17] INFO - sg_trainer.py - Best checkpoint overriden: validation
[email protected]: 0.1583136022090912
[2024-05-15 02:07:42] INFO - base_sg_logger.py - Checkpoint saved in
checkpoints/potholes_yolonas_run/RUN_20240515_015743_420372/ckpt_best.pth
[2024-05-15 02:07:42] INFO - sg_trainer.py - Best checkpoint overriden: validation
[email protected]: 0.20079851150512695
[2024-05-15 02:08:08] INFO - base_sg_logger.py - Checkpoint saved in
checkpoints/potholes_yolonas_run/RUN_20240515_015743_420372/ckpt_best.pth
[2024-05-15 02:08:08] INFO - sg_trainer.py - Best checkpoint overriden: validation
[email protected]: 0.21229144930839539
[2024-05-15 02:08:28] INFO - base_sg_logger.py - Checkpoint saved in
checkpoints/potholes_yolonas_run/RUN_20240515_015743_420372/ckpt_best.pth
[2024-05-15 02:08:28] INFO - sg_trainer.py - Best checkpoint overriden: validation
[email protected]: 0.21877868473529816
[2024-05-15 02:08:51] INFO - base_sg_logger.py - Checkpoint saved in
checkpoints/potholes_yolonas_run/RUN_20240515_015743_420372/ckpt_best.pth
[2024-05-15 02:08:51] INFO - sg_trainer.py - Best checkpoint overriden: validation
[email protected]: 0.25192686915397644
[2024-05-15 02:09:17] INFO - base_sg_logger.py - Checkpoint saved in
checkpoints/potholes_yolonas_run/RUN_20240515_015743_420372/ckpt_best.pth
[2024-05-15 02:09:17] INFO - sg_trainer.py - Best checkpoint overriden: validation
[email protected]: 0.2603192925453186
[2024-05-15 02:09:35] INFO - base_sg_logger.py - Checkpoint saved in
checkpoints/potholes_yolonas_run/RUN_20240515_015743_420372/ckpt_best.pth
[2024-05-15 02:09:35] INFO - sg_trainer.py - Best checkpoint overriden: validation
[email protected]: 0.26699507236480713
[2024-05-15 02:10:00] INFO - base_sg_logger.py - Checkpoint saved in
checkpoints/potholes_yolonas_run/RUN_20240515_015743_420372/ckpt_best.pth
[2024-05-15 02:10:00] INFO - sg_trainer.py - Best checkpoint overriden: validation
[email protected]: 0.2702477276325226
[2024-05-15 02:10:42] INFO - base_sg_logger.py - Checkpoint saved in
checkpoints/potholes_yolonas_run/RUN_20240515_015743_420372/ckpt_best.pth
[2024-05-15 02:10:42] INFO - sg_trainer.py - Best checkpoint overriden: validation
[email protected]: 0.2715684175491333
[2024-05-15 02:11:06] INFO - base_sg_logger.py - Checkpoint saved in
checkpoints/potholes_yolonas_run/RUN_20240515_015743_420372/ckpt_best.pth
[2024-05-15 02:11:06] INFO - sg_trainer.py - Best checkpoint overriden: validation
[email protected]: 0.27291247248649597
[2024-05-15 02:11:30] INFO - base_sg_logger.py - Checkpoint saved in
checkpoints/potholes_yolonas_run/RUN_20240515_015743_420372/ckpt_best.pth
[2024-05-15 02:11:30] INFO - sg_trainer.py - Best checkpoint overriden: validation
[email protected]: 0.27786684036254883
[2024-05-15 02:12:09] INFO - base_sg_logger.py - Checkpoint saved in
checkpoints/potholes_yolonas_run/RUN_20240515_015743_420372/ckpt_best.pth
[2024-05-15 02:12:09] INFO - sg_trainer.py - Best checkpoint overriden: validation
[email protected]: 0.2893027067184448
[2024-05-15 02:12:31] INFO - base_sg_logger.py - Checkpoint saved in
checkpoints/potholes_yolonas_run/RUN_20240515_015743_420372/ckpt_best.pth
[2024-05-15 02:12:31] INFO - sg_trainer.py - Best checkpoint overriden: validation
[email protected]: 0.296338826417923
[2024-05-15 02:12:54] INFO - base_sg_logger.py - Checkpoint saved in
checkpoints/potholes_yolonas_run/RUN_20240515_015743_420372/ckpt_best.pth
[2024-05-15 02:12:54] INFO - sg_trainer.py - Best checkpoint overriden: validation
[email protected]: 0.3521914780139923
[2024-05-15 02:13:20] INFO - base_sg_logger.py - Checkpoint saved in
checkpoints/potholes_yolonas_run/RUN_20240515_015743_420372/ckpt_best.pth
[2024-05-15 02:13:20] INFO - sg_trainer.py - Best checkpoint overriden: validation
[email protected]: 0.3618684411048889
[2024-05-15 02:14:04] INFO - base_sg_logger.py - Checkpoint saved in
checkpoints/potholes_yolonas_run/RUN_20240515_015743_420372/ckpt_best.pth
[2024-05-15 02:14:04] INFO - sg_trainer.py - Best checkpoint overriden: validation
[email protected]: 0.3791787624359131
[2024-05-15 02:14:24] INFO - base_sg_logger.py - Checkpoint saved in
checkpoints/potholes_yolonas_run/RUN_20240515_015743_420372/ckpt_best.pth
[2024-05-15 02:14:24] INFO - sg_trainer.py - Best checkpoint overriden: validation
[email protected]: 0.3856964111328125
[2024-05-15 02:15:06] INFO - base_sg_logger.py - Checkpoint saved in
checkpoints/potholes_yolonas_run/RUN_20240515_015743_420372/ckpt_best.pth
[2024-05-15 02:15:06] INFO - sg_trainer.py - Best checkpoint overriden: validation
[email protected]: 0.42024701833724976
[2024-05-15 02:16:21] INFO - base_sg_logger.py - Checkpoint saved in
checkpoints/potholes_yolonas_run/RUN_20240515_015743_420372/ckpt_best.pth
[2024-05-15 02:16:21] INFO - sg_trainer.py - Best checkpoint overriden: validation
[email protected]: 0.42577797174453735
[2024-05-15 02:20:28] INFO - base_sg_logger.py - Checkpoint saved in
checkpoints/potholes_yolonas_run/RUN_20240515_015743_420372/ckpt_best.pth
[2024-05-15 02:20:28] INFO - sg_trainer.py - Best checkpoint overriden: validation
[email protected]: 0.43646055459976196
[2024-05-15 02:20:47] INFO - base_sg_logger.py - Checkpoint saved in
checkpoints/potholes_yolonas_run/RUN_20240515_015743_420372/ckpt_best.pth
[2024-05-15 02:20:47] INFO - sg_trainer.py - Best checkpoint overriden: validation
[email protected]: 0.44257038831710815
[2024-05-15 02:21:10] INFO - base_sg_logger.py - Checkpoint saved in
checkpoints/potholes_yolonas_run/RUN_20240515_015743_420372/ckpt_best.pth
[2024-05-15 02:21:10] INFO - sg_trainer.py - Best checkpoint overriden: validation
[email protected]: 0.45205697417259216
[2024-05-15 02:21:25] INFO - base_sg_logger.py - Checkpoint saved in
checkpoints/potholes_yolonas_run/RUN_20240515_015743_420372/ckpt_best.pth
[2024-05-15 02:21:25] INFO - sg_trainer.py - Best checkpoint overriden: validation
[email protected]: 0.4660283029079437
[2024-05-15 02:22:58] INFO - base_sg_logger.py - Checkpoint saved in
checkpoints/potholes_yolonas_run/RUN_20240515_015743_420372/ckpt_best.pth
[2024-05-15 02:22:58] INFO - sg_trainer.py - Best checkpoint overriden: validation
[email protected]: 0.4689520597457886
[2024-05-15 02:23:13] INFO - base_sg_logger.py - Checkpoint saved in
checkpoints/potholes_yolonas_run/RUN_20240515_015743_420372/ckpt_best.pth
[2024-05-15 02:23:13] INFO - sg_trainer.py - Best checkpoint overriden: validation
[email protected]: 0.48519501090049744
[2024-05-15 02:23:37] INFO - base_sg_logger.py - Checkpoint saved in
checkpoints/potholes_yolonas_run/RUN_20240515_015743_420372/ckpt_best.pth
[2024-05-15 02:23:37] INFO - sg_trainer.py - Best checkpoint overriden: validation
[email protected]: 0.48908162117004395
[2024-05-15 02:23:52] INFO - base_sg_logger.py - Checkpoint saved in
checkpoints/potholes_yolonas_run/RUN_20240515_015743_420372/ckpt_best.pth
[2024-05-15 02:23:52] INFO - sg_trainer.py - Best checkpoint overriden: validation
[email protected]: 0.5072428584098816
[2024-05-15 02:24:15] INFO - base_sg_logger.py - Checkpoint saved in
checkpoints/potholes_yolonas_run/RUN_20240515_015743_420372/ckpt_best.pth
[2024-05-15 02:24:15] INFO - sg_trainer.py - Best checkpoint overriden: validation
[email protected]: 0.512956976890564
[2024-05-15 02:24:30] INFO - base_sg_logger.py - Checkpoint saved in
checkpoints/potholes_yolonas_run/RUN_20240515_015743_420372/ckpt_best.pth
[2024-05-15 02:24:30] INFO - sg_trainer.py - Best checkpoint overriden: validation
[email protected]: 0.5162416696548462
[2024-05-15 02:24:52] INFO - base_sg_logger.py - Checkpoint saved in
checkpoints/potholes_yolonas_run/RUN_20240515_015743_420372/ckpt_best.pth
[2024-05-15 02:24:53] INFO - sg_trainer.py - Best checkpoint overriden: validation
[email protected]: 0.5271567702293396
[2024-05-15 02:25:56] INFO - base_sg_logger.py - Checkpoint saved in
checkpoints/potholes_yolonas_run/RUN_20240515_015743_420372/ckpt_best.pth
[2024-05-15 02:25:56] INFO - sg_trainer.py - Best checkpoint overriden: validation
[email protected]: 0.5327532887458801
[2024-05-15 02:26:17] INFO - base_sg_logger.py - Checkpoint saved in
checkpoints/potholes_yolonas_run/RUN_20240515_015743_420372/ckpt_best.pth
[2024-05-15 02:26:17] INFO - sg_trainer.py - Best checkpoint overriden: validation
[email protected]: 0.5334205031394958
[2024-05-15 02:36:29] INFO - sg_trainer.py - RUNNING ADDITIONAL TEST ON THE
AVERAGED MODEL...

0%| | 0/2 [00:00<?, ?it/s]

Validating epoch 120: 0%| | 0/2 [00:00<?, ?it/s]
Validating epoch 120: 50%|█████ | 1/2 [00:00<00:00, 1.03it/s]
Validating epoch 120: 100%|██████████| 2/2 [00:01<00:00, 1.80it/s]
[2024-05-15 02:36:30] INFO - base_sg_logger.py - [CLEANUP] - Successfully stopped
system monitoring process
[2024-05-15 02:36:33] INFO - checkpoint_utils.py - Successfully loaded model
weights from /content/checkpoints/potholes_yolonas_run/RUN_20240515_015743_420372/
ckpt_best.pth EMA checkpoint.
 0%| | 0/2 [00:00<?, ?it/s]
Testing: 0%| | 0/2 [00:00<?, ?it/s]
Testing: 50%|█████ | 1/2 [00:01<00:01, 1.33s/it]
Testing: 100%|██████████| 2/2 [00:01<00:00, 1.47it/s]
Testing: 100%|██████████| 2/2 [00:01<00:00, 1.20it/s]
[2024-05-15 02:41:43] INFO - sg_trainer.py - Starting a new run with
`run_id=RUN_20240515_024143_233164`
[2024-05-15 02:41:43] INFO - sg_trainer.py - Checkpoints directory:
checkpoints/potholes_yolonas_run/RUN_20240515_024143_233164
The console stream is now moved to
checkpoints/potholes_yolonas_run/RUN_20240515_024143_233164/console_May15_02_41_43.
txt
[2024-05-15 02:41:43] INFO - sg_trainer.py - Using EMA with params {'decay': 0.99,
'decay_type': 'threshold'}
[2024-05-15 02:41:48] INFO - sg_trainer_utils.py - TRAINING PARAMETERS:
- Mode: Single GPU
- Number of GPUs: 1 (1 available on the machine)
- Full dataset size: 90 (len(train_set))
- Batch size per GPU: 16 (batch_size)
- Batch Accumulate: 1 (batch_accumulate)
- Total batch size: 16 (num_gpus * batch_size)
- Effective Batch size: 16 (num_gpus * batch_size *
batch_accumulate)
- Iterations per epoch: 5 (len(train_loader))
- Gradient updates per epoch: 5 (len(train_loader) /
batch_accumulate)
- Model: YoloNAS_S (19.02M parameters, 19.02M optimized)
- Learning Rates and Weight Decays:
- default: (19.02M parameters). LR: 0.00015 (19.02M parameters) WD: 0.0,
(42.13K parameters), WD: 0.04, (18.98M parameters)

[2024-05-15 02:42:02] INFO - base_sg_logger.py - Checkpoint saved in

checkpoints/potholes_yolonas_run/RUN_20240515_024143_233164/ckpt_best.pth
[2024-05-15 02:42:02] INFO - sg_trainer.py - Best checkpoint overriden: validation
[email protected]: 0.4811771512031555
[2024-05-15 02:44:52] INFO - base_sg_logger.py - Checkpoint saved in
checkpoints/potholes_yolonas_run/RUN_20240515_024143_233164/ckpt_best.pth
[2024-05-15 02:44:52] INFO - sg_trainer.py - Best checkpoint overriden: validation
[email protected]: 0.5170091390609741
[2024-05-15 02:45:18] INFO - base_sg_logger.py - Checkpoint saved in
checkpoints/potholes_yolonas_run/RUN_20240515_024143_233164/ckpt_best.pth
[2024-05-15 02:45:18] INFO - sg_trainer.py - Best checkpoint overriden: validation
[email protected]: 0.5269309878349304
[2024-05-15 02:45:51] INFO - base_sg_logger.py - Checkpoint saved in
checkpoints/potholes_yolonas_run/RUN_20240515_024143_233164/ckpt_best.pth
[2024-05-15 02:45:51] INFO - sg_trainer.py - Best checkpoint overriden: validation
[email protected]: 0.5359565615653992
[2024-05-15 02:49:30] INFO - base_sg_logger.py - Checkpoint saved in
checkpoints/potholes_yolonas_run/RUN_20240515_024143_233164/ckpt_best.pth
[2024-05-15 02:49:30] INFO - sg_trainer.py - Best checkpoint overriden: validation
[email protected]: 0.5448123216629028
[2024-05-15 02:51:15] INFO - base_sg_logger.py - Checkpoint saved in
checkpoints/potholes_yolonas_run/RUN_20240515_024143_233164/ckpt_best.pth
[2024-05-15 02:51:15] INFO - sg_trainer.py - Best checkpoint overriden: validation
[email protected]: 0.5509297251701355
[2024-05-15 02:55:19] INFO - base_sg_logger.py - Checkpoint saved in
checkpoints/potholes_yolonas_run/RUN_20240515_024143_233164/ckpt_best.pth
[2024-05-15 02:55:19] INFO - sg_trainer.py - Best checkpoint overriden: validation
[email protected]: 0.5947632789611816
[2024-05-15 02:55:48] INFO - base_sg_logger.py - Checkpoint saved in
checkpoints/potholes_yolonas_run/RUN_20240515_024143_233164/ckpt_best.pth
[2024-05-15 02:55:48] INFO - sg_trainer.py - Best checkpoint overriden: validation
[email protected]: 0.6220775842666626
[2024-05-15 02:58:22] INFO - base_sg_logger.py - Checkpoint saved in
checkpoints/potholes_yolonas_run/RUN_20240515_024143_233164/ckpt_best.pth
[2024-05-15 02:58:22] INFO - sg_trainer.py - Best checkpoint overriden: validation
[email protected]: 0.6336398124694824
[2024-05-15 02:59:18] INFO - base_sg_logger.py - Checkpoint saved in
checkpoints/potholes_yolonas_run/RUN_20240515_024143_233164/ckpt_best.pth
[2024-05-15 02:59:18] INFO - sg_trainer.py - Best checkpoint overriden: validation
[email protected]: 0.6356057524681091
[2024-05-15 03:01:38] INFO - base_sg_logger.py - Checkpoint saved in
checkpoints/potholes_yolonas_run/RUN_20240515_024143_233164/ckpt_best.pth
[2024-05-15 03:01:38] INFO - sg_trainer.py - Best checkpoint overriden: validation
[email protected]: 0.6368426084518433
[2024-05-15 03:06:46] INFO - base_sg_logger.py - Checkpoint saved in
checkpoints/potholes_yolonas_run/RUN_20240515_024143_233164/ckpt_best.pth
[2024-05-15 03:06:46] INFO - sg_trainer.py - Best checkpoint overriden: validation
[email protected]: 0.639693021774292
[2024-05-15 03:07:06] INFO - base_sg_logger.py - Checkpoint saved in
checkpoints/potholes_yolonas_run/RUN_20240515_024143_233164/ckpt_best.pth
[2024-05-15 03:07:06] INFO - sg_trainer.py - Best checkpoint overriden: validation
[email protected]: 0.6461964249610901
[2024-05-15 03:32:20] INFO - base_sg_logger.py - Checkpoint saved in
checkpoints/potholes_yolonas_run/RUN_20240515_024143_233164/ckpt_best.pth
[2024-05-15 03:32:20] INFO - sg_trainer.py - Best checkpoint overriden: validation
[email protected]: 0.6464382410049438
[2024-05-15 03:32:37] INFO - base_sg_logger.py - Checkpoint saved in
checkpoints/potholes_yolonas_run/RUN_20240515_024143_233164/ckpt_best.pth
[2024-05-15 03:32:37] INFO - sg_trainer.py - Best checkpoint overriden: validation
[email protected]: 0.6488229632377625
[2024-05-15 03:33:21] INFO - sg_trainer.py - RUNNING ADDITIONAL TEST ON THE
AVERAGED MODEL...

0%| | 0/2 [00:00<?, ?it/s]

Validating epoch 120: 0%| | 0/2 [00:00<?, ?it/s]
Validating epoch 120: 50%|█████ | 1/2 [00:00<00:00, 1.05it/s]
Validating epoch 120: 100%|██████████| 2/2 [00:01<00:00, 1.84it/s]
[2024-05-15 03:33:22] INFO - base_sg_logger.py - [CLEANUP] - Successfully stopped
system monitoring process
[2024-05-15 03:33:23] INFO - checkpoint_utils.py - Successfully loaded model
weights from /content/checkpoints/potholes_yolonas_run/RUN_20240515_024143_233164/
ckpt_best.pth EMA checkpoint.

0%| | 0/2 [00:00<?, ?it/s]

Testing: 0%| | 0/2 [00:00<?, ?it/s]
Testing: 50%|█████ | 1/2 [00:01<00:01, 1.08s/it]
Testing: 100%|██████████| 2/2 [00:01<00:00, 1.83it/s]
Testing: 100%|██████████| 2/2 [00:01<00:00, 1.49it/s]
[2024-05-15 03:36:10] INFO - sg_trainer.py - Starting a new run with
`run_id=RUN_20240515_033610_422916`
[2024-05-15 03:36:10] INFO - sg_trainer.py - Checkpoints directory:
checkpoints/potholes_yolonas_run/RUN_20240515_033610_422916
The console stream is now moved to
checkpoints/potholes_yolonas_run/RUN_20240515_033610_422916/console_May15_03_36_10.
txt
[2024-05-15 03:36:10] INFO - sg_trainer.py - Using EMA with params {'decay': 0.995,
'decay_type': 'threshold'}
[2024-05-15 03:36:15] INFO - sg_trainer_utils.py - TRAINING PARAMETERS:
- Mode: Single GPU
- Number of GPUs: 1 (1 available on the machine)
- Full dataset size: 90 (len(train_set))
- Batch size per GPU: 16 (batch_size)
- Batch Accumulate: 1 (batch_accumulate)
- Total batch size: 16 (num_gpus * batch_size)
- Effective Batch size: 16 (num_gpus * batch_size *
batch_accumulate)
- Iterations per epoch: 5 (len(train_loader))
- Gradient updates per epoch: 5 (len(train_loader) /
batch_accumulate)
- Model: YoloNAS_S (19.02M parameters, 19.02M optimized)
- Learning Rates and Weight Decays:
- default: (19.02M parameters). LR: 0.0001 (19.02M parameters) WD: 0.0,
(42.13K parameters), WD: 0.02, (18.98M parameters)

[2024-05-15 03:36:30] INFO - base_sg_logger.py - Checkpoint saved in

checkpoints/potholes_yolonas_run/RUN_20240515_033610_422916/ckpt_best.pth
[2024-05-15 03:36:30] INFO - sg_trainer.py - Best checkpoint overriden: validation
[email protected]: 0.6485021114349365
[2024-05-15 03:40:59] INFO - base_sg_logger.py - Checkpoint saved in
checkpoints/potholes_yolonas_run/RUN_20240515_033610_422916/ckpt_best.pth
[2024-05-15 03:40:59] INFO - sg_trainer.py - Best checkpoint overriden: validation
[email protected]: 0.6633641719818115
[2024-05-15 03:48:50] INFO - base_sg_logger.py - Checkpoint saved in
checkpoints/potholes_yolonas_run/RUN_20240515_033610_422916/ckpt_best.pth
[2024-05-15 03:48:50] INFO - sg_trainer.py - Best checkpoint overriden: validation
[email protected]: 0.6661654710769653
[2024-05-15 03:49:15] INFO - base_sg_logger.py - Checkpoint saved in
checkpoints/potholes_yolonas_run/RUN_20240515_033610_422916/ckpt_best.pth
[2024-05-15 03:49:15] INFO - sg_trainer.py - Best checkpoint overriden: validation
[email protected]: 0.6704705357551575
[2024-05-15 03:50:56] INFO - base_sg_logger.py - Checkpoint saved in
checkpoints/potholes_yolonas_run/RUN_20240515_033610_422916/ckpt_best.pth
[2024-05-15 03:50:56] INFO - sg_trainer.py - Best checkpoint overriden: validation
[email protected]: 0.6853491067886353
[2024-05-15 03:51:24] INFO - base_sg_logger.py - Checkpoint saved in
checkpoints/potholes_yolonas_run/RUN_20240515_033610_422916/ckpt_best.pth
[2024-05-15 03:51:24] INFO - sg_trainer.py - Best checkpoint overriden: validation
[email protected]: 0.6867925524711609
[2024-05-15 03:51:52] INFO - base_sg_logger.py - Checkpoint saved in
checkpoints/potholes_yolonas_run/RUN_20240515_033610_422916/ckpt_best.pth
[2024-05-15 03:51:52] INFO - sg_trainer.py - Best checkpoint overriden: validation
[email protected]: 0.7051923274993896
[2024-05-15 03:54:40] INFO - base_sg_logger.py - Checkpoint saved in
checkpoints/potholes_yolonas_run/RUN_20240515_033610_422916/ckpt_best.pth
[2024-05-15 03:54:40] INFO - sg_trainer.py - Best checkpoint overriden: validation
[email protected]: 0.7082313895225525
[2024-05-15 03:55:24] INFO - base_sg_logger.py - Checkpoint saved in
checkpoints/potholes_yolonas_run/RUN_20240515_033610_422916/ckpt_best.pth
[2024-05-15 03:55:24] INFO - sg_trainer.py - Best checkpoint overriden: validation
[email protected]: 0.7191919088363647
[2024-05-15 03:56:11] INFO - base_sg_logger.py - Checkpoint saved in
checkpoints/potholes_yolonas_run/RUN_20240515_033610_422916/ckpt_best.pth
[2024-05-15 03:56:11] INFO - sg_trainer.py - Best checkpoint overriden: validation
[email protected]: 0.7263413071632385
[2024-05-15 04:35:01] INFO - sg_trainer.py - RUNNING ADDITIONAL TEST ON THE
AVERAGED MODEL...
0%| | 0/2 [00:00<?, ?it/s]Validating epoch 150: 0%| | 0/2
[00:00<?, ?it/s]Validating epoch 150: 50%|█████ | 1/2 [00:00<00:00, 1.01it/s]
Validating epoch 150: 100%|██████████| 2/2 [00:01<00:00, 1.76it/s]