AI Report 2023

Essays and insights from the world's largest data

science and machine learning community
 Abir Eltaief Meghana Bhange
Ali Jalali Nghi Huynh
Antong C. Parul Pandey
Anya Mathur Patrik Joslin Kenfack
Arya Gaikwad Paul Mooney
Bojan Tunguz Paulina Skorupska
Christof Henkel Phil Culliton
Chuandong Tang Piyush Mathur
D. Sculley Pranav Mohan Belhekar
Danial Sultanov Raghav Awasthi
Dariusz Kleczek Rhys Cook
Dave Harold Mbiazi Njanda Rob Mulla

Authors Diego Flores

Dmitri Kalinin
Samantha Lycett
Sanyam Bhutani
Harshit Mishra Shreya Mishra
Hoda Jalali Najafabadi Svetlana Nosova
Ivaxi Sheth Theo Flaus
Julia Elliott Trushant Kalyanpur
Karnika Kapoor Will Cukierski
Kobbie Manrique Xinxi Chen
Leonie Monigatti Yassine Motie
Lezhi Li Yuqi Liu
Lorresprz Yuxi Li
Mark McDonald Zhengping Zhou
Martin Henze
AI Report 2023 2 Maryam Babaei
TABLE OF CONTENTS
02
INTRODUCTION Text Data

● Foreword, D. Sculley (Kaggle CEO)

● Section Overview, Christof Henkel
● About the Kaggle AI Report, Phil Culliton
● "Contemporary Large Language Models LLMs",
Abir Eltaief
ESSAYS
● "Large Language Models: Reasoning ability", Théo
01 Flaus, Yassine Motie
Generative AI ● "Mini-Giants: “Small” Language Models",
Zhengping Zhou, Xinxi Chen, Yuxi Li, Lezhi Li
● Section Overview, Karnika Kapoor
● “2023 Kaggle AI Report - Generative AI”,
Trushant Kalyanpur 03
● "Understand, Generate and Transform the Image / Video Data
World", Yuqi Liu
● Section Overview, Rob Mulla
● "A Glimpse into the Realm of Generative AI",
● Advances in "AI Vision Models in the Last
Pranav Mohan Belhekar, Arya Gaikwad
Two Years", Dmitri Kalinin
● "Image and Video Data", Danial Sultanov

AI Report 2023 3
TABLE OF CONTENTS
06
ESSAYS [continued] AI Ethics
04
● Section Overview, Parul Pandey
Tabular / Time Series Data ● "Exploring the landscape of AI Ethics", Patrik
Joslin Kenfack, Meghana Bhange, Maryam
● Section Overview, Bojan Tunguz
Babaei, Ivaxi Sheth, Dave Harold Mbiazi Njanda
● "Learnings from the Typical Tabular Modelling
● "Developments in AI and Ethics in the past 2
Pipeline", Rhys Cook
years", Antong C.
● "AI Report: Time Series and Tabular Data",
● "Ethical AI is all we need!!", Shreya Mishra,
Chuandong Tang, Paulina Skorupska
Piyush Mathur, Raghav Awasthi, Anya Mathur,
● "Tabular Data in the Age of AI", Kobbie Manrique
Harshit Mishra
05
Kaggle Competitions

● Section Overview, Sanyam Bhutani

● "Towards Green AI", Leonie Monigatti
● "How to Win a Kaggle Competition", Dariusz Kleczek
● "Medical Imaging Competitions", Nghi Huynh

AI Report 2023 4
TABLE OF CONTENTS
ESSAYS [continued]

07
Other Topics

● Section Overview, Martin Henze

● "Optimization Algorithms in Deep Learning",
Svetlana Nosova
● "Applications of Artificial Intelligence and Machine
Learning Models within the Biosciences",
Samantha Lycett
● "Applying AI/ML to theoretical physics", Lorresprz
● “Kaggle AI Report: Medical Data”, Diego Flores
● "Graph Learning and Complex Networks", Hoda
Jalali Najafabadi, Ali Jalali

AI Report 2023 5
Introduction

AI Report 2023 6
 The world of AI has seen breathtaking progress over recent years, with rapid advances in
the capabilities of models as large as ChatGPT, Llama, and PaLM, and as small as those
that can fit on device or in a web browser. The advances of AI have not been confined
only to models: we have also seen an incredible spread of knowledge and expertise
across the globe, with AI experts participating in the field from every corner of the world
and every walk of life.

At Kaggle, we believe that this global community of AI and ML experts – now 15 million

Foreword members strong – is one of the most valuable open resources in the world today. Our
community works together to learn, share, compete, collaborate, stress test, and evaluate

D. Sculley what really works in AI and ML, and does so in a deeply rigorous fashion.

It is a great pleasure to welcome you to the 2023 Kaggle AI Report, created by our
community and selected from hundreds of submissions. Each paper within this report
gives a unique viewpoint on the most interesting or most important recent developments
in the field of AI and ML.

Enjoy the reading, and as always, our deepest thanks to the millions of members of the
Kaggle community!

Very best wishes,

D. Sculley
AI Report 2023 7
 The Kaggle AI Report is a collection of essays written and submitted by the
About the Kaggle community as part of a competition, broken down into seven sections that

Report we feel represent significant areas within the research and practice of modern ML.
The submissions were evaluated and edited by a member of our community with
Phil Culliton noteworthy expertise in their section’s area. Each expert selected the winner in
their section, as well as a number of honorable mentions.

AI Report 2023 8
 About the Report (Cont.), Phil Culliton

Sections
Section includes: ● Kaggle competitions: Kaggle is perhaps most famous
for its competitions, curated and led with care by our
● Generative AI: A frontier of machine learning that is team of experts, in partnership with hosts including
coming into new focus in the last few years, this area researchers, educators, and industry giants.
has combined the best of text and image research to Competitions present cutting-edge problems and
create a fundamental shift in the usability and utility of challenges that our diverse community solves in myriad
machine learning models. exciting ways.
● Text data: Natural language processing and statistical ● AI ethics: Humans in the world of machine learning
language modeling are the backbone of some of the have always struggled to make our discoveries fair,
most exciting recent advances in AI. unbiased, and equitable for other people.
● Image / video data: Image data was the foundation for ● Other: The above six sections are extensive, but
early advances in deep learning and the past decade is ultimately machine learning and artificial intelligence are
a testament to the ingenuity of researchers and oceans of possibility. We include a section for some
practitioners in this area, with ever-expanding datasets fascinating essays that don’t quite fit into any of the
and problem formulations met with fresh, new ideas. others.
● Tabular / time series data: Tabular data problems are
the most common type of problem to solve, and an area
that has led to incredible research and development –
some by our very own community members.
AI Report 2023 9
 About the Report (Cont.), Phil Culliton

The essays in our report were written as notebooks, a rich, multimedia communication form that can include text,
images, video, and even runnable code. Please be sure to click on the link for each essay to fully explore the
experience that the authors created.

Area Chairs

We created this report with the understanding that our community holds a sum of ability and knowledge far
greater than our small team of editors. We reached out to prominent members of our community with
backgrounds and proven skill in each of the report’s 7 topic areas to act as judges and expert editors. These
members of the community are our Area Chairs.

AI Report 2023 10
About the Report (Cont.), Phil Culliton

Sanyam Bhutani Christof Henkel

Sanyam Bhutani drinks chai and makes content for Christof is a Deep Learning Researcher at NVIDIA. He
the community at H2O.ai. When not drinking chai, you is particularly interested in novel deep learning
can find him hiking the Himalayas often with LLM architectures with respect to graphs, computer vision
papers. He is best known for chai, GPUs and and audio. His background is in mathematics and he
mountains. He is the host of the most popular Kaggle completed his PhD at the
podcast with top Kaggle Grandmaster interviews on Ludwig-Maximilians-University in Munich about
Chai Time Data Science. On the internet, he's known stochastic processes and financial markets. He started
for learning in public and "maximizing compute per Kaggling six years ago and after participating in more
cubic inch of ATX". than 70 competitions currently holds first rank in the
world-wide competition ranking.

AI Report 2023 11
About the Report (Cont.), Phil Culliton

Martin Henze Karnika Kapoor

Martin has a Ph.D. in astrophysics and an academic Karnika Kapoor is a data scientist with a background in
background in researching exploding stars in nearby mechanical engineering. She is skilled in machine
galaxies. He got into the fields of data science and learning and AI, and is currently employed as a senior
machine learning through Kaggle, where he became the data scientist where she focuses on growth
first ever Kaggle Notebooks Grandmaster and for a time optimization and logistics. She is actively engaged
held the number one spot in the notebooks rankings. with the Kaggle community and became a Kaggle
He has spoken at various conferences about his Notebooks Grandmaster in part due to her ability to
passion for effective data communication and transform data into well-communicated insights. With
storytelling, and he curated 100 episodes of the Hidden a passion for learning, Karnika stays AI-current,
Gems series on underrated Kaggle Notebooks (example offering a unique blend of technical prowess and
post and dataset). Martin is the Lead Data Scientist for data-driven acumen.
the market research company YipitData.

AI Report 2023 12
About the Report (Cont.), Phil Culliton

Rob Mulla Parul Pandey

Rob Mulla has over 10 years experience working with Parul Pandey has a background in electrical engineering
data, using his skills at companies in sectors including and currently works as a Principal Data Scientist at
pharmaceuticals, hospitality, energy, and sports sciences. H2O.ai. She is also a Kaggle Grandmaster in the
His journey with data has led him to become a 4x Kaggle notebooks category, where she composes compelling
Grandmaster, where he has had the chance to participate stories through the medium of notebooks. Her strength
alongside some of the best in the field. Outside of work, lies in analyzing data and eliciting useful insights from
Rob enjoys sharing his knowledge of Python and data them with the help of powerful visuals. Parul is one of
science on his YouTube channel, @robmulla, which has the co-authors of the Machine Learning for High-Risk
garnered a community of over 100,000 subscribers. Rob Applications textbook which focuses on the responsible
is an alumnus of Virginia Tech with a BS, Kansas State implementation of AI. Parul has written multiple articles
University with an MSE, and UC Berkeley where he focused on data science and she mentors, speaks, and
earned his Masters in information and data science. delivers workshops on topics related to responsible AI.

AI Report 2023 13
About the Report (Cont.), Phil Culliton

mRNA Open Vaccine challenge and has been part of

three distinct Nature publications (#1, #2, #3). Bojan
considers himself a machine learning generalist, and
has considerable experience working with a wide
variety of Machine Learning problems - image, NLP,
voice, tabular, etc. He is passionate about learning and
Bojan Tunguz exploring new applications of machine learning,
especially those that can have a significant impact on
Bojan is a Machine Learning Modeler at NVIDIA. Bojan
real world problems. He is currently working on the
holds a Ph.D. in theoretical physics,
application of machine learning in RNA
and has been working in machine learning and data
bioinformatics and in social media analysis. He is
science fields for eight years, where he has worked with
particularly interested in fundamental issues
real-world fintech problems. He is one of seven 4x Kaggle
of machine learning for tabular data.
Grandmasters, and is the first person to be ranked in the
Bojan is a voracious reader, he is passionate about
top 10 in all four Kaggle categories simultaneously. He has
tinkering with all sorts of tools and gadgets,
nine Kaggle competition gold medals, and with his team,
loves digital photography, and really enjoys hiking in
won the largest Kaggle competition at the time – the
the woods and biking.
Home Credit Classification challenge. He has worked as a
competition participant and organizer in the Human
Bojan is happily married, and lives with his wife and
Protein Classification challenge and the
three little boys in Florida.

AI Report 2023 14
Essays

AI Report 2023 15
 01
Generative AI

AI Report 2023 16
 Generative AI

Section Overview by Karnika Kapoor

Topic Summary Enhanced generative modeling algorithms, like diffusion
models and normalizing flows, coupled with refined
This section delves into the dynamic world of generative training methods, further elevate this transformation by
AI, highlighting its rapid progress, particularly over the leveraging larger datasets, more powerful hardware, and
last two years. It unveils core advancements and innovative applications like medical image synthesis and
transformative applications across various media forms. financial model crafting, the field is propelled to new
Generative AI focuses on creating new content such as heights.
images, text, and music, driven by breakthroughs in
generative adversarial networks (GANs) and large Despite being in its early stages, generative AI holds the
language models (LLMs). potential to revolutionize many fields, its versatility
showcased by applications as diverse as writing, music,
GANs, complex neural networks for creating realistic and data production. Looking ahead, generative AI
data, and LLMs, skilled in text and language generation promises even more captivating advancements,
have been central to recent advances in the ability of AI influencing various industries and warranting a
models to produce convincingly realistic outputs. comprehensive understanding of its ethics and risk
management.

AI Report 2023 17
 Generative AI

Trends & Predictions As generative AI becomes increasingly sophisticated, it's

crucial to consider the potential risks and ethical
The evolution of generative AI is a captivating and implications of its usage. These risks include creating
significant topic deserving in-depth exploration. This misinformation, perpetuating bias, vulnerability to
technology is rapidly transforming how we create and disclosing private data if misused in a training corpus,
interact with content, with the potential to revolutionize inappropriate or unattributed use of copyright or artistic
numerous industries. One noteworthy trend in intellectual property, etc. Continued research and
generative AI is its expanding use beyond content development of generative AI should be conducted in an
generation. For instance, AI-generated imagery is being ethical and responsible manner.
integrated in healthcare diagnostics and personalized
educational experiences.

AI Report 2023 18
 Generative AI

Overview of Essays Ethical considerations in AI, particularly related to

energy consumption, are also extensively discussed.
This collection of essays provides a concise history of
AI's evolution, followed by an in-depth exploration of Opinions differ on AI's impact on employment
current developments in generative AI. The consensus opportunities, with some embracing its transformative
among many essay authors is that the release of the potential and others expressing concerns about job
'Attention is All You Need' paper in 2017 marked a displacement. This duality mirrors historical patterns of
pivotal moment in the advancement of generative AI, technological shifts, where initial apprehensions
with transformer architectures serving as the foundation gradually transform into productive integration. The
for sophisticated models with billions to hundreds of overarching message emphasizes harnessing AI as a
billions of parameters (such as GPT-4). Architectural supportive tool rather than a disruptive force, aligning
complexity and rigorous research play a central role in with historical trends that steer initial concerns towards
enhancing generative AI's performance. productive applications.

Several essays delve into user-friendly generative AI

solutions, catering to individuals seeking streamlined
content creation.

AI Report 2023 19
 Generative AI Essay #1

“2023 Kaggle AI Report - Generative AI”

Award Winner
By Trushant Kalyanpur

Spanning 2021 to 2023, this essay charts the profound The essay carefully addresses ethical concerns tied to
evolution of generative AI, spotlighting strides in image AI's capabilities. Looking forward, it acknowledges
synthesis, language models, and audio generation. ongoing research, tools like LangChain and AutoGPT,
Noteworthy innovations like GPT4, DALL-E, and and the promise of innovative, ethically conscious AI
ChatGPT take center stage, propelling AI-generated integration.
content into new realms. This narrative journey
navigates pivotal years: 2021 witnesses DALL-E's In summary, this essay captures the dynamic trajectory
text-to-image feats and Github Copilot's code of generative AI's evolution, showcasing significant
suggestions, 2022 showcases Meta's contributions, achievements, addressing challenges, and envisioning a
ChatGPT's emergence, and 2023 highlights the rapid future where AI's potential is harnessed responsibly.
rise of multimodal AI with GPT4's unveiling.

Link to Notebook
AI Report 2023 20
 Generative AI Essay #2

“Understand, Generate and Transform the World”

Honorable Mention
By Yuqi Liu

This essay traces the evolution and growing influence of transformative structures such as GANs and Vision
generative AI across many domains. With an adept blend Transformers, offering a glimpse into the collaborative
of historical insight and technological exploration, the future of multimodal learning.
essay delves into the societal significance and intricate
mechanisms of generative AI. The essay candidly confronts challenges and ethical
considerations, casting a balanced light on the practical
From its historical roots to the accelerated growth fueled applications and limitations of generative AI. Ultimately,
by Deep Learning, the essay navigates pivotal models the essay comprehensively celebrates generative AI's
like DALL·E 2, highlighting the synergy between present impact while illuminating its promising trajectory
computational resources, dataset scale, and AI potential. into the future.
Bridging Computer Vision and Natural Language
Processing, the exploration showcases

Link to Notebook
AI Report 2023 21
 Generative AI Essay #3

“A Glimpse Into the Realm of Generative AI”

Honorable Mention
By Pranav Mohan Belhekar, Arya Gaikwad

This essay explores the innovative journey and paradigm implications of AI-generated content. The essay
shift that generative AI has inspired. From its navigates the challenges, particularly deepfakes, while
resemblance to an enchanted data-powered box that inviting us to witness a revolution that defines a new
generates patterns to its evolution from Boltzmann balance between human ingenuity and generative AI
Machines to Generative Adversarial Networks (GANs), capabilities.
generative AI's impact on image synthesis, text
generation, and more is discussed in detail. Recent
achievements, including the fusion of transformers and
GANs, showcase its collaborative potential in enhancing
human creativity. Amid these strides, ethical
considerations emerge, emphasizing impartial training
data and addressing the societal

Link to Notebook
AI Report 2023 22
 02
Text Data

AI Report 2023 23
 Text Data

Section Overview by Christof Henkel

Topic Summary
term-frequency based feature engineering combined
This section covers natural language processing (NLP) with non-neural-network-based machine learning
— understanding of natural language texts for methods. The rise of deep learning enabled solutions for
classification and regression tasks, as well as text larger datasets by learning word representations and
generation tasks such as summarization or translation. applying machine learning models to interpret them.
Insights can be gained from text data in a diverse Deep learning also became state of the art for some
number of languages including both common languages small datasets through transfer learning as only a small
and programming languages. The methods can even be amount of data was needed to fine-tune a pre-trained
applied to data tables that are presented in text format. model to become relevant to new applications.

When the entire internet is used as a training corpus for Similar to the benefits of scaling neural network
AI models, knowledge transfer using deep learning architectures, transformer-based models can be trained
becomes an incredibly powerful technique – and the on trillions of tokens and the resulting pre-trained model
major advancements of the past few years are a weights can either be used directly or fine-tuned to be
testament to that. state-of-the-art for a large and diverse number of tasks.
Large language models (LLMs) are unsurprisingly the
Early approaches to small datasets often included primary focus of attention for many or most AI
researchers today.
AI Report 2023 24
 Text Data

Trends and Predictions

This is evident in the rise of increasingly complex
An observable trend in text-based Kaggle competitions competitions hosted on Kaggle, ranging from analyzing
is that instead of training a model from scratch, Jupyter notebooks, to understanding and grading
competitors tend to fine-tune publicly available models student essays, to problems where text data intersects
which have been pre-trained on a similar language or with other data types like images or audio. With LLMs
task. With text-based models becoming more and more becoming more and more powerful, they will not only be
accurate, simply because they are pre-trained on bigger the basis for solving text-based problems but also will
datasets and hence carry more knowledge to transfer be used to generate or augment training data. They
via fine-tuning, they often surpass human-level might also be used to automatically judge generative
performance. models on complex tasks like abstract reasoning.

An interesting result of stronger text modeling is that

they enable the ability to solve more challenging use
cases.

AI Report 2023 25
 Text Data

Overview of Essays
Interestingly, the top essays cover different aspects of
The top essays in this section provide a diverse and LLMs. The winning essay discusses the emergence of
high-quality overview of contemporary LLMs. They contemporary large language models as well as the
notably skip over foundational NLP work that preceded most recent methods and techniques that make them
LLMs, and similarly gave light treatment to the building possible. The other essays discuss fine-tuning and
blocks that enable their breakthrough performance on applicability of LLMs to small datasets, and the
benchmarks and interactive use cases. Contributions capabilities of LLMs with respect to the ability to
focused primarily on the scientific merits of models, perform reasoning. As such the top essays cover diverse
largely sidelining discussion of practicalities like and interesting topics of LLMs.
hardware and the productionization of models. The
focus on LLMs is not surprising, given they are currently
in the center of society's attention when it comes to the
applicability of artificial intelligence solving human
problems.

AI Report 2023 26
 Text Data Essay #1

“Contemporary Large Language Models LLMs”

Award Winner The essay discusses core concepts and features of
By Abir Eltaief LLMs. It examines the efficient utilization of extensive
training data and parameters by pretrained LLMs. It
further explores prompt engineering which spans from
This essay explores the rise of contemporary language basic techniques like zero-shot, one-shot, and few-shot
models, delving into the latest methodologies enabling prompting to more advanced methods including Chain
them, and reflecting on the insights gained by the of Thought (CoT) and Reasoning & Acting (ReAct). The
machine learning community in the past couple of years. focus then shifts to enhancing LLMs through human
feedback, exploring reinforcement learning-based
It also draws from the author's individual odyssey to fine-tuning (RLHF) for targeted optimization. The essay
learn the captivating realm of LLMs. The journey also investigates the augmentation of LLMs for app
encompasses initial interactions with diverse GPT-based development, highlighting Retrieval Augmented
chatbots, the creation of applications fueled by LLMs, Generation (RAG) as a framework to equip LLMs with
and the reading of remarkable research papers. external knowledge. The content concludes by providing
referenced sources.

Link to Notebook
AI Report 2023 27
 Text Data Essay #2

“Large Language Models: Reasoning Ability”

Honorable Mention
By Théo Flaus, Yassine Motie

This essay focuses on insights gained over the past two By analyzing related papers and their findings, they
years of working and experimenting with LLM reasoning. provide a comprehensive overview of the architecture
After giving an overview of different types of reasoning, and progress of LLM reasoning. They put special
which are important to understand in tackling reasoning emphasis on highlighting the lessons learned and the
challenges, the authors explore the advancements made way forward in ensuring ethically safe and reliable AI
through Chain of Thought prompting, Tree of Thought systems.
frameworks, linguistic feedback reinforcement,
interleaved reasoning and action, and handling complex
mathematical reasoning.

Link to Notebook
AI Report 2023 28
 Text Data Essay #3

“Mini-Giants: ‘Small’ Language Models”

Honorable Mention
By Zhengping Zhou, Lezhi Li, Xinxi Chen, Yuxi Li

With giant LLMs becoming expensive to train and presents a comparative study of small language models,
prohibitively large to finetune for individuals or small and a brief discussion of evaluation methods. The
companies, small language models are flourishing and authors discuss the application scenarios where small
becoming more and more competent. The authors call language models are most needed in the real world, and
them "mini-giants" and argue a win-win for the open conclude with discussion and outlook.
source community by focusing on small language
models.

This essay presents a brief yet rich background,

discusses how to attain small language models,

Link to Notebook
AI Report 2023 29
 03
Image /
Video Data

AI Report 2023 30
 Image / Video Data

Section Overview by Rob Mulla

Topic Summary

This section explores some of the latest advancements In addition to discussing model architectures, this
in computer vision, particularly as it relates to the use section covers standard practices used in
of image and video data. While the field of computer computer vision today like preprocessing and
vision dates back to the 1960s, its evolution has been augmentation techniques. Additionally, this section
especially exciting over the last few decades. explores the practical applications of these
Specifically, over the past two years, there have been technologies across industries like healthcare (for
significant advancements not only in traditional medical imaging), agriculture (for crop monitoring),
computer vision tasks like classification and object and the automotive sector (for self-driving cars). It
detection, but also in emerging areas such as Vision also covers some of the limitations that computer
Transformers (ViT) and few-shot learning. vision still faces.

AI Report 2023 31
 Image / Video Data

Trends & Predictions and adapted to solve specific tasks. Looking ahead, I
anticipate that within the next five years, tasks such as
Computer vision can be traced back to the 1950s and segmentation, object detection, and image classification
1960s, when researchers started the development of will continue to be addressed by generalized large-scale
algorithms for detecting edges and patterns in images. models that can be fine-tuned for specific challenges,
While improvements in these areas continue, new mirroring what we’ve seen occur with large language
challenges like object detection, self-supervised models. We will also see the intersection of computer
learning, and knowledge reasoning are being vision and generative AI continue to progress in areas like
addressed with innovative model architectures and augmented reality, deep fakes, and AI-enhanced photo
training techniques. Particularly, video data is and video editing. These advancements inevitably bring
witnessing developments in multi-object tracking, with them ethical and philosophical considerations that
action recognition, and spatiotemporal reasoning. A require our attention.
noticeable area of active research in the field revolves
around generalization and transformer-based It’s also important to note some of the limitations that
architectures. research has yet to overcome. Some of these areas
include the limitation of multi-modal models that
The popularity of models like Segment Anything Model incorporate image and video, as well as vision model’s
(SAM) and YOLO (You Only Look Once) showcase how ability to perform in uncontrolled environments, like
generalized, open source models can be leveraged self-driving cars on new roads.

AI Report 2023 32
 Image / Video Data

Overview of Essays Lastly, Dron Bespilotnik's essay “Image and Video Data |
Kaggle AI Report'' covers some of the practical areas of
The essays chosen for this section cover the working with image and video data, including
significant research advancements in image and video preprocessing techniques, and discussing how data
understanding in recent years. They reference augmentation enhances the training performance of
breakthrough papers that have reshaped our vision models. He describes the role of computer vision
understanding of machine learning and the types of in tasks like image classification, segmentation, and
tasks that computer vision is able to solve. question answering.

Dmitri Kalinin’s essay, "Advances in AI Vision Models in

the Last Two Years," is a comprehensive summary of
recent advances in computer vision. It covers the
recent progress in semantic segmentation, vision
transformers, and continual learning methodologies,
among others. It serves as a great resource for anyone
desiring a deeper understanding of these subjects.

AI Report 2023 33
 Image / Video Data Essay #1

“Advances in AI Vision Models in the Last Two Years”

Few- and Zero-shot learning involves leveraging large language
Award Winner models with methods like label-free parameter tuning and
By Dmitri Kalinin feature extraction adapters. The Contrastive Language–Image
Pre-training (CLIP) architecture has used these approaches to
Dmitri's essay summarizes the most recent advancements in set new benchmarks in various tasks.
computer vision models, spotlighting six pivotal areas: Semantic
Segmentation, Vision Transformers, Few- and Zero-shot Generalizing computer vision models remains challenging, as
Learning, Generalization of Computer Vision Models, Continual evidenced by Human-Object Interaction (HOI) detection models
Learning, and Human-Assisted AI in Computer Vision. While not underperforming on new datasets; research is being made in
all of these areas are necessarily new, Dmitri's essay emphasizes increasing data diversity and integrating natural language
the latest studies, showing the newest evolution in computer resources. Additionally, to address computational intensity in
vision. Online Continual Learning (OCL), methods like the Summarizing
Stream Data (SSD) offer efficient memory usage and outperform
Semantic Segmentation models are highlighted for their role in prior models in benchmark datasets.
both autonomous driving and medical imaging sectors, and an
emphasis is put on the emergence of novel models, unique loss In addition to these advancements, Human-in-the-Loop
functions, and the surge in weakly supervised strategies for large techniques in computer vision are discussed. These methods
image dataset processing. Vision Transformers have emerged as integrate human expertise directly into the AI model's learning
a novel approach, providing breakthroughs in efficiency through process, enhancing generalization, explainability, and causal
image decomposition. understanding.
Link to Notebook
AI Report 2023 34
 Image / Video Data Essay #2

“Image and Video Data | Kaggle AI Report”

Honorable Mention
By Danial Sultanov

Dron’s report highlights the growing trend in image and video Next, the evolution of image classification and segmentation
data usage. Specifically, he points to the increase in papers models are covered, highlighting the rise of architectures like
published in the field of computer vision and how the number of Visual Transformers and ConvNeXt, as well as their use with
papers released has continued to increase each year following natural language processing for visual comprehension. An
the groundbreaking AlexNet publishing in 2012. The report then exploration of top papers from 2021 and 2022 to the Conference
covers 5 areas of computer vision: Data Preprocessing, computer on Computer Vision and Pattern Recognition (CVPR) conference
vision applications, an analysis of CVPR conference submissions, further shows that transformers and ViT topics are some of the
CV use cases, and future perspectives for computer vision. hottest topics in the field.

Data preprocessing and augmentation are discussed as essential In the final sections, the report discusses real-life computer
components of computer vision pipelines. The use of a vision applications and its foreseeable challenges.
combination of real and synthetic data sources have been shown
to enhance image and video processing in competitions and
research.

Link to Notebook
AI Report 2023 35
 04
Tabular / Time
Series Data

AI Report 2023 36
 Tabular / Time Series Data

Section Overview by Bojan Tunguz

Topic Summary
The temporal nature of those data points becomes a
major underlying feature of time-series datasets,
Tabular data, in the form of transactional data and
requiring special considerations in analysis and
records of exchange and trade, has existed since the
modeling.
dawn of writing. It may even precede written language.
In most organizations, it is the most commonly used
Tabular data, and to much lesser extent time-series
form of data. There is no definitive measure, but it is
data, has proven largely impervious to the deep learning
estimated that between 50% and 90% of practicing data
revolution. Non-neural-network-based ML techniques
scientists use tabular data as their primary type of data
and tools are still widely used and have stood the test of
in their professional setting.
time. Nonetheless, there have been some interesting
recent developments on that front as well. This remains
Time series data is, in many respects, similar to tabular
a kind of data where a wide variety of tools and
data. It is often used to encode the same kinds of
techniques are relevant, and there exists tremendous
transactions as non-temporal tabular data, with one
potential for further research and improvement.
important distinction: inclusion of temporal information.

AI Report 2023 37
 Tabular / Time Series Data

Trends & Predictions The dominance of gradient boosted trees has, until
recently, not been substantially covered in the ML
There are three main trends with machine learning for research literature, but over the past few years there
tabular data: have been more attempts to understand this
1. Need for unique approaches for every phenomenon and evaluate alternative approaches. In
dataset/problem. particular, there have been many more attempts to use
2. Outsized importance of data munging and feature neural networks with tabular data, but those attempts –
engineering. even when successful – come at the added expense of
3. Continuing dominance of gradient boosted trees computational complexity. Most of that research has
as the algorithm of choice. had minimal effect on applied ML modeling for tabular
The first two trends have been well known in the Kaggle data.
circles ever since the platform launched, and the last
since XGBoost was first introduced on Kaggle in 2014 It is very likely that, in the upcoming years, we will see
(you can find a collection of winning Kaggle solutions even more research on neural networks for tabular data,
that use XGBoost here). Even though there have been as well as new innovations for gradient boosted trees. A
comparatively few featured tabular data competitions on very promising new area would be the application of
Kaggle over the past couple of years, the ones that were generative AI to automated feature engineering and
held there reinforced these trends. AutoML for tabular data in general.

AI Report 2023 38
 Tabular / Time Series Data

These essays take a closer look at the special data wrangling

Overview of Essays
and feature engineering techniques that have proven crucial
Kaggle has its roots in hosting competitions for tabular and for extracting the most amount of information from these
time series data. However, over the years, demand has datasets. These in-depth looks help to highlight how
shifted to other areas of ML, and the primary venue for these idiosyncratic and unique each one of those datasets and
“classic” ML problems have become the monthly Tabular problems are.
Playground Series competitions. These competitions rely on
synthetic data, and are constructed in such a way as to The winning essays also incorporate information from recent
highlight similar “real” problems that have been featured on research literature on Tabular Data ML. The main finding of
Kaggle in the past. Overall, these competitions provide a good such literature – continuing dominance of gradient boosted
testing ground for various tabular and time series ML trees for ML modeling – has been in line with the findings
approaches, which has especially been emphasized in the from Kaggle competitions. However, as has been noted in the
winning essay in this collection. winning essay in this series, there has been very little work
done on exploration of feature engineering in the research
The winning essays also incorporate information about the literature, even though this is arguably the most valuable part
few "featured" tabular competitions that have taken place of ML modeling for tabular data. This would be a very
over the past couple of years (here, the distinction is that productive area of future research.
"featured" competitions reward large cash prizes while
"playground" competitions do not).

AI Report 2023 39
 Tabular / Time Series Data Essay #1

“Learnings From the Typical Tabular Modelling Pipeline”

Award Winner
By Rhys Cook It methodically quantifies various models used, feature
engineering techniques, and breadth of writeups for these
Quick, accurate and expert handling of tabular data is key in competitions. The learnings are in line with what most
machine learning. This essay examines the recent tabular long-term Kaggle competitors have observed in the past, and
data modeling pipelines of successful Kagglers by exploring have reinforced the notion that these “playground”
individual discussion posts and aggregated Kaggle metadata. competitions are indeed representative enough of the kind of
Its goal is to extract key learnings from these recent, high work that is required for these sorts of ML problems in the
performing tabular data solutions and contrast them to the real world.
latest developments in the literature.
Among the main learnings in this essay are that: (1) feature
This essay is noteworthy for its detailed and systematic engineering is one of the most important aspects of the ML
analysis of all the Tabular Playground Series competitions modeling pipeline for tabular data; (2) gradient boosted trees
(2023 was the third year for this series). dominate as the go-to algorithms for tackling these problems;
and (3) ensembling methods work extremely well for
increasing the predictive power of tabular-data models.

Link to Notebook
AI Report 2023 40
 Tabular / Time Series Data Essay #2

“AI Report: Time Series and Tabular Data”

Honorable Mention
such as the American Express Default Prediction, G-Research Crypto
By Chuandong Tang, Paulina Skorupska Forecasting, Tokyo Stock Exchange Prediction, Optiver Realized
Volatility Prediction, and Ubiquant Market Prediction competitions.
Kaggle has long served as a pivotal platform for fostering Additionally, the essay extends its scope to encompass a range of
advancements in machine learning, hosting a rich array of competitions that, while not exclusively tabular in nature, manifest a
approximately 60 competitions focused on a range of data types rich tapestry of techniques and approaches that are emblematic of
over the past two years alone. This report narrows its scope to the tabular data challenges.
intricate application of state-of-the-art ML models in time series and
tabular data, primarily within the framework of supervised learning. Moreover, this essay allocates significant space to scrutinizing the
Drawing upon diverse case studies, the report elucidates innovative impact of diverse neural network architectures and methodologies in
feature engineering and modeling techniques that have proven the contemporary landscape of Kaggle's tabular competitions. It delves
effective across various sectors, from natural sciences to finance. into the nuanced applications and performance implications of
convolutional, recurrent, and transformer-based neural networks,
The strength of this essay lies in its meticulous exploration of a among others, in shaping the outcomes of these cutting-edge
curated selection of recent "featured" Kaggle competitions that contests. This segment serves not only as an intellectual exposition but
focus predominantly on tabular data series. The essay offers a also as a practical guide for leveraging advanced neural network
comprehensive breakdown of highly effective strategies and paradigms in data science challenges that involve tabular data sets.
intricate feature engineering methodologies employed in contests

Link to Notebook
AI Report 2023 41
 Tabular / Time Series Data Essay #3

“Tabular Data in the Age of AI”

Honorable Mention
By Kobbie Manrique

The purpose of this report is to provide an overview of the Even though the essay doesn’t cover experience with Kaggle
recent advancements in AI techniques for tabular data. Our competitions and tabular datasets, it provides an interesting
goal is to provide valuable insights to the Kaggle data science and unique overview of some of the most advanced recent
community and inspire future innovations. By grasping the research developments in this area. The essay is particularly
latest AI techniques and their practical applications in tabular noteworthy for its overviews of AutoML and explainable AI,
data analysis, data professionals can remain at the forefront both of which are very important for many everyday
of this rapidly evolving field. applications.

This essay is noteworthy for providing an excellent historical,

applied, and conceptual background on tabular data. It
provided rich context for dealing with such problems, and
their importance in the development of computing in
particular.

Link to Notebook
AI Report 2023 42
 05
Kaggle
Competitions

AI Report 2023 43
 Kaggle Competitions

Section Overview By Sanyam Bhutani

Topic Summary The real value of competitions emerges over time, where
one can observe winning solutions from older competitions
Kaggle competitions are the most meritocratic avenue for becoming new baselines. Eventually the “tricks” of winners
enthusiasts and veterans alike to establish their AI become standard practice in the next competitions: ideas
credentials. The leaderboard is based on an objective like pseudo labeling, seed averaging, hill climbing - to name
measurement to provide a score for each submission and just a few - were “tricks” that went from being explicitly
therefore does not lie. It is regarded by many as one of the mentioned in winning solutions to now frequently
hardest and truest challenges in data science. appearing in many solutions.

This section aims to cover the developments and

observations from Kaggle competitions from the last 2
years. Competitions have always been hailed as a go-to
arena for testing skills of competitors and evaluating ideas,
papers and frameworks. Over the years, XGBoost, H2O-3,
PyTorch, albumentations, and fastai have emerged as the
trusted frameworks of the community.

AI Report 2023 44
 Kaggle Competitions

Trends & Predictions

Although heavily context dependent, some model
There are a number of trends in the types of Kaggle
architectures have been thoroughly tested in domains:
competitions and winning approaches which we can
for example, the DeBERTa family, EfficientNet, and ViT
expect to continue in the coming years.
have emerged as strong candidates for a wide variety of
tasks. Even though these methods are computationally
Recently, we have seen some repeat hosts, such as
heavy, most recent competitions have attempted to
RSNA, Learning Agency, BirdCLEF. These competition
encourage submission of small and efficient models by
series provide an opportunity for participants to build
having an inference constraint limit, and some with a
on prior approaches or become more deeply
new efficiency track, where your models are allowed
connected to similar competition themes by hosts with
only to make predictions on small CPU servers.
compelling machine learning use cases. Kaggle itself
has also introduced some new, more innovative
As competitions get fiercer every year, the Kaggle team
variations of competitions, for example the COVID
has also put extra care into beginner friendly
competitions which ran in sprints, and the Stable
competitions, adding incentives for sharing ideas and
Diffusion competition which focused on reversing a
new competition types. It’s clear we can expect more of
generative text-to-image model.
this; Kaggle is the home of data science for a reason.

AI Report 2023 45
 Kaggle Competitions

Overview of Essays

The Top essays of this category are surprisingly diverse.

Instead of just focussing on winning tricks and bits, they
also share the bigger picture. These essays provide an
insight into emerging efficient tips and winning tricks in
Kaggle competitions.

The essays from this section can alternatively be looked

at as “best practice” essays, and, depending on the
topic of your interest, you can find battle tested ideas
here.

AI Report 2023 46
 Kaggle Competitions Essay #1

“Towards Green AI”

Award Winner
By Leonie Monigatti

Kaggle is usually known as an “ensembling playground” performance and inference time to diminish the carbon
to the outside world: Kaggle competitors often combine footprint of deep learning models.
a variety of methods and models in order to increase
their score without needing to balance the The report "Towards Green AI" shines a light on the
computational costs of their solutions. To counter this pivotal challenge of our times: crafting deep learning
trend, Kaggle has been awarding special prizes to models that deliver on performance without exacting a
solutions that are both accurate and performant. This heavy carbon toll. Driven by Kaggle's visionary
report shares learnings from Kaggle competitions Efficiency Prize, the study delves deep into the heart of
concerning efficient models and efficient modeling techniques like pruning, low-rank factorization, and
practices, in particular. quantization, scrutinizing their true potential in the
real-world AI landscape.
In this essay the author tells the story of the significance
of striking a balance between predictive
Link to Notebook
AI Report 2023 47
 Kaggle Competitions Essay #2

“How to Win a Kaggle Competition”

Honorable Mention
By Dariusz Kleczek

This essay dives deep into the minds of Kaggle winners, From the nuances of data augmentation to the might of
and the author uses LLMs to systematically extract and gradient boosted decision trees, this report paints a
analyze structured data from a myriad of Kaggle comprehensive picture of what it takes to clinch that
competition writeups (dedicated discussion posts where coveted top spot. A brilliant blend of data-driven
winners of Kaggle competitions describe their solutions). insights and personal experiences, this essay is an
essential guide for anyone interested in climbing the
It distills wisdom and ideas from the most coveted leaderboard.
methods and strategies.

Link to Notebook
AI Report 2023 48
 Kaggle Competitions Essay #3

“Kaggle AI Report: Medical Imaging Competitions”

Honorable Mention
By Nghi Huynh

Medical competitions have historically been some of the

most popular on Kaggle. These competitions involve
different imaging modalities, such as MRI, CT scans, and
X-rays. This report offers an in-depth analysis of Kaggle
competitions centered around medical imaging, aiming
to unearth the prevailing methodologies and
architectures employed by the machine learning
community. The study meticulously dissects
competitions into specific categories, including Object
Detection, Classification, and Segmentation.
Furthermore, it delves into the nuances of the most
favored Deep Learning models.

Link to Notebook
AI Report 2023 49
 06
AI Ethics

AI Report 2023 50
 AI Ethics

Section Overview by Parul Pandey

Topic Summary Machine learning (ML), a branch of artificial intelligence

(AI), has become a crucial tool in both corporate and
The section is dedicated to the continued study of AI governmental sectors worldwide, influencing areas from
ethics. It encompasses not only a detailed discussion of consumer products to security measures. Its utilization
ethical principles in AI, but also an exploration of the tools spans high-risk decisions in areas such as employment,
and strategies that can be employed for monitoring, parole, lending, security, and other high-impact realms
understanding, and mitigating risks in high-stakes across global economies and governments. With the recent
machine-learning applications. We sample the complex surge in generative AI, the stakes are now even higher,
debates surrounding ethical considerations in AI, amplifying both opportunities and challenges. Its
broadening our focus to include the means by which widespread adoption over the past years testifies to its
potential risks in critical machine-learning scenarios can be potential; however, it also underscores the inherent risks to
managed. users, organizations, and the public at large. Addressing
these challenges is essential for both organizations and the
general populace to genuinely harness the potential of this
innovative technology.

AI Report 2023 51
 AI Ethics
Trends & Predictions Several emerging trends in this field warrant attention. One
such trend is the continuous auditing of AI systems,
The study of AI ethics is not merely an academic endeavor especially those in critical sectors. Such systems might
but a societal imperative. As AI continues to shape the undergo ethical audits to ensure they adhere to
world, ensuring its ethical deployment becomes paramount established guidelines. It’s anticipated that future AI
to harness its benefits while safeguarding collective values. systems will adopt an ‘ethics-by-design’ approach,
A growing consensus has emerged that ethics cannot be emphasizing ethical considerations from the onset. For
an afterthought; instead, they must be an integral part of AI instance, Meta decided to approach the release of
system design. However, there’s a palpable need for LLaMA-2 with strong focus on responsibility, providing
globally accepted standards on AI ethics. Some resources and best practices for responsible development
authoritative guidance is appearing to emerge, like the of products powered by large language models.
ISO’s technical standards for AI and the NIST (National
Institute for Standards and Technology) AI Risk Moreover, as AI becomes more ingrained in daily life,
Management Framework. This framework highlights key there’s an expected increase in public involvement. People
characteristics of trustworthy AI systems, including are likely to have a more pronounced role in AI’s ethical
validity, reliability, safety, security, resiliency, transparency, considerations, leading to more public forums,
accountability, explainability, interpretability, bias consultations, and potential referendums on significant AI
management, and enhanced privacy. To implement these deployments. In essence, the landscape of AI adoption and
characteristics, it offers actionable guidance for risk management is continuously changing, making it
organizations in four areas: map, measure, manage, and crucial to stay vigilant and proactive when addressing the
govern. ethical ramifications of these technologies.

AI Report 2023 52
 AI Ethics
Overview of Essays

The AI Ethics section garnered numerous insightful Others focus on more specific areas. For example, some
submissions, highlighting how rapidly this area is discuss the latest advancements in AI, like the exciting
evolving. The essays received covered a diverse set of world of generative AI, and the ethical questions these
topics, reflecting the depth and breadth of this new technologies bring. Others take a closer look at
important discipline. research in the AI world, analyzing the ethical
considerations in AI studies and publications.

Some essays covered the big picture of AI ethics, But despite their different angles, all the essays
discussing the broader challenges and opportunities it converge on a shared sentiment: AI holds immense
presents. They touch upon how we need to shape AI in a promise for our future, but we need to approach its
way that is responsible and considers everyone in development and use with care, thought, and a strong
society, from ensuring fairness in its decisions to sense of responsibility.
thinking about its impact on the environment.

AI Report 2023 53
 AI Ethics Essay #1

“Exploring the Landscape of AI Ethics”

Award Winner
By Patrik Joslin Kenfack, Meghana Bhange, Maryam Babaei, Ivaxi Sheth, Dave Harold Mbiazi Njanda

This winning essay delves into the critical urgency of Notably, the essay offers a dual lens, analyzing each
AI’s ethical implications in our ever-evolving digital era. guideline from societal impacts to technical
The essay talks about central principles, crucial for advancements. Each guideline is dissected not only for
instilling trust in AI, including privacy, data protection, its societal importance and value, but also for the
transparency, explainability, fairness, accountability, technical innovations that aid its implementation. In
safety, robustness, and even environmental doing so, the essay bridges the perceived gap between
considerations. This essay offers a comprehensive ethical principles and actionable technological solutions.
exploration of each principle, presenting contemporary Through this comprehensive approach, the essay makes
efforts and strategies to seamlessly integrate them a compelling case for these principles as the foundation
within AI’s lifecycle. for fostering genuine trust amongst all stakeholders
throughout the AI system lifecycle.

Link to Notebook
AI Report 2023 54
 AI Ethics Essay #2

“Developments in AI and Ethics in the Past 2 Years”

Honorable Mention
By Antong C.

The essay covers the evolution of AI ethics over the past Designed as an introductory piece rather than a deep
two pivotal years, highlighting enduring challenges and dive, the essay seeks to amplify awareness of AI ethics
the continuous efforts to address them. During this and champions a future where AI is more inclusive,
period, the AI domain has witnessed significant growth beneficial, and ethically grounded.
and heightened interest in its capabilities. This rapid
advancement, however, has accentuated the critical
need for robust AI ethics.

Emphasizing regulatory facets over philosophical ones,

the essay provides an overview of AI ethics, delves into
its core principles, pinpoints application gaps, and
suggests potential remedies.

Link to Notebook
AI Report 2023 55
 AI Ethics Essay #3

“Ethical AI Is All We Need!!”

Honorable Mention
By Shreya Mishra, Piyush Mathur, Raghav Awasthi, Anya Mathur, Harshit Mishra

In this essay, the authors underscore the essentiality of examples, the authors emphasize the tangible
ethics for building trust in AI systems and fostering implications of AI systems, aiming to spotlight real-world
sustainable advancement in AI research. The paper sets ethical challenges and considerations.
out to investigate and assess the ethical dimensions
embedded in research publications, focusing specifically
on AI articles from 2007 to 2023.

Leveraging informative data visualizations, the authors

probe into four principal dimensions: “Contextual Topics
Learning,” “Development Maturity Learning,” “Gender
Representation Learning,” and “Sentiments Learning” in
the context of AI publications. Using illustrative

Link to Notebook
AI Report 2023 56
 07
Other Topics

AI Report 2023 57
 Other Topics

Section Overview by Martin Henze

Topic Summary With one major exception, the topics of the essays have
little in common with one another. Participants analyze
The rapid growth that the field of Machine Learning has such diverse topics as optimization algorithms, graph
been experiencing over the recent years has led to an networks, theoretical physics, robotics, healthcare, the
explosion of tools, techniques, and applications. In our future of work, data-centric AI, mathematical research,
final category - simply called “Other” - we cover those autonomous cars, and many more. Together, they offer a
aspects of the ML landscape that may not fit neatly into panoramic snapshot of the wide-ranging influence of
any of the previous, more specific categories. This long Machine Learning at this pivotal moment in time.
tail of topics is reflected in a wide variety of topics
chosen by the essayists. In turn, the multitude of
fascinating subjects showcases the exceptional range
and diversity of expertise within the Kaggle community.

AI Report 2023 58
 Other Topics

Trends & Predictions including those focused on COVID-19. As ML-driven

tools are becoming more mature and more training data
Participants notably focused on the increasing is being built through real-world experience, the medical
importance of ML algorithms in the medical and and healthcare fields will likely see significant efficiency
healthcare fields. gains through automation in the near future.

From humble beginnings of applying computer vision Another exciting trend are multi-modal ML applications
tools in assisting with diagnosis in radiology or powered by growing repositories of pretrained models.
tomography, the power of ML to save lives and cure This is relevant for generative AI applications utilizing
diseases is being increasingly leveraged by medical agent systems, but also for tailored projects that draw
professionals and researchers. Modern applications on diverse data sources. In the coming years, models
include, among others, genetic sequencing (with NLP and applications that can derive insights from
techniques), robotic surgery, medical teaching, drug multi-sensory inputs in a human-like manner will likely
discovery, and of course the extremely relevant area of play an increasingly important role.
vaccine research. This trend is also reflected in a large
number of healthcare-related Kaggle competitions in
recent years,

AI Report 2023 59
 Other Topics

Overview of Essays research area of optimizers lies at the foundations of

learning methodologies. Thinking in graph networks can
This section will present a collection of the best and provide fresh perspectives on challenging problems in
most interesting essays, including the winning many areas. And, the areas of medical research and
contribution. They will cover the subjects of optimization theoretical physics illustrate the increasing impact of ML
algorithms, graph networks, theoretical physics and techniques on topics ranging from improving our daily
string theory, as well as the healthcare sector. lives to deepening our understanding of the fundamental
laws of nature. It is a golden age of ML progress, and
the following essays contain many nuggets of valuable
Any anthology in such a vibrant and fast-moving field
insights for the curious reader.
can only aspire to offer a tasting menu of some of the
delicacies that the research and praxis have to offer.
This selection of essays represents the various ways in
which advancements in ML transcend the boundaries
between the traditional categories in the field and exert
a growing interdisciplinary influence. The evergreen

AI Report 2023 60
 Other Topics Essay #1

“Optimization Algorithms in Deep Learning”

Award Winner
By Svetlana Nosova
the Adaptive Moment Estimation that Adam provides allows
This winning essay presents a well-structured and for more efficient learning rates and faster convergence. But
exceptionally accessible overview of one of the key progress is ongoing, and the essay concludes by highlighting
ingredients of modern ML: optimizers. The author eases the two recent developments: the versatile Momentum Models
audience into the report by providing great background (MoMo) approach alleviates the need for learning rate
information on the fundamental technique of gradient schedulers. In contrast, the Scalable Stochastic
descent, along with its stochastic and mini-batch variations. Second-Order Optimizer (Sophia) is considering second
Gradient descent and backpropagation are at the heart of the derivatives of the complex objective landscape in which we
training process for all neural network models. Gradient are aiming to find the minimum.
descent also powers the boosted tree ensembles, which
remain the gold standard for tabular data challenges. This essay expertly optimizes the balance between
beginner-friendly descriptions and technical details. Its
Through a concise description of the important language is clear and its narrative is engaging. It leaves the
momentum-based and adaptive methods, the author draws reader well prepared to dive into the treasure trove of
an elegant arc to what is arguably the most popular optimizer references to learn more.
at present: Adam. The default choice in many ML libraries,
Link to Notebook
AI Report 2023 61
 Other Topics Essay #2

“Applications of Artificial Intelligence and Machine Learning

Models Within the Biosciences”
Honorable Mention
By Samantha Lycett early-stage time-series forecasting challenges for infection
numbers, a medical imaging competition to detect COVID-19
abnormalities in chest radiographs, and an urgent mRNA
This concise article studies the various ways in which ML
Vaccine Degradation Prediction to help bring a vaccine to
approaches were used in the response to the COVID-19
mass production.
pandemic. Against a backdrop of describing the healthcare
and bioscience responses to the virus, the author analyzes
The author then links those events to the larger picture in
the relevant ML contributions through the examples of Kaggle
biosciences and specifically to the novel methodology of
datasets and competitions.
using language models for biological sequence data, where
molecules act like letters and structures like proteins
Especially during the early pandemic years, the Kaggle
correspond to sentences. The success of the AlphaFold
community was a microcosm of the challenges that the ML
project for 3D protein modeling is a key example for the
field was facing under considerable time pressure. Institutions
successful synergy of ML and bioscience research.
and community members published large volumes of
COVID-19 data on Kaggle. This led to fine-tuned NLP models
tailored to COVID-19 literature, which are now finding
increasing adoption by researchers. Link to Notebook
AI Report 2023 62
 Other Topics Essay #3

“Applying AI/ML to Theoretical Physics”

Honorable Mention
By Lorresprz
hidden away (aka compactified), but their geometries (aka
manifolds) affect those physical particles that we can
This work manages to describe recent challenges in the observe. Both supervised and reinforcement learning can help
complex field of string theory in a way that is remarkably to filter through a myriad of possible manifolds to discover
accessible to ML practitioners. String theory, with its viable candidates for geometries that are consistent with
manifolds and compactified dimensions, is arguably one of observational results.
the more complex areas in a complex field. But, its aspirations
to construct a unified “theory of everything” make it a leading As the author writes, at present more and more theoretical
contender for rewriting our understanding of physics and of physicists are incorporating ML-based approaches into their
the entire universe. In the already decades-long endeavor to research projects of increasingly diverse natures. Similar to a
study string theory, ML has only recently arrived at the scene breakthrough like AlphaFold in biology, the ML revolution
and might play an increasingly pivotal role in the future. might contribute to a quantum leap in our understanding of
the universe.
A main subject of this essay is the research to use ML to
study problems involving spacetimes with more dimensions
than four. String theory predicts extra dimensions that are
Link to Notebook
AI Report 2023 63
 Other Topics Essay #4

“Kaggle AI Report: Medical Data”

Honorable Mention
By Diego Flores
This report does a great job of introducing both the
challenges and the triumphs that AI researchers have
In Diego Flores essay "Lessons learned from working with encountered while working with medical data.
medical data", the author explores the use of AI
technologies in the medical space, and they discuss some
of the challenges and lessons learned. The report
highlights important concepts such as federated learning
and how it relates to privacy and security, and they discuss
the outsized importance of model explainability in medical
settings as well. Large language models (LLMs) can be
used to summarize both medical records and biomedical
research, and specialized tools such as ClinicalGPT are
presented to the reader along with additional LLM-based
applications.

Link to Notebook
AI Report 2023 64
 Other Topics Essay #5

“Graph Learning and Complex Networks”

Honorable Mention
By Hoda Jalali Najafabadi, Ali Jalali
In a joyously fractal way, this essay represents a microcosm
of the category we chose to call “Other”. Just like graphs can
Relational graphs are a central concept in understanding
teach us about path finding in decentralized robots, physical
the structure of complex systems like social networks or
particle dynamics, insurance fraud, neurological disorders, or
traffic flow patterns. Following a gentle introduction to the
disease contagions, so are the modern ML methods
world of graph theory, the authors of this review article
becoming relevant to more and more aspects of our world. As
take the reader on a tour of graph applications.
the graphs of our lives and endeavors are extending towards
the future, the connections to be built and cemented through
This essay touches on the areas of particle physics,
new ML technologies are bound to further accelerate the rate
anomaly and fraud detection, transportation and traffic,
of human progress in ever new and exciting ways.
protein folding (where we meet AlphaFold again),
cheminformatics and computational materials science,
brain and computational neuroscience, drug-drug
interaction, text data, and robotics and multi-agent
systems. Not content with this dizzying variety, the article
contains an additional plethora of well-organized
references and resources for the curious reader. Link to Notebook
AI Report 2023 65
Wrap-up

AI Report 2023 66
 In this effort, our community wrote hundreds of essays covering a broad array of topics,
and then experts from our community selected the best. The result is a collective
perspective on the rapid advancements of AI, shedding light on the most salient topics in
modern machine learning.

Many of the essays discussed the recent progress and potential of generative AI to

Conclusion revolutionize multiple industries and massively broaden access, making it the front runner
for shaping the future of AI. Ethics and “green” AI will command even more attention as
Phil Culliton scale, audience, and capabilities expand. Making sense of this future will require
validation through competitions, improved benchmarks, and other tools capable of
incorporating diverse and representative feedback.

We are excited by the potential of this report to highlight insights and advancements in
our field, as told through the collective mindshare of the Kaggle community. As machine
learning becomes more accessible, sampling from the diverse opinions of its practitioners
is an effective strategy to understand an ever-evolving discipline.

AI Report 2023 67
 We extend a tremendous thank you to the people
that made this report possible, including:

Bojan Tunguz - Area Chair

Camille McMorrow - Design
Christof Henkel - Area Chair
Claudia Sanchez - Design
D. Sculley - Supervising Author
Jephunney Nduati - Design
Julia Elliott - Coordinator

Credits
Karnika Kapoor - Area Chair
Kinjal Parekh - Coordinator
Mark McDonald - Copyeditor
Martin Henze - Area Chair
Parul Pandey - Area Chair
Paul Mooney - First author
Phil Culliton - First author
Raphael Kerley - Design
Rob Mulla - Area Chair
Sanyam Bhutani - Area Chair
Sara Wolley - Coordinator
Siddhita Upare - Illustrations
Will Cukierski - Copyeditor
AI Report 2023 68
 Paul Mooney*, Phil Culliton*, Abir Eltaief, Ali Jalali, Antong
C., Anya Mathur, Arya Gaikwad, Bojan Tunguz, Christof
Henkel, Chuandong Tang, Danial Sultanov, Dariusz
Kleczek, Dave Harold Mbiazi Njanda, Diego Flores, Dmitri
Kalinin, Harshit Mishra, Hoda Jalali Najafabadi, Julia Elliott,
Ivaxi Sheth, Karnika Kapoor, Kobbie Manrique, Leonie
Monigatti, Lezhi Li, Lorresprz, Mark McDonald, Martin
Henze, Maryam Babaei, Meghana Bhange, Nghi Huynh,
Citation Parul Pandey, Patrik Joslin Kenfack, Paulina Skorupska,
Piyush Mathur, Pranav Mohan Belhekar, Raghav Awasthi,
Rhys Cook, Rob Mulla, Samantha Lycett, Sanyam Bhutani,
Shreya Mishra, Svetlana Nosova, Theo Flaus, Trushant
Kalyanpur, Will Cukierski, Xinxi Chen, Yassine Motie, Yuqi
Liu, Yuxi Li, Zhengping Zhou,
D. Sculley.

(2023). AI Report 2023. Kaggle.

Kaggle.com / AI-Report-2023
AI Report 2023 69

*Equal first authors

Thanks

AI Report 2023 70
AI Report 2023 71