Linux Memory Management

Dec 12, 2012Download as PPTX, PDF13 likes7,211 views

This document discusses Linux memory management. It outlines the buddy system, zone allocation, and slab allocator used by Linux to manage physical memory. It describes how pages are allocated and initialized at boot using the memory map. The slab allocator is used to optimize allocation of kernel objects and is implemented as caches of fixed-size slabs and objects. Per-CPU allocation improves performance by reducing locking and cache invalidations.

Linux –
Memory Management
D3 Tsung-Han “Johnny” Lin

Outline
• Memory in Linux
– Dynamic Allocation
– Contiguous Allocation

•Memory Allocation
– Buddy System
– Zone
– SLAB Allocator
• Per-CPU data

12/4/26 Linux - Memory Management 3

MMU - Simple History

Early Linux on x86
Linux 2.5.46
(MMU)

Embedded Developer?!

µClinux
(Linux for microconcroller)

Jeff Dionne et al.1998

12/4/26 Linux - Memory Management 4

All about PAGE

• Physical page -> page frame
– Page descriptor, page

– All pages, mem_map

12/4/26 Linux - Memory Management 5

Initialization

query or invoke Kernel
BIOS

machine_specific_memory_setup()

setup_memory_map()

struct page* mem_map

Physical Memory

12/4/26 Linux - Memory Management 6

Memory

12/4/26 Linux - Memory Management 7

How pages are allocated?

12/4/26 Linux - Memory Management 8

Kernel API
• alloc_pages

• __get_free_pages

• kmalloc

• …

• But, seriously

12/4/26 Linux - Memory Management 9

HOW?

12/4/26 Linux - Memory Management 10

Hierarchy Overview
App w/ glibc heap

Zone Allocator
Slab Allocator
glibc

Buddy Allocator
MMU

Kernel Physical
Subsystems VM Subsystem Memory

kswapd
App w/ heap

bdflush Disk
Driver

User Space Kernel Space Disk
12/4/26 Linux - Memory Management 11

Memory Zone
Physical Memory (mem_map)

0x37999999, 896MB

ZONE_DMA ZONE_NORMAL ZONE_HIGHMEM
12/4/26 0x01000000, 16MB Linux - Memory Management 12
0x40000000, 1GB

Buddy System
zone descriptor
free_area[10] 1024 pages ……
………

………

………
free_area[11]
free_area[4]
free_area[3]

free_area[2] 4 pages ……
………

free_area[1] 2 pages 2 pages ……
free_area[0] 1 page 1 page 1 page ……

12/4/26 Linux - Memory Management 13

Zone, again

• ZONE_DMA, 0 ~ 16 MB

896MB ?!
• ZONE_NORMAL, 16 MB ~ 896 MB

• ZONE_HIGHMEM, rest of them …

12/4/26 Linux - Memory Management 14

Kernel View
Linear – 4 GB Physical
TASK_SIZE
0x0

3 GB

PAGE_OFFSET ZONE_HIGHMEM
0xc0000000 0x40000000, 1GB
Kernel
8 MB Image Reserved
0x37999999, 896MB

mem_map 128 MB

ZONE_NORMAL 1 GB
…

0x01000000, 16MB
896 MB ZONE_NORMAL ZONE_DMA
+
ZONE_DMA
0xffffffff
12/4/26 Linux - Memory Management 15

Do we need this large?!

12/4/26 Linux - Memory Management 16

SLAB Allocator
• First introduced in
大辞泉
SunOS 5.4
slab/slǽb/
• Implemented in Linux
名詞
2.2 1 （石木材金属などの）広い厚板；（材木
の）背板.
• SLAB system 2 （食べ物などの）厚切り
– SLAB a thick slab of apple pie｜厚切りのアップル
パイ.
– SLOB, embedded system 3｟俗語｟〘野球〙ピッチャーズプレート.
– SLUB, efficient ver. 4｟～｟｟
the 俗語｟（病院などの）死体置
き台
– SLQB, per-CPU scaling on the slab｜（死体となって）安置されて.
• kmalloc()!

12/4/26 Linux - Memory Management
Cache !! 17

Design of SLAB
cache_chain

slab_full
……
page struct
cache
slab
slab
object Page 1
page

…
cache
slab_partial slab
(kmem_cache) object
page

…
slab
object Page 2
cache
page
slab_partial
……

12/4/26 Linux - Memory Management 18

Per-CPU Allocation
• Request/Release single page
– Frequently

• Per-CPU
– Reducing cache invalidation
– Reducing locking
– A simple data array

• In 2.4 kernel
– get_cpu(), preemption disabled
– Access array directly

• In 2.6 kernel
– Wrapper
– percpu interface
– Cache alignment

12/4/26 Linux - Memory Management 19

Review
App w/ glibc heap

Zone Allocator
Slab Allocator
glibc

Buddy Allocator
MMU

Kernel Physical
Subsystems VM Subsystem Memory

kswapd
App w/ heap

bdflush Disk
Driver

User Space Kernel Space Disk
12/4/26 Linux - Memory Management 20

Reference
• Linux Kernel Development, 3rd
• Understanding the Linux Kernel, 3rd
– http://www.makelinux.net/books/ulk3/t
oc
• Understanding the Virtual Memory
Manager
– http://www.kernel.org/doc/gorman/htm
l/understand/index.html

12/4/26 Linux - Memory Management 21

Основные темы, затронутые на семинаре: Задачи и компоненты подсистемы управления памятью; Аппаратные возможности платформы x86_64; Как описывается в ядре физическая и виртуальная память; API подсистемы управления памятью; Высвобождение ранее занятой памяти; Инструменты мониторинга; Memory Cgroups; Compaction — дефрагментация физической памяти.

Physical Memory Models.pdfAdrian Huang

The document discusses four physical memory models in Linux: flat memory model, discontinuous memory model, sparse memory model, and sparse memory virtual memmap. It describes how each model addresses physical memory (page frames) and maps them to page descriptors. The sparse memory model is currently used, using memory sections to allocate page structures and support memory hotplug. It initializes by walking memory ranges from memblocks and allocating/initializing mem_section data structures.

Block I/O Layer Tracing: blktraceBabak Farrokhi

This document provides an overview of blktrace, a Linux kernel feature and set of utilities that allow detailed tracing of operations within the block I/O layer. Blktrace captures events for each I/O request as it is processed, including queue operations, merges, remapping by software RAID, and driver handling. The blktrace utilities extract these events and allow live tracing or storage for later analysis. Analysis tools like btt can analyze the stored blktrace data to measure processing times and identify bottlenecks or anomalies in how I/O requests are handled throughout the block I/O stack.

spinlock.pdfAdrian Huang

Page cache in Linux kernelAdrian Huang

Physical Memory Management.pdfAdrian Huang

Linux MMAP & Ioremap introductionGene Chang

The document discusses ioremap and mmap functions in Linux for mapping physical addresses into the virtual address space. Ioremap is used when physical addresses are larger than the virtual address space size. It maps physical addresses to virtual addresses that can be accessed by the CPU. Mmap allows a process to map pages of a file into virtual memory. It is useful for reducing memory copies and improving performance of file read/write operations. The document outlines the functions, flags, and flows of ioremap, mmap, and implementing a custom mmap file operation for direct physical memory mapping.

Kernel Recipes 2017 - Understanding the Linux kernel via ftrace - Steven RostedtAnne Nicolas

Ftrace is the official tracer of the Linux kernel. It has been apart of Linux since 2.6.31, and has grown tremendously ever since. Ftrace’s name comes from its most powerful feature: function tracing. But the ftrace infrastructure is much more than that. It also encompasses the trace events that are used by perf, as well as kprobes that can dynamically add trace events that the user defines. This talk will focus on learning how the kernel works by using the ftrace infrastructure. It will show how to see what happens within the kernel during a system call; learn how interrupts work; see how ones processes are being scheduled, and more. A quick introduction to some tools like trace-cmd and KernelShark will also be demonstrated. Steven Rostedt, VMware

Linux Memory Management with CMA (Contiguous Memory Allocator)Pankaj Suryawanshi

Process Address Space: The way to create virtual address (page table) of user...Adrian Huang

Kdump and the kernel crash dump analysisBuland Singh

Kdump is a kernel crash dumping mechanism that uses kexec to load a separate crash kernel to capture a kernel memory dump (vmcore file) when the primary kernel crashes. It can be configured to dump the vmcore file to local storage or over the network. Testing involves triggering a kernel panic using SysRq keys which causes the crash kernel to load and dump diagnostic information to the configured target path for analysis.

LISA2019 Linux Systems PerformanceBrendan Gregg

Talk by Brendan Gregg for USENIX LISA 2019: Linux Systems Performance. Abstract: " Systems performance is an effective discipline for performance analysis and tuning, and can help you find performance wins for your applications and the kernel. However, most of us are not performance or kernel engineers, and have limited time to study this topic. This talk summarizes the topic for everyone, touring six important areas of Linux systems performance: observability tools, methodologies, benchmarking, profiling, tracing, and tuning. Included are recipes for Linux performance analysis and tuning (using vmstat, mpstat, iostat, etc), overviews of complex areas including profiling (perf_events) and tracing (Ftrace, bcc/BPF, and bpftrace/BPF), and much advice about what is and isn't important to learn. This talk is aimed at everyone: developers, operations, sysadmins, etc, and in any environment running Linux, bare metal or the cloud."

Memory managementAdrien Mahieux

Memory is organized in a hierarchy from fast but small CPU caches to larger but slower dynamic RAM (DRAM) and static RAM (SRAM), with the Linux kernel using zone allocation, buddy allocation, and slab allocation to manage physical memory and handle fragmentation issues across these storage and processing layers. Applications rely on standard library allocators like malloc while the kernel uses slab allocators like SLAB, SLUB, and SLOB to manage memory for internal data structures.

Linux memory-management-kamalKamal Maiti

Memory Management with Page FoliosAdrian Huang

Linux Systems Performance 2016Brendan Gregg

Talk for PerconaLive 2016 by Brendan Gregg. Video: https://www.youtube.com/watch?v=CbmEDXq7es0 . "Systems performance provides a different perspective for analysis and tuning, and can help you find performance wins for your databases, applications, and the kernel. However, most of us are not performance or kernel engineers, and have limited time to study this topic. This talk summarizes six important areas of Linux systems performance in 50 minutes: observability tools, methodologies, benchmarking, profiling, tracing, and tuning. Included are recipes for Linux performance analysis and tuning (using vmstat, mpstat, iostat, etc), overviews of complex areas including profiling (perf_events), static tracing (tracepoints), and dynamic tracing (kprobes, uprobes), and much advice about what is and isn't important to learn. This talk is aimed at everyone: DBAs, developers, operations, etc, and in any environment running Linux, bare-metal or the cloud."

Linux memoryericrain911

Linux Initialization Process (2)shimosawa

The document provides an overview of the initialization process in the Linux kernel from start_kernel to rest_init. It lists the functions called during this process organized by category including functions for initialization of multiprocessor support (SMP), memory management (MM), scheduling, timers, interrupts, and architecture specific setup. The setup_arch section focuses on x86 architecture specific initialization functions such as reserving memory regions, parsing boot parameters, initializing memory mapping and MTRRs.

malloc & vmalloc in LinuxAdrian Huang

Memory Mapping Implementation (mmap) in Linux KernelAdrian Huang

semaphore & mutex.pdfAdrian Huang

The document summarizes Linux synchronization mechanisms including semaphores and mutexes. It discusses: 1. Semaphores can be used to solve producer-consumer problems and are implemented using a count and wait list. 2. Mutexes enforce serialization on shared memory and have fast, mid, and slow paths for lock and unlock. The mid path uses optimistic spinning and OSQs. 3. Only the lock owner can unlock a mutex, and mutexes transition to the slow path if the owner is preempted, a spinner is preempted, or the owner sleeps.

Reverse Mapping (rmap) in Linux KernelAdrian Huang

Linux Performance Profiling and MonitoringGeorg Schönberger

Linux Network StackAdrien Mahieux

- The document discusses Linux network stack monitoring and configuration. It begins with definitions of key concepts like RSS, RPS, RFS, LRO, GRO, DCA, XDP and BPF. - It then provides an overview of how the network stack works from the hardware interrupts and driver level up through routing, TCP/IP and to the socket level. - Monitoring tools like ethtool, ftrace and /proc/interrupts are described for viewing hardware statistics, software stack traces and interrupt information.

Linux Kernel Booting Process (2) - For NLKBshimosawa

eBPF Perf Tools 2019Brendan Gregg

Kernel_Crash_Dump_AnalysisBuland Singh

Slab Allocator in Linux KernelAdrian Huang

Memory management in LinuxRaghu Udiyar

Linux Memory ManagementAnil Kumar Pugalia

Linux uses memory management to partition memory between kernel and application spaces, organize memory using virtual addresses, and swap memory between primary and secondary storage. It divides memory using paging into equal-sized pages, creates virtual address spaces, and uses an MMU to translate between virtual and physical addresses. This allows processes to run independently with their own logical view of memory while the physical memory is shared.

More Related Content

What's hot (20)

Linux Memory Management with CMA (Contiguous Memory Allocator)Pankaj Suryawanshi

Process Address Space: The way to create virtual address (page table) of user...Adrian Huang

Kdump and the kernel crash dump analysisBuland Singh

LISA2019 Linux Systems PerformanceBrendan Gregg

Memory managementAdrien Mahieux

Linux memory-management-kamalKamal Maiti

Memory Management with Page FoliosAdrian Huang

Linux Systems Performance 2016Brendan Gregg

Linux memoryericrain911

Linux Initialization Process (2)shimosawa

malloc & vmalloc in LinuxAdrian Huang

Memory Mapping Implementation (mmap) in Linux KernelAdrian Huang

semaphore & mutex.pdfAdrian Huang

Reverse Mapping (rmap) in Linux KernelAdrian Huang

Linux Performance Profiling and MonitoringGeorg Schönberger

Linux Network StackAdrien Mahieux

Linux Kernel Booting Process (2) - For NLKBshimosawa

eBPF Perf Tools 2019Brendan Gregg

Kernel_Crash_Dump_AnalysisBuland Singh

Slab Allocator in Linux KernelAdrian Huang

Linux Memory Management with CMA (Contiguous Memory Allocator)Pankaj Suryawanshi

Process Address Space: The way to create virtual address (page table) of user...Adrian Huang

Kdump and the kernel crash dump analysisBuland Singh

LISA2019 Linux Systems PerformanceBrendan Gregg

Memory managementAdrien Mahieux

Linux memory-management-kamalKamal Maiti

Memory Management with Page FoliosAdrian Huang

Linux Systems Performance 2016Brendan Gregg

Linux memoryericrain911

Linux Initialization Process (2)shimosawa

malloc & vmalloc in LinuxAdrian Huang

Memory Mapping Implementation (mmap) in Linux KernelAdrian Huang

semaphore & mutex.pdfAdrian Huang

Reverse Mapping (rmap) in Linux KernelAdrian Huang

Linux Performance Profiling and MonitoringGeorg Schönberger

Linux Network StackAdrien Mahieux

Linux Kernel Booting Process (2) - For NLKBshimosawa

eBPF Perf Tools 2019Brendan Gregg

Kernel_Crash_Dump_AnalysisBuland Singh

Slab Allocator in Linux KernelAdrian Huang

Viewers also liked (20)

Memory management in LinuxRaghu Udiyar

Linux Memory ManagementAnil Kumar Pugalia

Memory management in linuxDr. C.V. Suresh Babu

The Linux kernel tracks each process's memory usage through data structures stored in the process's task_struct. The mm_struct stored there contains pointers to vm_area_struct objects representing each memory mapping. When a process calls malloc(), the kernel allocates physical pages and updates the process's mm_struct and vm_area_structs to map the new memory region into its virtual address space. Similarly when a process forks, the child process inherits copies of the parent's mm_struct and vm_area_structs, giving it the same memory mappings while keeping the two processes' memory private.

Process' Virtual Address Space in GNU/LinuxVarun Mahajan

Linux memory consumptionhaish

Linux MemoryVitaly Nahshunov

The document discusses virtual memory and paging. It describes why virtual memory is used, how it maps virtual to physical addresses, and its key features like protection, sharing, and swapping. Paging is discussed as a common virtual memory implementation where memory is divided into equal-sized pages and pages are mapped to equal-sized frames in physical memory using page tables. Page faults and translation look-up buffers are described to improve performance of virtual memory systems.

Understanding of linux kernel memory modelSeongJae Park

SeongJae Park introduces himself and his work contributing to the Linux kernel memory model documentation. He developed a guaranteed contiguous memory allocator and maintains the Korean translation of the kernel's memory barrier documentation. The document discusses how the increasing prevalence of multi-core processors requires careful programming to ensure correct parallel execution given relaxed memory ordering. It notes that compilers and CPUs optimize for instruction throughput over programmer goals, and memory accesses can be reordered in ways that affect correctness on multi-processors. Understanding the memory model is important for writing high-performance parallel code.

Linux Memory ManagementSuvendu Kumar Dash

Linux/DB Tuning (DevSumi2010, Japanese)Yoshinori Matsunobu

Christo kutrovsky oracle, memory & linuxKyle Hailey

This document provides information about Pythian, a company that provides database management and consulting services. It begins by introducing the presenter, Christo Kutrovsky, and his background. It then provides details about Pythian, including that it was founded in 1997, has over 200 employees, 200 customers worldwide, and 5 offices globally. It notes Pythian's partnerships and awards. The document emphasizes Pythian's expertise in Oracle, SQL Server, and other technologies. It positions Pythian as a recognized leader in database management.

Linux Kernel OverviewAnil Kumar Pugalia

The document provides an overview of the Linux kernel, including its architecture, startup process, functionality, configuration, and compilation. It discusses the differences between micro and monolithic kernels. It also explains the Linux kernel architecture with user space and kernel space separated by a system call interface. Key aspects covered include process management, memory management, device management, and the kernel build system.

Linux File SystemAnil Kumar Pugalia

The document discusses the Linux file system at three levels: hardware space, kernel space, and user space. At the hardware level, it describes how data is organized on physical storage devices like hard disks using partitions, tracks, sectors, and block allocation. In kernel space, file system drivers decode the physical layout and interface with the virtual file system (VFS) to provide a unified view to user space. Common Linux file systems like ext2, ext3, and their data structures are also outlined.

Memory managementRajni Sirohi

The document discusses memory management in operating systems. It covers key concepts like logical versus physical addresses, binding logical addresses to physical addresses, and different approaches to allocating memory like contiguous allocation. It also discusses dynamic storage allocation using a buddy system to merge adjacent free spaces, as well as compaction techniques to reduce external fragmentation by moving free memory blocks together. Memory management aims to efficiently share physical memory between processes using mechanisms like partitioning memory and enforcing protection boundaries.

Day 1-1-intro-sw-vulsStephanie Rogers

Here are the key points to consider when choosing an appropriate threat model: - The criticality and sensitivity of assets/data being protected - Attackers' likely incentives and capabilities given the assets/system in question - Costs of security measures vs potential losses/damages from attacks - Operational and technical constraints of the system/environment There is no single "right" threat model - the goal is to choose a realistic and useful one for security analysis and improvement, not an unachievably strong one. An appropriate balance of protection and practicality needs to be determined based on systematic risk analysis.

Putting a Fork in Fork (Linux Process and Memory Management)David Evans

The document discusses several topics related to computer science class cs4414 at University of Virginia: - Updates were due Sunday at 11:59pm including progress updates and scheduling design reviews. - Tuesday's class will feature a guest lecture on authentication using single sign-on. - The last class covered translation lookaside buffers and paging/segmentation concepts. - A code sample is shown and analyzed that causes a segmentation fault due to accessing memory outside the allocated space. - Details are provided on limiting resources and viewing process limits.

Linux memorymanagementpradeepelinux

This document discusses user-level memory management in Linux programming. It describes the different memory segments of a Linux process including the code, data, BSS, heap and stack segments. It explains how programs can allocate and free dynamic memory at runtime using library calls like malloc(), calloc(), realloc() and free(), as well as system calls like brk() and sbrk(). Examples of allocating, changing the size, and freeing memory are also provided.

Penginstalan linux slackware alfianostefano

Fedora15 lovelock-presJulio Villarreal Pelegrino

Fedora 15 "Lovelock" introduces several new features for desktop users, developers, and system administrators. For desktop users, it features GNOME 3.0, Btrfs filesystem, improved power management, LibreOffice, and Tryton business applications. Developers will benefit from tools like BoxGrinder for appliance generation, GCC 4.6 improvements, Maven 3, Python 3.2, and Rails 3. System administrators gain consistent network device naming, a dynamic firewall, and improved compression for live images using the xz algorithm. Alternative desktop environments and applications are available in different Fedora Spins.

Fedora Embedded at foss.in 2010Saleem Ansari

Segmentation fault (Web - Site - Project)George Tzinos

Δημιουργία μια διαδικτυακής υπηρεσίας όπου οι χρήστες θα μπορούν να κάνουν ερωτήσεις σχετικά με την πληροφορική. Την δημιουργία ενός μέρους όπου επιστήμονες της πληροφορικής θα μπορούν να συζητούν και να ανταλλάσσουν απόψεις. Είναι εμπνευσμένη από το Stack Overflow

Memory management in LinuxRaghu Udiyar

Linux Memory ManagementAnil Kumar Pugalia

Memory management in linuxDr. C.V. Suresh Babu

Process' Virtual Address Space in GNU/LinuxVarun Mahajan

Linux memory consumptionhaish

Linux MemoryVitaly Nahshunov

Understanding of linux kernel memory modelSeongJae Park

Linux Memory ManagementSuvendu Kumar Dash

Linux/DB Tuning (DevSumi2010, Japanese)Yoshinori Matsunobu

Christo kutrovsky oracle, memory & linuxKyle Hailey

Linux Kernel OverviewAnil Kumar Pugalia

Linux File SystemAnil Kumar Pugalia

Memory managementRajni Sirohi

Day 1-1-intro-sw-vulsStephanie Rogers

Putting a Fork in Fork (Linux Process and Memory Management)David Evans

Linux memorymanagementpradeepelinux

Penginstalan linux slackware alfianostefano

Fedora15 lovelock-presJulio Villarreal Pelegrino

Fedora Embedded at foss.in 2010Saleem Ansari

Segmentation fault (Web - Site - Project)George Tzinos

Similar to Linux Memory Management (20)

Controlling Memory Footprint at All Layers: Linux Kernel, Applications, Libra...peknap

Reducing memory usage is well covered in the history of this conference, yet new tricks still do exist. When optimizing memory footprint for an home gateway device, the author found some unexpected places where small changes can save valuable amount of DRAM or Flash space. This talk will visit different areas including - Kernel: fragmentation threshold, page frame reclamation task and atomic memory. Application level: Memory inefficient shared libraries due to ABI compliance and dynamic loading. Toolchain: Tuning malloc allocator parameters and compiler options. System level: General kernel might be more memory efficient than MMU-less uClinux, and preventing lock up when the system is on the brink of running out of memory.

Dsmp Whitepaper Release 3gelfstrom

This document provides an overview of the Distributed Symmetric Multiprocessing (DSMP) software architecture. DSMP transforms an InfiniBand connected cluster of commodity servers into a shared memory supercomputer through two unique software components: 1) a host operating system that runs on the head node, and 2) a lightweight microkernel that runs on the other servers. Key aspects of DSMP include a shared memory system, optimized InfiniBand drivers, an application-driven memory page coherency scheme, enhanced multithreading support, and distributed disk storage. DSMP allows commodity clusters to provide shared memory capabilities at a lower cost than proprietary supercomputers.

Memory Management in Linux-.ppt|FREE DOWNLOADfaizanayub16

“Memory Management in Linux: An In-depth Overview” This presentation provides a comprehensive guide to memory management in Linux, covering essential concepts such as virtual memory, memory allocation, paging, and swapping. It delves into how Linux handles process memory and the strategies used to optimize memory usage, including caching, buffer management, and kernel memory allocation. Ideal for beginners and advanced users alike, this slide deck aims to enhance understanding of Linux’s robust memory management framework and its role in ensuring system efficiency and performance.

Virtual memory 20070222-enTetsuyuki Kobayashi

Coerced Cache Eviction: Dealing with Misbehaving Disks through Discreet-Mode ...vchidambaram

Jaime Peñalba - Kernel exploitation. ¿El octavo arte? [rooted2019]RootedCON

The document discusses kernel exploitation. It begins with an introduction to basic kernel concepts like virtual memory and mitigations like SMAP, SMEP, and KPTI. It then covers topics like basic kernel exploitation through NULL pointer dereferences, dynamic memory management techniques like kmem caches and allocators, and real world exploitation using use-after-free vulnerabilities and doubly linked lists. The presentation uses examples to illustrate concepts at a high level since each topic could be its own full talk.

CSC1100 - Chapter05 - StorageYhal Htet Aung

The document discusses different types of storage devices and media, describing the characteristics of magnetic disks like hard disks, optical discs like CDs and DVDs, as well as other storage technologies. It covers the basics of how these storage devices work, their capacities and speeds, and differentiates between read-only, recordable, and rewritable variants. The goal is to help students understand computer hardware storage and how various storage devices and media store and retrieve data.

Linux Symposium 2009 Slide Suzaki "Effect of readahead and file system block ...Kuniyasu Suzaki

Ext2/3optimizer and readahead optimization can improve the performance of LBCAS (LoopBack Content Addressable Storage). Ext2/3optimizer rearranges file system blocks based on access profiles to reduce the number of small readahead requests. This increases the readahead coverage size, reduces redundant block downloads, and improves disk access locality. Performance analysis shows these optimizations reduce I/O utilization and consumption time in LBCAS during system booting.

Massively Parallel ArchitecturesJason Hearne-McGuiness

This document discusses massively parallel architectures and processing in memory (PIM) as ways to overcome the memory wall. It introduces several PIM architectures including Cyclops, DIMES, Gilgamesh, and Shamrock. Cyclops uses a cellular architecture with many thread units on a chip connected by a crossbar network. DIMES is a simplified emulator of Cyclops implemented on FPGA. Gilgamesh extends PIM capabilities with task and data virtualization. Shamrock arranges nodes in a grid, where each node contains a CPU and four memory arrays for shared memory access. The document explores how these architectures reduce latency and improve bandwidth to overcome limitations of traditional memory hierarchies.

My sql with enterprise storageCaroline_Rose

This document discusses using logical volume management (LVM) with MySQL for enterprise storage. Some key points include: - LVM allows virtualizing storage into logical volumes mapped to physical disks, enabling features like snapshots and clones for backups and replication. - Snapshots provide point-in-time copies while clones are writable copies, both using copy-on-write to avoid duplicating data. - Backing up MySQL with snapshots is fast, and recovery involves reverting the logical volumes rather than restoring files. Replication slaves can be quickly recloned from snapshots. - With shared storage, replication slaves maintain the same blocks as the master for efficient scaling and resyncing slaves by recloning

Driver development – memory managementVandana Salve

1. The document discusses various methods for managing memory in the Linux kernel, including physical memory, virtual memory, page tables, and different allocators like kmalloc, vmalloc, and SLAB for allocating memory to processes and the kernel. 2. It explains concepts like physical vs virtual addresses, page tables that map virtual to physical memory, and the Memory Management Unit (MMU) that handles virtual address translation. 3. Different allocators like kmalloc, vmalloc and SLAB are used depending on the size and properties of the memory needed, with kmalloc and SLAB handling physically contiguous memory and vmalloc only requiring virtual contiguity.

Os solaris memory managementTech_MX

The document discusses Solaris memory management. It describes Solaris' memory architecture including backing store, virtual memory system, and process memory allocation. It then discusses Solaris' memory management techniques, including swapping and demand paging. Demand paging loads pages of memory on demand to lower memory footprint and startup time, while swapping is used as a last resort. Memory is shared between processes and protected via virtual memory and page protections.

EuroSec2012 "Effects of Memory Randomization, Sanitization and Page Cache on ...Kuniyasu Suzaki

Windows memory manager internalsSisimon Soman

The document discusses Windows memory and cache manager internals. It covers several topics: - The virtual memory manager (VMM) abstracts physical memory to make it feel infinite to applications. It protects OS memory and enables memory sharing between applications. - Paging is used to divide physical memory into equal size pages. Address translation uses page directories (PDEs) and page tables (PTEs) along with a translation lookaside buffer (TLB) for faster lookups. - The cache manager improves performance by caching frequently used disk blocks in physical memory. It facilitates read-ahead and write-back caching to reduce disk access.

OsAgnas Jasmine

This document summarizes key aspects of disk structure, disk scheduling, disk management, swap space management, and Windows 2000. It discusses how disks are logically structured and addressed, factors that impact disk access time like seek time and rotational latency. It covers disk formatting, partitioning, boot blocks, handling bad blocks. It explains how operating systems manage swap space, where it is located, and examples of swap space use. It provides an overview of Windows 2000, describing its design, components like the kernel, virtual memory manager, I/O manager, and environmental subsystems.

Linux.Conf.AU 2009 (LCA09) Slide "OS Circular: Internet bootable OS Archive" ...Kuniyasu Suzaki

Conceptual framework storage devices (2)Rajendra Sharma

The document discusses different types of computer memory and storage devices. It describes primary memory such as RAM and ROM, as well as secondary storage devices like magnetic disks, tapes, flash drives, and optical discs. RAM is volatile and used for active programs and data, while ROM is permanent and stores the basic input/output system. Cache memory stores frequently used data to speed up CPU access time. Secondary storage includes hard disks, floppy disks, magnetic tapes, flash drives, and optical discs like CDs, DVDs, which provide larger non-volatile storage.

Spectrum Scale Memory UsageTomer Perry

This document discusses Spectrum Scale memory usage. It outlines Spectrum Scale basics like clusters, nodes, and filesystems. It describes the different Spectrum Scale memory pools: pagepool for data, shared segment for metadata references, and external heap for daemons. It provides information on calculating memory needs based on parameters like files to cache, stat cache size, nodes, and access patterns. Other topics covered include related Linux memory usage and out of scope memory components.

INFLOW-2014-NVM-CompressionDhananjoy ( Joy ) Das

This document provides an overview of NVM compression, a hybrid flash-aware application level compression solution. It discusses the drawbacks of existing row-level compression in MySQL and outlines an architecture for NVM compression that avoids these drawbacks. Key aspects of the NVM compression approach include performing compression only during flush, using sparse addressing to avoid over-provisioning flash space, and adding a new multi-threaded flush framework. Evaluation results and building blocks of the solution are also briefly mentioned.

UpdatesIsa Ansharullah

The document discusses several key aspects of processes and memory management in Linux: 1. A process is represented by a task_struct structure that contains information like the process ID, open files, address space, state, and stack. 2. Processes have both a user stack and a fixed-size kernel stack. Context switches occur when switching between these stacks for system calls or exceptions. 3. The fork() system call duplicates a process by using copy-on-write techniques to efficiently copy resources from the parent process. 4. Memory allocation for kernel objects like task_struct uses slab allocators to improve performance over the buddy allocator through object caching and reducing initialization overhead.

Controlling Memory Footprint at All Layers: Linux Kernel, Applications, Libra...peknap

Dsmp Whitepaper Release 3gelfstrom

Memory Management in Linux-.ppt|FREE DOWNLOADfaizanayub16

Virtual memory 20070222-enTetsuyuki Kobayashi

Coerced Cache Eviction: Dealing with Misbehaving Disks through Discreet-Mode ...vchidambaram

Jaime Peñalba - Kernel exploitation. ¿El octavo arte? [rooted2019]RootedCON

CSC1100 - Chapter05 - StorageYhal Htet Aung

Linux Symposium 2009 Slide Suzaki "Effect of readahead and file system block ...Kuniyasu Suzaki

Massively Parallel ArchitecturesJason Hearne-McGuiness

My sql with enterprise storageCaroline_Rose

Driver development – memory managementVandana Salve

Os solaris memory managementTech_MX

EuroSec2012 "Effects of Memory Randomization, Sanitization and Page Cache on ...Kuniyasu Suzaki

Windows memory manager internalsSisimon Soman

OsAgnas Jasmine

Linux.Conf.AU 2009 (LCA09) Slide "OS Circular: Internet bootable OS Archive" ...Kuniyasu Suzaki

Conceptual framework storage devices (2)Rajendra Sharma

Spectrum Scale Memory UsageTomer Perry

INFLOW-2014-NVM-CompressionDhananjoy ( Joy ) Das

UpdatesIsa Ansharullah

Linux Memory Management

1. Linux – Memory Management D3 Tsung-Han “Johnny” Lin

2. 12/4/26 Linux - Memory Management 2

3. Outline • Memory in Linux – Dynamic Allocation – Contiguous Allocation •Memory Allocation – Buddy System – Zone – SLAB Allocator • Per-CPU data 12/4/26 Linux - Memory Management 3

4. MMU - Simple History Early Linux on x86 Linux 2.5.46 (MMU) Embedded Developer?! µClinux (Linux for microconcroller) Jeff Dionne et al.1998 12/4/26 Linux - Memory Management 4

5. All about PAGE • Physical page -> page frame – Page descriptor, page – All pages, mem_map 12/4/26 Linux - Memory Management 5

6. Initialization query or invoke Kernel BIOS machine_specific_memory_setup() setup_memory_map() struct page* mem_map Physical Memory 12/4/26 Linux - Memory Management 6

7. Memory 12/4/26 Linux - Memory Management 7

8. How pages are allocated? 12/4/26 Linux - Memory Management 8

9. Kernel API • alloc_pages • __get_free_pages • kmalloc • … • But, seriously 12/4/26 Linux - Memory Management 9

10. HOW? 12/4/26 Linux - Memory Management 10

11. Hierarchy Overview App w/ glibc heap Zone Allocator Slab Allocator glibc Buddy Allocator MMU Kernel Physical Subsystems VM Subsystem Memory kswapd App w/ heap bdflush Disk Driver User Space Kernel Space Disk 12/4/26 Linux - Memory Management 11

12. Memory Zone Physical Memory (mem_map) 0x37999999, 896MB ZONE_DMA ZONE_NORMAL ZONE_HIGHMEM 12/4/26 0x01000000, 16MB Linux - Memory Management 12 0x40000000, 1GB

13. Buddy System zone descriptor free_area[10] 1024 pages …… ……… ……… ……… free_area[11] free_area[4] free_area[3] free_area[2] 4 pages …… ……… free_area[1] 2 pages 2 pages …… free_area[0] 1 page 1 page 1 page …… 12/4/26 Linux - Memory Management 13

14. Zone, again • ZONE_DMA, 0 ~ 16 MB 896MB ?! • ZONE_NORMAL, 16 MB ~ 896 MB • ZONE_HIGHMEM, rest of them … 12/4/26 Linux - Memory Management 14

15. Kernel View Linear – 4 GB Physical TASK_SIZE 0x0 3 GB PAGE_OFFSET ZONE_HIGHMEM 0xc0000000 0x40000000, 1GB Kernel 8 MB Image Reserved 0x37999999, 896MB mem_map 128 MB ZONE_NORMAL 1 GB … 0x01000000, 16MB 896 MB ZONE_NORMAL ZONE_DMA + ZONE_DMA 0xffffffff 12/4/26 Linux - Memory Management 15

16. Do we need this large?! 12/4/26 Linux - Memory Management 16

17. SLAB Allocator • First introduced in 大辞泉 SunOS 5.4 slab/slǽb/ • Implemented in Linux 名詞 2.2 1 （石木材金属などの）広い厚板；（材木の）背板. • SLAB system 2 （食べ物などの）厚切り – SLAB a thick slab of apple pie｜厚切りのアップルパイ. – SLOB, embedded system 3｟俗語｟〘野球〙ピッチャーズプレート. – SLUB, efficient ver. 4｟～｟｟ the 俗語｟（病院などの）死体置き台 – SLQB, per-CPU scaling on the slab｜（死体となって）安置されて. • kmalloc()! 12/4/26 Linux - Memory Management Cache !! 17

18. Design of SLAB cache_chain slab_full …… page struct cache slab slab object Page 1 page … cache slab_partial slab (kmem_cache) object page … slab object Page 2 cache page slab_partial …… 12/4/26 Linux - Memory Management 18

19. Per-CPU Allocation • Request/Release single page – Frequently • Per-CPU – Reducing cache invalidation – Reducing locking – A simple data array • In 2.4 kernel – get_cpu(), preemption disabled – Access array directly • In 2.6 kernel – Wrapper – percpu interface – Cache alignment 12/4/26 Linux - Memory Management 19

20. Review App w/ glibc heap Zone Allocator Slab Allocator glibc Buddy Allocator MMU Kernel Physical Subsystems VM Subsystem Memory kswapd App w/ heap bdflush Disk Driver User Space Kernel Space Disk 12/4/26 Linux - Memory Management 20

21. Reference • Linux Kernel Development, 3rd • Understanding the Linux Kernel, 3rd – http://www.makelinux.net/books/ulk3/t oc • Understanding the Virtual Memory Manager – http://www.kernel.org/doc/gorman/htm l/understand/index.html 12/4/26 Linux - Memory Management 21

Linux Memory Management

Recommended

More Related Content

What's hot (20)

Viewers also liked (20)

Similar to Linux Memory Management (20)

Linux Memory Management