userspace                                    kernel
═════════════════════════════════════════════════════════════════


 virtual address 0x5... inside .text
        │
        │ process executes from mapped ELF region
        ▼
┌─────────────────────────────┐
│ VMA / userspace mapping     │
│ r-x file-backed region      │ e.g.
│ e.g. /usr/bin/su .text      │ 555555554000-555555556000 r--p /usr/bin/su
└──────────────┬──────────────┘ 555555556000-55555557a000 r-xp /usr/bin/su
               │
               │ page-table lookup
               │ virtual → physical translation
               │
               └────────────────────────────────┐
                                                ▼
                                  ┌─────────────────────────────┐
                                  │     page-table entries      │ 
        0x5555... → phys 0x1ab000 │ virtual → physical mapping  │ 
                                  └──────────────┬──────────────┘
                                                 │
                                                 │ points to
                                                 ▼
                                  ┌─────────────────────────────┐
                                  │       struct page           │
                   e.g. PFN 0x1ab │     physical page frame     │
                                  └──────────────┬──────────────┘
                                                 │
                                                 │ represents cached bytes from
                                                 ▼
                                  ┌─────────────────────────────┐
                                  │       page cache page       │
          e.g. file offset 0x2000 │    /usr/bin/su + 0x2000     │
                                  └──────────────┬──────────────┘
                                                 │
                                                 │ tracked by
                                                 ▼
                                  ┌─────────────────────────────┐
                                  │     inode(/usr/bin/su)      │
                                  │  address_space cache tree   │
                                  └──────────────┬──────────────┘
                                                 │
                                                 │ originally loaded from
                                                 ▼
                                  ┌─────────────────────────────┐
                                  │        file on disk         │
                                  │         /usr/bin/su         │
                                  └─────────────────────────────┘

 userspace                                    kernel
═════════════════════════════════════════════════════════════════


 virtual address 0x5... inside .text
        │
        │ process executes from mapped ELF region
        ▼
┌─────────────────────────────┐
│ VMA / userspace mapping     │
│ r-x file-backed region      │ e.g.
│ e.g. /usr/bin/su .text      │ 555555554000-555555556000 r--p /usr/bin/su
└──────────────┬──────────────┘ 555555556000-55555557a000 r-xp /usr/bin/su
               │
               │ page-table lookup
               │ virtual → physical translation
               │
               └────────────────────────────────┐
                                                ▼
                                  ┌─────────────────────────────┐
                                  │     page-table entries      │ 
        0x5555... → phys 0x1ab000 │ virtual → physical mapping  │ 
                                  └──────────────┬──────────────┘
                                                 │
                                                 │ points to
                                                 ▼
                                  ┌─────────────────────────────┐
                                  │       struct page           │
                   e.g. PFN 0x1ab │     physical page frame     │
                                  └──────────────┬──────────────┘
                                                 │
                                                 │ represents cached bytes from
                                                 ▼
                                  ┌─────────────────────────────┐
                                  │       page cache page       │
          e.g. file offset 0x2000 │    /usr/bin/su + 0x2000     │
                                  └──────────────┬──────────────┘
                                                 │
                                                 │ tracked by
                                                 ▼
                                  ┌─────────────────────────────┐
                                  │     inode(/usr/bin/su)      │
                                  │  address_space cache tree   │
                                  └──────────────┬──────────────┘
                                                 │
                                                 │ originally loaded from
                                                 ▼
                                  ┌─────────────────────────────┐
                                  │        file on disk         │
                                  │         /usr/bin/su         │
                                  └─────────────────────────────┘

 userspace                             kernel
════════════════════════════════════════════════════════

 process calls syscall
 execve("/usr/bin/su")
        │               context switch -> kernel mode
        └────────────────────────────────┐
                                         │
                                         ▼
                            ┌─────────────────────────┐
                            │ handle execve syscall   │
                            └────────────┬────────────┘
                                         │
                                         ▼
                            ┌─────────────────────────┐
                            │ readfile /usr/bin/su    │
                            │ parse ELF headers       │
                            │ inspect PT_LOAD entries │
                            └────────────┬────────────┘
                                         │
                                         ▼
                            ┌─────────────────────────┐
                            │ create userspace VMAs   │
                            │ backed by /usr/bin/su   │
                            │                         │
                            │ .text   → r-x           │
                            │ .rodata → r--           │
                            │ .data   → rw-           │
                            └────────────┬────────────┘
                                         │
                                         ▼
                            ┌─────────────────────────┐
                            │  [!] connect VMAs to    │
                            │   page-cache pages      │
                            └────────────┬────────────┘
                                         │
                                         │ returns to user mode
        ┌────────────────────────────────┘
        │
        ▼
process begins execution
at ELF entry point using mapped .text pages

 userspace                             kernel
════════════════════════════════════════════════════════

 process calls syscall
 execve("/usr/bin/su")
        │               context switch -> kernel mode
        └────────────────────────────────┐
                                         │
                                         ▼
                            ┌─────────────────────────┐
                            │ handle execve syscall   │
                            └────────────┬────────────┘
                                         │
                                         ▼
                            ┌─────────────────────────┐
                            │ readfile /usr/bin/su    │
                            │ parse ELF headers       │
                            │ inspect PT_LOAD entries │
                            └────────────┬────────────┘
                                         │
                                         ▼
                            ┌─────────────────────────┐
                            │ create userspace VMAs   │
                            │ backed by /usr/bin/su   │
                            │                         │
                            │ .text   → r-x           │
                            │ .rodata → r--           │
                            │ .data   → rw-           │
                            └────────────┬────────────┘
                                         │
                                         ▼
                            ┌─────────────────────────┐
                            │  [!] connect VMAs to    │
                            │   page-cache pages      │
                            └────────────┬────────────┘
                                         │
                                         │ returns to user mode
        ┌────────────────────────────────┘
        │
        ▼
process begins execution
at ELF entry point using mapped .text pages

static unsigned long elf_map(struct file *filep, unsigned long addr,
                            const struct elf_phdr *eppnt, int prot, int type,
                            unsigned long total_size)
{
    ...
    map_addr = vm_mmap(filep, addr, size, prot, type, off);
    ...
}

static unsigned long elf_map(struct file *filep, unsigned long addr,
                            const struct elf_phdr *eppnt, int prot, int type,
                            unsigned long total_size)
{
    ...
    map_addr = vm_mmap(filep, addr, size, prot, type, off);
    ...
}

file on disk
    │
    │ read / mmap / execve
    ▼
page-cache page
    │
    │ mapped into process as file-backed memory
    ▼
userspace VMA

file on disk
    │
    │ read / mmap / execve
    ▼
page-cache page
    │
    │ mapped into process as file-backed memory
    ▼
userspace VMA

struct inode {
    ...
    struct address_space *i_mapping;  // pointer to page-cache mapping for this file
    ...
    struct address_space i_data;      // stores metadata about the file's cached pages
    ...
};

struct inode {
    ...
    struct address_space *i_mapping;  // pointer to page-cache mapping for this file
    ...
    struct address_space i_data;      // stores metadata about the file's cached pages
    ...
};

struct address_space {
    struct inode            *host;
    struct xarray           i_pages;
    ...
    struct rb_root_cached   i_mmap;
    ...
};

struct address_space {
    struct inode            *host;
    struct xarray           i_pages;
    ...
    struct rb_root_cached   i_mmap;
    ...
};

file on disk: /usr/bin/su
              │
              ▼
      ┌───────────────┐
      │     inode     │
      └───────┬───────┘
              │ owns
              ▼
      ┌─────────────────────────────┐
      │       address_space         │
      │                             │
      │ host    -> this inode       │
      │ i_pages -> cached folios    │────▶ cached executable page, e.g. file offset 0x2000
      │ i_mmap  -> mapped VMAs      │────▶ VMA in proc A: /usr/bin/su r-xp
      └─────────────────────────────┘      VMA in proc B: /usr/bin/su r-xp
                                           VMA in proc C: /usr/bin/su r-xp
                                           ...

file on disk: /usr/bin/su
              │
              ▼
      ┌───────────────┐
      │     inode     │
      └───────┬───────┘
              │ owns
              ▼
      ┌─────────────────────────────┐
      │       address_space         │
      │                             │
      │ host    -> this inode       │
      │ i_pages -> cached folios    │────▶ cached executable page, e.g. file offset 0x2000
      │ i_mmap  -> mapped VMAs      │────▶ VMA in proc A: /usr/bin/su r-xp
      └─────────────────────────────┘      VMA in proc B: /usr/bin/su r-xp
                                           VMA in proc C: /usr/bin/su r-xp
                                           ...

vm_fault_t filemap_fault(struct vm_fault *vmf)
{
    // file mapped by the faulting VMA 
    struct file *file = vmf->vma->vm_file;

    // page-cache mapping associated with the file
    struct address_space *mapping = file->f_mapping;

    // inode owning this page-cache mapping
    struct inode *inode = mapping->host;

    // page-cache index for the faulting offset
    pgoff_t index = vmf->pgoff;
    ...

    /*  
    * Do we have something in the page cache already?  
    */  
    folio = filemap_get_folio(mapping, index);
        // Try to find the cached folio/page for this file offset
    ...
}

vm_fault_t filemap_fault(struct vm_fault *vmf)
{
    // file mapped by the faulting VMA 
    struct file *file = vmf->vma->vm_file;

    // page-cache mapping associated with the file
    struct address_space *mapping = file->f_mapping;

    // inode owning this page-cache mapping
    struct inode *inode = mapping->host;

    // page-cache index for the faulting offset
    pgoff_t index = vmf->pgoff;
    ...

    /*  
    * Do we have something in the page cache already?  
    */  
    folio = filemap_get_folio(mapping, index);
        // Try to find the cached folio/page for this file offset
    ...
}

struct folio *__filemap_get_folio(struct address_space *mapping, pgoff_t index,
                                fgf_t fgp_flags, gfp_t gfp)
{
    struct folio *folio;

repeat:

    // lookup cached folio/page by file offset 
    folio = filemap_get_entry(mapping, index);

    ...

    if (!folio)
        goto no_page;

    ...

    return folio;

no_page:

    // on a cache miss, allocate and insert a new folio
    // if the caller requested creation
    if (!folio && (fgp_flags & FGP_CREAT)) {

        ...

        folio = filemap_alloc_folio(alloc_gfp, order);

        ...

        err = filemap_add_folio(mapping, folio, index, gfp);

        ...
    }

    return folio;
}

struct folio *__filemap_get_folio(struct address_space *mapping, pgoff_t index,
                                fgf_t fgp_flags, gfp_t gfp)
{
    struct folio *folio;

repeat:

    // lookup cached folio/page by file offset 
    folio = filemap_get_entry(mapping, index);

    ...

    if (!folio)
        goto no_page;

    ...

    return folio;

no_page:

    // on a cache miss, allocate and insert a new folio
    // if the caller requested creation
    if (!folio && (fgp_flags & FGP_CREAT)) {

        ...

        folio = filemap_alloc_folio(alloc_gfp, order);

        ...

        err = filemap_add_folio(mapping, folio, index, gfp);

        ...
    }

    return folio;
}

/**
 * filemap_read - Read data from the page cache.
 * @iocb: The iocb to read.
 * @iter: Destination for the data.
 * @already_read: Number of bytes already read by the caller.
 *
 * Copies data from the page cache.  If the data is not currently present,
 * uses the readahead and read_folio address_space operations to fetch it.
 *
 * ...
 */
ssize_t filemap_read(struct kiocb *iocb, struct iov_iter *iter,
        ssize_t already_read) { ... }

/**
 * filemap_read - Read data from the page cache.
 * @iocb: The iocb to read.
 * @iter: Destination for the data.
 * @already_read: Number of bytes already read by the caller.
 *
 * Copies data from the page cache.  If the data is not currently present,
 * uses the readahead and read_folio address_space operations to fetch it.
 *
 * ...
 */
ssize_t filemap_read(struct kiocb *iocb, struct iov_iter *iter,
        ssize_t already_read) { ... }

system boot
══════════════════════════════════════════════════════

 initialize memory manager
 initialize page allocator
 initialize page-cache infrastructure
                │
                │ no file pages cached yet
                ▼
         page cache starts empty


runtime
══════════════════════════════════════════════════════

process executes, reads, or maps /usr/bin/su
                │
                │ execve() / read() / mmap()
                ▼
    kernel requests file-backed data
                │
                │ load from disk on cache miss
                ▼
    ┌──────────────────────────────┐
    │ page cache                   │
    │ cached /usr/bin/su pages     │
    │ executable file data in RAM  │
    └──────────────┬───────────────┘
                   │
                   │ reused by later access
                   ▼
          proc A → /usr/bin/su
          proc B → /usr/bin/su
          proc C → /usr/bin/su

system boot
══════════════════════════════════════════════════════

 initialize memory manager
 initialize page allocator
 initialize page-cache infrastructure
                │
                │ no file pages cached yet
                ▼
         page cache starts empty


runtime
══════════════════════════════════════════════════════

process executes, reads, or maps /usr/bin/su
                │
                │ execve() / read() / mmap()
                ▼
    kernel requests file-backed data
                │
                │ load from disk on cache miss
                ▼
    ┌──────────────────────────────┐
    │ page cache                   │
    │ cached /usr/bin/su pages     │
    │ executable file data in RAM  │
    └──────────────┬───────────────┘
                   │
                   │ reused by later access
                   ▼
          proc A → /usr/bin/su
          proc B → /usr/bin/su
          proc C → /usr/bin/su

case MAP_SHARED:
case MAP_SHARED_VALIDATE:

        // shared writable mmap requires writable file descriptor
        if (prot & PROT_WRITE) {
                if (!(file->f_mode & FMODE_WRITE))
                        return -EACCES;
                ...
        }

        ...

        // remove VMA write/share capability if fd is read-only
        if (!(file->f_mode & FMODE_WRITE))
                vm_flags &= ~(VM_MAYWRITE | VM_SHARED);

case MAP_SHARED:
case MAP_SHARED_VALIDATE:

        // shared writable mmap requires writable file descriptor
        if (prot & PROT_WRITE) {
                if (!(file->f_mode & FMODE_WRITE))
                        return -EACCES;
                ...
        }

        ...

        // remove VMA write/share capability if fd is read-only
        if (!(file->f_mode & FMODE_WRITE))
                vm_flags &= ~(VM_MAYWRITE | VM_SHARED);

int fd[2];
pipe(fd);

write(fd[1], "hello", 5);

char buf[16];
read(fd[0], buf, 5);

int fd[2];
pipe(fd);

write(fd[1], "hello", 5);

char buf[16];
read(fd[0], buf, 5);

struct pipe_inode_info {
    ...
    unsigned int head;
    unsigned int tail;
    unsigned int ring_size;
    struct pipe_buffer *bufs;  // ring of pipe_buffer descriptors
    ...
};

struct pipe_inode_info {
    ...
    unsigned int head;
    unsigned int tail;
    unsigned int ring_size;
    struct pipe_buffer *bufs;  // ring of pipe_buffer descriptors
    ...
};

pipe_inode_info
    └── bufs[] ring
            ├── pipe_buffer
            ├── pipe_buffer
            └── ...

pipe_inode_info
    └── bufs[] ring
            ├── pipe_buffer
            ├── pipe_buffer
            └── ...

struct pipe_buffer {
    struct page *page;                     // backing page
    unsigned int offset, len;              // where the data starts inside that page
    const struct pipe_buf_operations *ops; // how many bytes are valid
    unsigned int flags;                    // rules for handling this buffer
    unsigned long private;                 // buffer state
};

struct pipe_buffer {
    struct page *page;                     // backing page
    unsigned int offset, len;              // where the data starts inside that page
    const struct pipe_buf_operations *ops; // how many bytes are valid
    unsigned int flags;                    // rules for handling this buffer
    unsigned long private;                 // buffer state
};

                    struct pipe_inode_info
            +--------------------------------------+
            |      head / tail / ring_size         |
            |                                      |
            |            bufs[] ring               |
            |   tail                 head          |
            |     v                    v           |
            |  +------+------+------+------+---+   |
            |  | buf0 | buf1 | buf2 | buf3 |...|   |
            |  +------+------+------+------+---+   |
            |  indices advance modulo ring_size    |
            +--------------------------------------+
                      |       |       |
                      v       v       v

                struct pipe_buffer descriptors
                +-----------------------------+
                | page   -> backing page      |
                | offset -> start in page     |
                | len    -> visible bytes     |
                | ops    -> buffer operations |
                | flags  -> buffer provenance |
                +---------------+-------------+
                                |
                +---------------+---------------+
                |                               |
                v                               v
        anonymous page               file-backed page-cache page
         from ordinary write()        from splice()

                    struct pipe_inode_info
            +--------------------------------------+
            |      head / tail / ring_size         |
            |                                      |
            |            bufs[] ring               |
            |   tail                 head          |
            |     v                    v           |
            |  +------+------+------+------+---+   |
            |  | buf0 | buf1 | buf2 | buf3 |...|   |
            |  +------+------+------+------+---+   |
            |  indices advance modulo ring_size    |
            +--------------------------------------+
                      |       |       |
                      v       v       v

                struct pipe_buffer descriptors
                +-----------------------------+
                | page   -> backing page      |
                | offset -> start in page     |
                | len    -> visible bytes     |
                | ops    -> buffer operations |
                | flags  -> buffer provenance |
                +---------------+-------------+
                                |
                +---------------+---------------+
                |                               |
                v                               v
        anonymous page               file-backed page-cache page
         from ordinary write()        from splice()

struct pipe_buf_operations {
    int  (*confirm)(...);    // validate buffer before use
    void (*release)(...);    // drop references / cleanup
    bool (*try_steal)(...);  // transfer page ownership if possible
    bool (*get)(...);        // acquire another page reference
};

struct pipe_buf_operations {
    int  (*confirm)(...);    // validate buffer before use
    void (*release)(...);    // drop references / cleanup
    bool (*try_steal)(...);  // transfer page ownership if possible
    bool (*get)(...);        // acquire another page reference
};

static const struct pipe_buf_operations anon_pipe_buf_ops = {
    .release       = anon_pipe_buf_release,
    .try_steal     = anon_pipe_buf_try_steal,
    .get           = generic_pipe_buf_get,
};

...

buf->ops = &anon_pipe_buf_ops;  // dispatch pipe operations

static const struct pipe_buf_operations anon_pipe_buf_ops = {
    .release       = anon_pipe_buf_release,
    .try_steal     = anon_pipe_buf_try_steal,
    .get           = generic_pipe_buf_get,
};

...

buf->ops = &anon_pipe_buf_ops;  // dispatch pipe operations

pipe_buffer
    └── ops
          └── anon_pipe_buf_ops
                ├── .get       -> generic_pipe_buf_get()
                ├── .release   -> anon_pipe_buf_release()
                └── .try_steal -> anon_pipe_buf_try_steal()

pipe_buffer
    └── ops
          └── anon_pipe_buf_ops
                ├── .get       -> generic_pipe_buf_get()
                ├── .release   -> anon_pipe_buf_release()
                └── .try_steal -> anon_pipe_buf_try_steal()

bool generic_pipe_buf_get(struct pipe_inode_info *pipe,
                          struct pipe_buffer *buf)
{
    return try_get_page(buf->page);
}

bool generic_pipe_buf_get(struct pipe_inode_info *pipe,
                          struct pipe_buffer *buf)
{
    return try_get_page(buf->page);
}

void generic_pipe_buf_release(struct pipe_inode_info *pipe,
                              struct pipe_buffer *buf)
{
    put_page(buf->page);
}

void generic_pipe_buf_release(struct pipe_inode_info *pipe,
                              struct pipe_buffer *buf)
{
    put_page(buf->page);
}

bool generic_pipe_buf_try_steal(struct pipe_inode_info *pipe,
                                struct pipe_buffer *buf)
{
    struct page *page = buf->page;

    // Only steal if this is the last remaining page reference
    if (page_count(page) == 1) {

            // 1 means the pipe holds the only remaining page reference
            lock_page(page);     // caller now receives the locked page
            return true;         // ownership transfer succeeded
    }

    return false;                // still referenced elsewhere; cannot steal
}

bool generic_pipe_buf_try_steal(struct pipe_inode_info *pipe,
                                struct pipe_buffer *buf)
{
    struct page *page = buf->page;

    // Only steal if this is the last remaining page reference
    if (page_count(page) == 1) {

            // 1 means the pipe holds the only remaining page reference
            lock_page(page);     // caller now receives the locked page
            return true;         // ownership transfer succeeded
    }

    return false;                // still referenced elsewhere; cannot steal
}

ordinary copied path
════════════════════════════════════════

  source bytes
        │
        │ copy_page_from_iter()
        ▼
    anonymous pipe-owned page
        │
        ▼
pipe_buffer -> new page


zero-copy style path
════════════════════════════════════════

  existing page
e.g. page-cache page
        │
        │ install page reference
        ▼
pipe_buffer -> existing page

ordinary copied path
════════════════════════════════════════

  source bytes
        │
        │ copy_page_from_iter()
        ▼
    anonymous pipe-owned page
        │
        ▼
pipe_buffer -> new page


zero-copy style path
════════════════════════════════════════

  existing page
e.g. page-cache page
        │
        │ install page reference
        ▼
pipe_buffer -> existing page

copied = copy_page_from_iter(page, offset, chars, from);
buf->ops = &anon_pipe_buf_ops;

copied = copy_page_from_iter(page, offset, chars, from);
buf->ops = &anon_pipe_buf_ops;

    ordinary write()              splice()-style path
════════════════════════════════════════════════════════

    userspace buffer             file-backed page cache
  +-------------------+          +--------------------+
  | e.g. "hello pipe" |          | existing file page |
  +----------+--------+          +----------+---------+
             |                              |
             | copy_page_from_iter()        | reference page
             v                              v
 ┌─────────────────────┐          ┌─────────────────────┐
 │ anonymous pipe page │          │  file-backed page   │
 └───────────┬─────────┘          └─────────┬───────────┘
             |                              |
             | copied                       | zero-copy
             v                              v    
       +-----------------------------------------+
        |             struct pipe_buffer          |
        |-----------------------------------------|
        |                buf->page                |
        +-----------------------------------------+

    ordinary write()              splice()-style path
════════════════════════════════════════════════════════

    userspace buffer             file-backed page cache
  +-------------------+          +--------------------+
  | e.g. "hello pipe" |          | existing file page |
  +----------+--------+          +----------+---------+
             |                              |
             | copy_page_from_iter()        | reference page
             v                              v
 ┌─────────────────────┐          ┌─────────────────────┐
 │ anonymous pipe page │          │  file-backed page   │
 └───────────┬─────────┘          └─────────┬───────────┘
             |                              |
             | copied                       | zero-copy
             v                              v    
       +-----------------------------------------+
        |             struct pipe_buffer          |
        |-----------------------------------------|
        |                buf->page                |
        +-----------------------------------------+

file-backed page cache
        │
        │ splice(file -> pipe)
        ▼
pipe_buffer references file page
        │
        │ splice(pipe -> consumer)
        ▼
next kernel consumer receives page-backed data

file-backed page cache
        │
        │ splice(file -> pipe)
        ▼
pipe_buffer references file page
        │
        │ splice(pipe -> consumer)
        ▼
next kernel consumer receives page-backed data

SYSCALL_DEFINE6(splice, 
        int, fd_in, loff_t __user *, off_in,  // was userspace fd
        int, fd_out, loff_t __user *, off_out,
        size_t, len, unsigned int, flags)
        /* splice(int fd_in, ..., int fd_out, ...) */
{
    struct fd in, out;
    ssize_t error;

    ...

    in = fdget(fd_in);
    if (in.file) {
        out = fdget(fd_out);
        if (out.file) {
            /* 
             * transition 1
             * fd_in / fd_out are now kernel struct file objects
             */
            error = __do_splice(in.file, off_in, out.file, off_out,
                        len, flags);
            fdput(out);
        }
        fdput(in);
    }
    return error;
}

SYSCALL_DEFINE6(splice, 
        int, fd_in, loff_t __user *, off_in,  // was userspace fd
        int, fd_out, loff_t __user *, off_out,
        size_t, len, unsigned int, flags)
        /* splice(int fd_in, ..., int fd_out, ...) */
{
    struct fd in, out;
    ssize_t error;

    ...

    in = fdget(fd_in);
    if (in.file) {
        out = fdget(fd_out);
        if (out.file) {
            /* 
             * transition 1
             * fd_in / fd_out are now kernel struct file objects
             */
            error = __do_splice(in.file, off_in, out.file, off_out,
                        len, flags);
            fdput(out);
        }
        fdput(in);
    }
    return error;
}

static ssize_t __do_splice(struct file *in, loff_t __user *off_in,
                           struct file *out, loff_t __user *off_out,
                           size_t len, unsigned int flags)
{
    struct pipe_inode_info *ipipe;
    struct pipe_inode_info *opipe;
        loff_t offset, *__off_in = NULL, *__off_out = NULL;
    ssize_t ret;

    // Detect whether the input or output endpoint is a pipe-
    ipipe = get_pipe_info(in, true);
    opipe = get_pipe_info(out, true);

    ...

    // transition 2
    return do_splice(in, __off_in, out, __off_out, len, flags);
}

static ssize_t __do_splice(struct file *in, loff_t __user *off_in,
                           struct file *out, loff_t __user *off_out,
                           size_t len, unsigned int flags)
{
    struct pipe_inode_info *ipipe;
    struct pipe_inode_info *opipe;
        loff_t offset, *__off_in = NULL, *__off_out = NULL;
    ssize_t ret;

    // Detect whether the input or output endpoint is a pipe-
    ipipe = get_pipe_info(in, true);
    opipe = get_pipe_info(out, true);

    ...

    // transition 2
    return do_splice(in, __off_in, out, __off_out, len, flags);
}

ssize_t do_splice(struct file *in, loff_t *off_in,
                  struct file *out, loff_t *off_out,
                  size_t len, unsigned int flags)
{
    ...

    ipipe = get_pipe_info(in, true);
    opipe = get_pipe_info(out, true);

    // if both are pipes
    if (ipipe && opipe) {
        ret = splice_pipe_to_pipe(ipipe, opipe, len, flags);

    // if only input is pipe
    } else if (ipipe) {
        ret = do_splice_from(ipipe, out, &offset, len, flags);

    // if only output is pipe
    } else if (opipe) {
        /*
         * transition 3
         *
         * Copy Fail first-stage direction:
         *
         *     regular file -> pipe
         */
        ret = splice_file_to_pipe(in, opipe, &offset, len, flags);

    } else {
        ret = -EINVAL;
    }

    ...

    return ret;
}

ssize_t do_splice(struct file *in, loff_t *off_in,
                  struct file *out, loff_t *off_out,
                  size_t len, unsigned int flags)
{
    ...

    ipipe = get_pipe_info(in, true);
    opipe = get_pipe_info(out, true);

    // if both are pipes
    if (ipipe && opipe) {
        ret = splice_pipe_to_pipe(ipipe, opipe, len, flags);

    // if only input is pipe
    } else if (ipipe) {
        ret = do_splice_from(ipipe, out, &offset, len, flags);

    // if only output is pipe
    } else if (opipe) {
        /*
         * transition 3
         *
         * Copy Fail first-stage direction:
         *
         *     regular file -> pipe
         */
        ret = splice_file_to_pipe(in, opipe, &offset, len, flags);

    } else {
        ret = -EINVAL;
    }

    ...

    return ret;
}

ssize_t splice_file_to_pipe(struct file *in,
                            struct pipe_inode_info *opipe,
                            loff_t *offset,
                            size_t len, unsigned int flags)
{
    ssize_t ret;

    pipe_lock(opipe);
    ret = wait_for_space(opipe, flags);
    if (!ret)
        // transition 4
        ret = do_splice_read(in, offset, opipe, len, flags);
    pipe_unlock(opipe);
    ..
}

ssize_t splice_file_to_pipe(struct file *in,
                            struct pipe_inode_info *opipe,
                            loff_t *offset,
                            size_t len, unsigned int flags)
{
    ssize_t ret;

    pipe_lock(opipe);
    ret = wait_for_space(opipe, flags);
    if (!ret)
        // transition 4
        ret = do_splice_read(in, offset, opipe, len, flags);
    pipe_unlock(opipe);
    ..
}

static ssize_t do_splice_read(struct file *in, loff_t *ppos,
                              struct pipe_inode_info *pipe, size_t len,
                              unsigned int flags)
    {

    ...

    /*
     * O_DIRECT and DAX don't deal with the pagecache, so we allocate a
     * buffer, copy into it and splice that into the pipe.
     */
    if ((in->f_flags & O_DIRECT) || IS_DAX(in->f_mapping->host))
        // O_DIRECT and DAX bypass normal page cache
        // so kernel cannot perform normal page-cache-based splice operations
        // but perform an explicit buffered copy instead
        return copy_splice_read(in, ppos, pipe, len, flags);

    /* 
     * transition 5
     * normal buffered files use the file's splice_read handler
     */
    return in->f_op->splice_read(in, ppos, pipe, len, flags);
}

static ssize_t do_splice_read(struct file *in, loff_t *ppos,
                              struct pipe_inode_info *pipe, size_t len,
                              unsigned int flags)
    {

    ...

    /*
     * O_DIRECT and DAX don't deal with the pagecache, so we allocate a
     * buffer, copy into it and splice that into the pipe.
     */
    if ((in->f_flags & O_DIRECT) || IS_DAX(in->f_mapping->host))
        // O_DIRECT and DAX bypass normal page cache
        // so kernel cannot perform normal page-cache-based splice operations
        // but perform an explicit buffered copy instead
        return copy_splice_read(in, ppos, pipe, len, flags);

    /* 
     * transition 5
     * normal buffered files use the file's splice_read handler
     */
    return in->f_op->splice_read(in, ppos, pipe, len, flags);
}

return in->f_op->splice_read(...);  

return in->f_op->splice_read(...);  

struct file_operations {
    ...

    ssize_t (*splice_write)(struct pipe_inode_info *, struct file *, loff_t *, size_t, unsigned int);
    ssize_t (*splice_read)(struct file *, loff_t *, struct pipe_inode_info *, size_t, unsigned int);
    ...

} __randomize_layout;

struct file_operations {
    ...

    ssize_t (*splice_write)(struct pipe_inode_info *, struct file *, loff_t *, size_t, unsigned int);
    ssize_t (*splice_read)(struct file *, loff_t *, struct pipe_inode_info *, size_t, unsigned int);
    ...

} __randomize_layout;

splice(fd_file, ..., fd_pipe, ...)
        |
        v
  do_splice()
        |
        v
  splice_file_to_pipe()
        |
        v
  do_splice_read()
        |
        v
  in->f_op->splice_read(...)

splice(fd_file, ..., fd_pipe, ...)
        |
        v
  do_splice()
        |
        v
  splice_file_to_pipe()
        |
        v
  do_splice_read()
        |
        v
  in->f_op->splice_read(...)

const struct file_operations generic_ro_fops = {
    .read_iter    = generic_file_read_iter,
    .mmap         = generic_file_readonly_mmap,
    .splice_read  = filemap_splice_read,        // [!]
};

const struct file_operations generic_ro_fops = {
    .read_iter    = generic_file_read_iter,
    .mmap         = generic_file_readonly_mmap,
    .splice_read  = filemap_splice_read,        // [!]
};

/*
 * filemap_splice_read -  Splice data from a file's pagecache into a pipe
 *
 * This function gets folios from a file's pagecache and splices them into the
 * pipe.
 */

/*
 * filemap_splice_read -  Splice data from a file's pagecache into a pipe
 *
 * This function gets folios from a file's pagecache and splices them into the
 * pipe.
 */

ssize_t filemap_splice_read(struct file *in, loff_t *ppos,
                            struct pipe_inode_info *pipe,
                            size_t len, unsigned int flags)
{
    struct folio_batch fbatch;
    ...

    do {

        // retrieve page-cache folios from the file mapping
        error = filemap_get_pages(&iocb, len, &fbatch, true);

        ...

        for (i = 0; i < folio_batch_count(&fbatch); i++) {
            struct folio *folio = fbatch.folios[i];
            size_t n;

            ...

            /*
             * transition 6
             * insert the selected cached folio into the pipe
             */            
            n = splice_folio_into_pipe(pipe, folio, *ppos, n); 
            ...
        }

        folio_batch_release(&fbatch);
    } while (len);

    ...

    return total_spliced ? total_spliced : error;
}

ssize_t filemap_splice_read(struct file *in, loff_t *ppos,
                            struct pipe_inode_info *pipe,
                            size_t len, unsigned int flags)
{
    struct folio_batch fbatch;
    ...

    do {

        // retrieve page-cache folios from the file mapping
        error = filemap_get_pages(&iocb, len, &fbatch, true);

        ...

        for (i = 0; i < folio_batch_count(&fbatch); i++) {
            struct folio *folio = fbatch.folios[i];
            size_t n;

            ...

            /*
             * transition 6
             * insert the selected cached folio into the pipe
             */            
            n = splice_folio_into_pipe(pipe, folio, *ppos, n); 
            ...
        }

        folio_batch_release(&fbatch);
    } while (len);

    ...

    return total_spliced ? total_spliced : error;
}

/*
 * Splice subpages from a folio into a pipe.
 */
size_t splice_folio_into_pipe(struct pipe_inode_info *pipe,
                          struct folio *folio, loff_t fpos, size_t size)
{
    struct page *page;
    size_t spliced = 0, offset = offset_in_folio(folio, fpos);

    ...

    while (spliced < size &&
           !pipe_full(pipe->head, pipe->tail, pipe->max_usage)) {
        // retrieve the next pipe_buffer slot inside the pipe ring buffer
        struct pipe_buffer *buf = pipe_head_buf(pipe);
        size_t part = min_t(size_t, PAGE_SIZE - offset, size - spliced);

        // [!] insertion: attach file-backed page -> retrieved pipe buffer
        *buf = (struct pipe_buffer) {
            .ops    = &page_cache_pipe_buf_ops,  // [!] page cache marker
            .page   = page,
            .offset = offset,
            .len    = part,
        };
            ...
    }

    return spliced;
}

/*
 * Splice subpages from a folio into a pipe.
 */
size_t splice_folio_into_pipe(struct pipe_inode_info *pipe,
                          struct folio *folio, loff_t fpos, size_t size)
{
    struct page *page;
    size_t spliced = 0, offset = offset_in_folio(folio, fpos);

    ...

    while (spliced < size &&
           !pipe_full(pipe->head, pipe->tail, pipe->max_usage)) {
        // retrieve the next pipe_buffer slot inside the pipe ring buffer
        struct pipe_buffer *buf = pipe_head_buf(pipe);
        size_t part = min_t(size_t, PAGE_SIZE - offset, size - spliced);

        // [!] insertion: attach file-backed page -> retrieved pipe buffer
        *buf = (struct pipe_buffer) {
            .ops    = &page_cache_pipe_buf_ops,  // [!] page cache marker
            .page   = page,
            .offset = offset,
            .len    = part,
        };
            ...
    }

    return spliced;
}

// install operations for page-cache buffer
const struct pipe_buf_operations page_cache_pipe_buf_ops = {
    .confirm    = page_cache_pipe_buf_confirm,
    .release    = page_cache_pipe_buf_release,
    .try_steal  = page_cache_pipe_buf_try_steal,
    .get        = generic_pipe_buf_get,
};

// install operations for page-cache buffer
const struct pipe_buf_operations page_cache_pipe_buf_ops = {
    .confirm    = page_cache_pipe_buf_confirm,
    .release    = page_cache_pipe_buf_release,
    .try_steal  = page_cache_pipe_buf_try_steal,
    .get        = generic_pipe_buf_get,
};

file on disk (e.g. /usr/bin/su)
        │
        ▼
page cache folio/page
        │
        │  filemap_splice_read()
        ▼
pipe_buffer {
    .ops  = page_cache_pipe_buf_ops
    .page = file-backed page-cache page [!]
}

file on disk (e.g. /usr/bin/su)
        │
        ▼
page cache folio/page
        │
        │  filemap_splice_read()
        ▼
pipe_buffer {
    .ops  = page_cache_pipe_buf_ops
    .page = file-backed page-cache page [!]
}

/**
 * __splice_from_pipe - splice data from a pipe to given actor
 *
 * Description:
 *    This function does little more than loop over the pipe and call
 *    @actor to do the actual moving of a single struct pipe_buffer to
 *    the desired destination. See pipe_to_file, pipe_to_sendmsg, or
 *    pipe_to_user.
 */

/**
 * __splice_from_pipe - splice data from a pipe to given actor
 *
 * Description:
 *    This function does little more than loop over the pipe and call
 *    @actor to do the actual moving of a single struct pipe_buffer to
 *    the desired destination. See pipe_to_file, pipe_to_sendmsg, or
 *    pipe_to_user.
 */

/**
 * splice_to_socket - splice data from a pipe to a socket
 *
 * Description:
 *    Will send @len bytes from the pipe to a network socket. No data copying
 *    is involved.
 */

/**
 * splice_to_socket - splice data from a pipe to a socket
 *
 * Description:
 *    Will send @len bytes from the pipe to a network socket. No data copying
 *    is involved.
 */

struct bio_vec bvec[16];
struct msghdr msg = {};
...

struct pipe_buffer *buf = pipe_buf(pipe, tail);

...

bvec_set_page(
    &bvec[bc++],
    buf->page,     // page referenced by pipe_buffer 
    seg,           // byte length
    buf->offset    // offset inside that page
);

struct bio_vec bvec[16];
struct msghdr msg = {};
...

struct pipe_buffer *buf = pipe_buf(pipe, tail);

...

bvec_set_page(
    &bvec[bc++],
    buf->page,     // page referenced by pipe_buffer 
    seg,           // byte length
    buf->offset    // offset inside that page
);

struct bio_vec {
    struct page *bv_page;
    unsigned int    bv_len;
    unsigned int    bv_offset;
};

struct bio_vec {
    struct page *bv_page;
    unsigned int    bv_len;
    unsigned int    bv_offset;
};

pipe_buffer
    page   = cached file page
    offset = offset inside that page
    len    = visible byte range

        │
        ▼

bio_vec
    bv_page   = same cached file page
    bv_offset = same offset
    bv_len    = selected segment length

pipe_buffer
    page   = cached file page
    offset = offset inside that page
    len    = visible byte range

        │
        ▼

bio_vec
    bv_page   = same cached file page
    bv_offset = same offset
    bv_len    = selected segment length

// make msg_iter walk over bio_vec-backed pages
iov_iter_bvec(
    &msg.msg_iter,  // describes the payload being sent
    ITER_SOURCE,
    bvec,
    bc,
    len
);

msg.msg_flags = MSG_SPLICE_PAGES;  // [!] mark this send as splice-backed page data

ret = sock_sendmsg(sock, &msg);

// make msg_iter walk over bio_vec-backed pages
iov_iter_bvec(
    &msg.msg_iter,  // describes the payload being sent
    ITER_SOURCE,
    bvec,
    bc,
    len
);

msg.msg_flags = MSG_SPLICE_PAGES;  // [!] mark this send as splice-backed page data

ret = sock_sendmsg(sock, &msg);

pipe_buffer
    |
    | buf->page / buf->offset / buf->len
    v
  bio_vec
    |
    | iov_iter_bvec()
    v
msghdr.msg_iter
    |
    | MSG_SPLICE_PAGES
    v
sock_sendmsg()

pipe_buffer
    |
    | buf->page / buf->offset / buf->len
    v
  bio_vec
    |
    | iov_iter_bvec()
    v
msghdr.msg_iter
    |
    | MSG_SPLICE_PAGES
    v
sock_sendmsg()

file-backed cached page
        |
        v
   pipe_buffer
        |
        v
     bio_vec
        |
        v
msg.msg_iter + MSG_SPLICE_PAGES
        |
        v
socket-side consumer

file-backed cached page
        |
        v
   pipe_buffer
        |
        v
     bio_vec
        |
        v
msg.msg_iter + MSG_SPLICE_PAGES
        |
        v
socket-side consumer

file-backed page
    |
    v
pipe buffer
    |
    v
socket message
    |
    v
AF_ALG crypto request buffer

file-backed page
    |
    v
pipe buffer
    |
    v
socket message
    |
    v
AF_ALG crypto request buffer

socket() 
    |
    v 
bind algorithm type/name  
    |
    v
setsockopt configuration  
    |
    v
accept operation socket  
    |
    v
sendmsg input  
    |
    v
recvmsg output

socket() 
    |
    v 
bind algorithm type/name  
    |
    v
setsockopt configuration  
    |
    v
accept operation socket  
    |
    v
sendmsg input  
    |
    v
recvmsg output

#include <stdio.h>
#include <sys/socket.h>
#include <linux/if_alg.h>
#include <unistd.h>
#include <string.h>

int main(void)
{
    /* create AF_ALG control socket */
    int tfmfd = socket(AF_ALG, SOCK_SEQPACKET, 0);

    // [!] selected algorithm
    struct sockaddr_alg sa = {
        .salg_family = AF_ALG,
        .salg_type   = "hash",
        .salg_name   = "sha256",
    };

    /* bind algorithm type/name */
    bind(
        tfmfd,                     // AF_ALG socket fd 
        (struct sockaddr *)&sa,    // sockaddr_alg 
        sizeof(sa)
    );

    /* create operation socket */
    int opfd = accept(tfmfd, NULL, 0);

    /* submit input buffer */
    send(opfd, "AAAA", 4, 0);

    /* receive resulting digest */
    unsigned char digest[32];

    recv(opfd, digest, sizeof(digest), 0);

    /* print digest */
    printf("sha256(\"AAAA\") = ");

    for (int i = 0; i < 32; i++)
        printf("%02x", digest[i]);

    printf("\n");

    close(opfd);
    close(tfmfd);

    return 0;
}

#include <stdio.h>
#include <sys/socket.h>
#include <linux/if_alg.h>
#include <unistd.h>
#include <string.h>

int main(void)
{
    /* create AF_ALG control socket */
    int tfmfd = socket(AF_ALG, SOCK_SEQPACKET, 0);

    // [!] selected algorithm
    struct sockaddr_alg sa = {
        .salg_family = AF_ALG,
        .salg_type   = "hash",
        .salg_name   = "sha256",
    };

    /* bind algorithm type/name */
    bind(
        tfmfd,                     // AF_ALG socket fd 
        (struct sockaddr *)&sa,    // sockaddr_alg 
        sizeof(sa)
    );

    /* create operation socket */
    int opfd = accept(tfmfd, NULL, 0);

    /* submit input buffer */
    send(opfd, "AAAA", 4, 0);

    /* receive resulting digest */
    unsigned char digest[32];

    recv(opfd, digest, sizeof(digest), 0);

    /* print digest */
    printf("sha256(\"AAAA\") = ");

    for (int i = 0; i < 32; i++)
        printf("%02x", digest[i]);

    printf("\n");

    close(opfd);
    close(tfmfd);

    return 0;
}

struct sockaddr_alg {
    __u16 salg_family;
    __u8  salg_type[14];
    __u32 salg_feat;
    __u32 salg_mask;
    __u8  salg_name[];
};

struct sockaddr_alg {
    __u16 salg_family;
    __u8  salg_type[14];
    __u32 salg_feat;
    __u32 salg_mask;
    __u8  salg_name[];
};

.salg_type = "aead",
.salg_name = "authencesn(hmac(sha256),cbc(aes))",

.salg_type = "aead",
.salg_name = "authencesn(hmac(sha256),cbc(aes))",

userspace
    │
    │ socket(AF_ALG, SOCK_SEQPACKET, 0)
    │
    │ bind(
    │   type = "aead",
    │   name = "authencesn(hmac(sha256),cbc(aes))"
    │ )
    ▼
kernel AF_ALG layer
    │
    │ resolve family + algorithm
    ▼
AEAD operation socket
    │
    │ sendmsg() / recvmsg()
    ▼
selected AEAD implementation

userspace
    │
    │ socket(AF_ALG, SOCK_SEQPACKET, 0)
    │
    │ bind(
    │   type = "aead",
    │   name = "authencesn(hmac(sha256),cbc(aes))"
    │ )
    ▼
kernel AF_ALG layer
    │
    │ resolve family + algorithm
    ▼
AEAD operation socket
    │
    │ sendmsg() / recvmsg()
    ▼
selected AEAD implementation

struct sockaddr_alg sa = {
    .salg_family = AF_ALG,
    .salg_type   = "aead",
    .salg_name   = "authencesn(hmac(sha256),cbc(aes))",
};

bind(tfmfd, (struct sockaddr *)&sa, sizeof(sa));

struct sockaddr_alg sa = {
    .salg_family = AF_ALG,
    .salg_type   = "aead",
    .salg_name   = "authencesn(hmac(sha256),cbc(aes))",
};

bind(tfmfd, (struct sockaddr *)&sa, sizeof(sa));

err = READ_ONCE(sock->ops)->bind(
    sock,
    (struct sockaddr *)&address,
    addrlen
);

err = READ_ONCE(sock->ops)->bind(
    sock,
    (struct sockaddr *)&address,
    addrlen
);

type = alg_get_type(sa->salg_type);  // resolve AF_ALG interface family

...

private = type->bind(
    sa->salg_name,
    sa->salg_feat,
    sa->salg_mask
);

type = alg_get_type(sa->salg_type);  // resolve AF_ALG interface family

...

private = type->bind(
    sa->salg_name,
    sa->salg_feat,
    sa->salg_mask
);

sa->salg_type = "aead"

sa->salg_type = "aead"

struct af_alg_type {
    /* instantiate concrete algorithm by name */
    void *(*bind)(const char *name, u32 type, u32 mask);

    /* release family-private algorithm state */
    void (*release)(void *private);

    /* configure key material */
    int (*setkey)(void *private, const u8 *key, unsigned int keylen);

    /* configure AEAD authentication tag size */
    int (*setauthsize)(void *private, unsigned int authsize);

    /* create accepted operation socket */
    int (*accept)(void *private, struct sock *sk);
    int (*accept_nokey)(void *private, struct sock *sk);

    /* socket operations exposed by accepted operation socket */
    struct proto_ops *ops;
    struct proto_ops *ops_nokey;

    struct module *owner;

    /* AF_ALG family name, e.g. "aead" */
    char name[14];
};

struct af_alg_type {
    /* instantiate concrete algorithm by name */
    void *(*bind)(const char *name, u32 type, u32 mask);

    /* release family-private algorithm state */
    void (*release)(void *private);

    /* configure key material */
    int (*setkey)(void *private, const u8 *key, unsigned int keylen);

    /* configure AEAD authentication tag size */
    int (*setauthsize)(void *private, unsigned int authsize);

    /* create accepted operation socket */
    int (*accept)(void *private, struct sock *sk);
    int (*accept_nokey)(void *private, struct sock *sk);

    /* socket operations exposed by accepted operation socket */
    struct proto_ops *ops;
    struct proto_ops *ops_nokey;

    struct module *owner;

    /* AF_ALG family name, e.g. "aead" */
    char name[14];
};

static const struct af_alg_type algif_type_aead = {
    .bind          = aead_bind,
    .release       = aead_release,

    .setkey        = aead_setkey,
    .setauthsize   = aead_setauthsize,

    .accept        = aead_accept_parent,
    .accept_nokey  = aead_accept_parent_nokey,

    .ops           = &algif_aead_ops,
    .ops_nokey     = &algif_aead_ops_nokey,

    .name          = "aead",
    .owner         = THIS_MODULE
};

static const struct af_alg_type algif_type_aead = {
    .bind          = aead_bind,
    .release       = aead_release,

    .setkey        = aead_setkey,
    .setauthsize   = aead_setauthsize,

    .accept        = aead_accept_parent,
    .accept_nokey  = aead_accept_parent_nokey,

    .ops           = &algif_aead_ops,
    .ops_nokey     = &algif_aead_ops_nokey,

    .name          = "aead",
    .owner         = THIS_MODULE
};

type->bind(...)
        |
        v
aead_bind("authencesn(hmac(sha256), cbc(aes))", ...)

type->bind(...)
        |
        v
aead_bind("authencesn(hmac(sha256), cbc(aes))", ...)

bind(tfmfd, sockaddr_alg)
        |
        v
   alg_bind()
        |
        | salg_type = "aead"
        v
alg_get_type("aead")
        |
        v
algif_type_aead
        |
        | salg_name = "authencesn(hmac(sha256),cbc(aes))"
        v
aead_bind(...)

bind(tfmfd, sockaddr_alg)
        |
        v
   alg_bind()
        |
        | salg_type = "aead"
        v
alg_get_type("aead")
        |
        v
algif_type_aead
        |
        | salg_name = "authencesn(hmac(sha256),cbc(aes))"
        v
aead_bind(...)

setsockopt(..., ALG_SET_KEY, ...)
        |
        v
aead_setkey()


setsockopt(..., ALG_SET_AEAD_AUTHSIZE, ...)
        |
        v
aead_setauthsize()


accept(tfmfd, ...)
        |
        v
aead_accept_parent()


sendmsg(opfd, ...) / recvmsg(opfd, ...)
        |
        v
algif_aead_ops

setsockopt(..., ALG_SET_KEY, ...)
        |
        v
aead_setkey()


setsockopt(..., ALG_SET_AEAD_AUTHSIZE, ...)
        |
        v
aead_setauthsize()


accept(tfmfd, ...)
        |
        v
aead_accept_parent()


sendmsg(opfd, ...) / recvmsg(opfd, ...)
        |
        v
algif_aead_ops

static struct proto_ops algif_aead_ops = {
    ...
    .sendmsg = aead_sendmsg,
    .recvmsg = aead_recvmsg,
    ...
};

static struct proto_ops algif_aead_ops = {
    ...
    .sendmsg = aead_sendmsg,
    .recvmsg = aead_recvmsg,
    ...
};

userspace bind()
        |
        v
generic AF_ALG bind handler
        |
        v
resolve family by salg_type
        |
        v
instantiate algorithm by salg_name
        |
        v
install family-specific behavior for later 
setsockopt(), accept(), sendmsg(), recvmsg()

userspace bind()
        |
        v
generic AF_ALG bind handler
        |
        v
resolve family by salg_type
        |
        v
instantiate algorithm by salg_name
        |
        v
install family-specific behavior for later 
setsockopt(), accept(), sendmsg(), recvmsg()

// request AF_ALG socket returning control socket tfmfd
tfmfd = socket(AF_ALG, SOCK_SEQPACKET, 0)  

bind(tfmfd, ...)
setsockopt(tfmfd, ...)

// request AF_ALG socket returning control socket tfmfd
tfmfd = socket(AF_ALG, SOCK_SEQPACKET, 0)  

bind(tfmfd, ...)
setsockopt(tfmfd, ...)

int opfd = accept(tfmfd, NULL, 0);  // accept() returns operation socket

int opfd = accept(tfmfd, NULL, 0);  // accept() returns operation socket

const struct af_alg_type *type;

/*
 * newsock->ops assigned here to allow type->accept call to override
 * them when required.
 */
newsock->ops = type->ops;

const struct af_alg_type *type;

/*
 * newsock->ops assigned here to allow type->accept call to override
 * them when required.
 */
newsock->ops = type->ops;

static struct proto_ops algif_aead_ops = {
    ...
    .sendmsg = aead_sendmsg,
    .recvmsg = aead_recvmsg,
    ...
};

static struct proto_ops algif_aead_ops = {
    ...
    .sendmsg = aead_sendmsg,
    .recvmsg = aead_recvmsg,
    ...
};

control socket
════════════════════════════════════════

tfmfd
  ├─ bind()       -> select algorithm
  └─ setsockopt() -> configure transform


operation socket
════════════════════════════════════════

accept(tfmfd)
        │
        ▼
opfd
  ├─ sendmsg() -> aead_sendmsg()
  └─ recvmsg() -> aead_recvmsg()

control socket
════════════════════════════════════════

tfmfd
  ├─ bind()       -> select algorithm
  └─ setsockopt() -> configure transform


operation socket
════════════════════════════════════════

accept(tfmfd)
        │
        ▼
opfd
  ├─ sendmsg() -> aead_sendmsg()
  └─ recvmsg() -> aead_recvmsg()

input bytes -> digest bytes

input bytes -> digest bytes

static int aead_sendmsg(
    struct socket *sock,
    struct msghdr *msg,   
    size_t size)
{
    return af_alg_sendmsg(sock, msg, size, ivsize);
}

static int aead_sendmsg(
    struct socket *sock,
    struct msghdr *msg,   
    size_t size)
{
    return af_alg_sendmsg(sock, msg, size, ivsize);
}

int af_alg_sendmsg(struct socket *sock, struct msghdr *msg, size_t size,
                   unsigned int ivsize)
{

    ...

    while ((cmsg = af_alg_cmsg_send(msg, &con)) != NULL) {
        switch (cmsg->cmsg_type) {

        /* encrypt/decrypt selector */
        case ALG_SET_OP:
            ctx->op = *(u32 *)CMSG_DATA(cmsg);
            break;

        /* AEAD associated-data (AAD) length */
        case ALG_SET_AEAD_ASSOCLEN:
            /*
             * Number of bytes at the start of the input
             * that should be treated as AAD.
             */
            ctx->aead_assoclen = af_alg_control_aead(cmsg);
            break;

        ...

int af_alg_sendmsg(struct socket *sock, struct msghdr *msg, size_t size,
                   unsigned int ivsize)
{

    ...

    while ((cmsg = af_alg_cmsg_send(msg, &con)) != NULL) {
        switch (cmsg->cmsg_type) {

        /* encrypt/decrypt selector */
        case ALG_SET_OP:
            ctx->op = *(u32 *)CMSG_DATA(cmsg);
            break;

        /* AEAD associated-data (AAD) length */
        case ALG_SET_AEAD_ASSOCLEN:
            /*
             * Number of bytes at the start of the input
             * that should be treated as AAD.
             */
            ctx->aead_assoclen = af_alg_control_aead(cmsg);
            break;

        ...

iov_iter_extract_will_pin(&msg->msg_iter);
len = af_alg_make_sg(&ctx->sgl, &msg->msg_iter, 0, len);

iov_iter_extract_will_pin(&msg->msg_iter);
len = af_alg_make_sg(&ctx->sgl, &msg->msg_iter, 0, len);

sendmsg(opfd, msghdr)
        │
        ▼
aead_sendmsg()
        │
        ▼
af_alg_sendmsg()
        │
        ├─ control messages
        │      ├─ operation direction
        │      ├─ IV
        │      └─ AAD length
        │
        └─ msg_iter payload
               │
               ▼
        scatter-gather buffers

sendmsg(opfd, msghdr)
        │
        ▼
aead_sendmsg()
        │
        ▼
af_alg_sendmsg()
        │
        ├─ control messages
        │      ├─ operation direction
        │      ├─ IV
        │      └─ AAD length
        │
        └─ msg_iter payload
               │
               ▼
        scatter-gather buffers

AAD || payload || optional tag

AAD || payload || optional tag

encrypt:
    input  = AAD || plaintext
    output = AAD || ciphertext || tag

decrypt:
    input  = AAD || ciphertext || tag
    output = AAD || plaintext

encrypt:
    input  = AAD || plaintext
    output = AAD || ciphertext || tag

decrypt:
    input  = AAD || ciphertext || tag
    output = AAD || plaintext

static int _aead_recvmsg(
    struct socket *sock,
    struct msghdr *msg,
    size_t ignored,
    int flags)
{
    ...

    /* AEAD authentication tag size */
    unsigned int as = crypto_aead_authsize(tfm);

    ...

    /*
     * Total bytes queued earlier through sendmsg().
     *
     * Encrypt input:
     *     AAD || plaintext
     *
     * Decrypt input:
     *     AAD || ciphertext || tag
     */
    used = ctx->used;

    /*
     * outlen is the size of the recvmsg-side output buffer needed.
     *
     * Encrypt:
     *     output = AAD || ciphertext || tag
     *     outlen = input + tag
     *
     * Decrypt:
     *     output = AAD || plaintext
     *     outlen = input - tag
     */
    if (ctx->enc)
        outlen = used + as;  // 1 encryption
    else
        outlen = used - as;  // 2 decryption

    /*  
    * Rebase "used" from total input length to crypto payload length.  
    *  
    * AAD is not encrypted/decrypted, so it is removed here and stored  
    * separately as req->assoclen through aead_request_set_ad().  
    *  
    * After this:  
    *  
    * Encrypt:  
    * used = plaintext length  
    *  
    * Decrypt:  
    * used = ciphertext || tag length  
    */
    used -= ctx->aead_assoclen;

    ...
}

static int _aead_recvmsg(
    struct socket *sock,
    struct msghdr *msg,
    size_t ignored,
    int flags)
{
    ...

    /* AEAD authentication tag size */
    unsigned int as = crypto_aead_authsize(tfm);

    ...

    /*
     * Total bytes queued earlier through sendmsg().
     *
     * Encrypt input:
     *     AAD || plaintext
     *
     * Decrypt input:
     *     AAD || ciphertext || tag
     */
    used = ctx->used;

    /*
     * outlen is the size of the recvmsg-side output buffer needed.
     *
     * Encrypt:
     *     output = AAD || ciphertext || tag
     *     outlen = input + tag
     *
     * Decrypt:
     *     output = AAD || plaintext
     *     outlen = input - tag
     */
    if (ctx->enc)
        outlen = used + as;  // 1 encryption
    else
        outlen = used - as;  // 2 decryption

    /*  
    * Rebase "used" from total input length to crypto payload length.  
    *  
    * AAD is not encrypted/decrypted, so it is removed here and stored  
    * separately as req->assoclen through aead_request_set_ad().  
    *  
    * After this:  
    *  
    * Encrypt:  
    * used = plaintext length  
    *  
    * Decrypt:  
    * used = ciphertext || tag length  
    */
    used -= ctx->aead_assoclen;

    ...
}

AEAD decrypt contract
════════════════════════════════════════════════════════════

input queued through sendmsg():
┌────────────┬────────────────────┬────────────┐
│    AAD     │     ciphertext     │    tag     │
└────────────┴────────────────────┴────────────┘
0        assoclen          assoclen+ctlen     +authsize

                  decrypts to      verifies only
                       │                 │
                       ▼                 ×

output prepared for recvmsg():
┌────────────┬────────────────────┐
│    AAD     │     plaintext      │
└────────────┴────────────────────┘
0        assoclen          assoclen+ptlen
                                  ▲
                                  │
                         valid output boundary

AEAD decrypt contract
════════════════════════════════════════════════════════════

input queued through sendmsg():
┌────────────┬────────────────────┬────────────┐
│    AAD     │     ciphertext     │    tag     │
└────────────┴────────────────────┴────────────┘
0        assoclen          assoclen+ctlen     +authsize

                  decrypts to      verifies only
                       │                 │
                       ▼                 ×

output prepared for recvmsg():
┌────────────┬────────────────────┐
│    AAD     │     plaintext      │
└────────────┴────────────────────┘
0        assoclen          assoclen+ptlen
                                  ▲
                                  │
                         valid output boundary

logical crypto buffer
════════════════════════════════════════════════════════════

0                                                        end
│               [logical byte stream]                     │
▼                                                         ▼
┌───────────────┬──────────────────┬──────────────────────┐
│      AAD      │    ciphertext    │         tag          │
└───────────────┴──────────────────┴──────────────────────┘

logical crypto buffer
════════════════════════════════════════════════════════════

0                                                        end
│               [logical byte stream]                     │
▼                                                         ▼
┌───────────────┬──────────────────┬──────────────────────┐
│      AAD      │    ciphertext    │         tag          │
└───────────────┴──────────────────┴──────────────────────┘

scatterlist backing
════════════════════════════════════════════════════════════

┌───────────────┐ ┌───────────────┐ ┌──────────────────────┐
│ sg entry #0   │ │ sg entry #1   │ │ sg entry #2          │
│ user page     │ │ user page     │ │ file-backed page     │
└───────┬───────┘ └──────┬────────┘ └────────┬─────────────┘
        │                │                   │
        ▼                ▼                   ▼
┌───────────────┬──────────────────┬──────────────────────┐
│      AAD      │    ciphertext    │         tag          │
└───────────────┴──────────────────┴──────────────────────┘

scatterlist backing
════════════════════════════════════════════════════════════

┌───────────────┐ ┌───────────────┐ ┌──────────────────────┐
│ sg entry #0   │ │ sg entry #1   │ │ sg entry #2          │
│ user page     │ │ user page     │ │ file-backed page     │
└───────┬───────┘ └──────┬────────┘ └────────┬─────────────┘
        │                │                   │
        ▼                ▼                   ▼
┌───────────────┬──────────────────┬──────────────────────┐
│      AAD      │    ciphertext    │         tag          │
└───────────────┴──────────────────┴──────────────────────┘

struct scatterlist {
    unsigned long page_link;  // encoded page pointer + flags
    unsigned int  offset;     // byte offset inside the page
    unsigned int  length;     // number of bytes available
    dma_addr_t    dma_address;
    ...
};

struct scatterlist {
    unsigned long page_link;  // encoded page pointer + flags
    unsigned int  offset;     // byte offset inside the page
    unsigned int  length;     // number of bytes available
    dma_addr_t    dma_address;
    ...
};

/**
 * sg_set_page - Set sg entry to point at given page
 * @sg:         SG entry
 * @page:     The page
 * @len:     Length of data
 * @offset:     Offset into page
 *
 * Description:
 *   Use this function to set an sg entry pointing at a page, never assign
 *   the page directly. We encode sg table information in the lower bits
 *   of the page pointer. See sg_page() for looking up the page belonging
 *   to an sg entry.
 *
 **/
static inline void sg_set_page(struct scatterlist *sg, struct page *page,
                               unsigned int len, unsigned int offset)
{
    sg_assign_page(sg, page);  // attach backing page
    sg->offset = offset;       // start offset inside page
    sg->length = len;          // valid byte length
}

/**
 * sg_set_page - Set sg entry to point at given page
 * @sg:         SG entry
 * @page:     The page
 * @len:     Length of data
 * @offset:     Offset into page
 *
 * Description:
 *   Use this function to set an sg entry pointing at a page, never assign
 *   the page directly. We encode sg table information in the lower bits
 *   of the page pointer. See sg_page() for looking up the page belonging
 *   to an sg entry.
 *
 **/
static inline void sg_set_page(struct scatterlist *sg, struct page *page,
                               unsigned int len, unsigned int offset)
{
    sg_assign_page(sg, page);  // attach backing page
    sg->offset = offset;       // start offset inside page
    sg->length = len;          // valid byte length
}

#define SG_CHAIN    0x01UL
#define SG_END        0x02UL


static inline void sg_chain(
    struct scatterlist *prv,
    unsigned int prv_nents,
    struct scatterlist *sgl)
{
    __sg_chain(&prv[prv_nents - 1], sgl);
}


static inline void __sg_chain(struct scatterlist *chain_sg,
                              struct scatterlist *sgl)
{
    /*
     * offset and length are unused for chain entry. Clear them.
     */
    chain_sg->offset = 0;
    chain_sg->length = 0;

    /*
     * Set lowest bit to indicate a link pointer, and make sure to clear
     * the termination bit if it happens to be set.
     */
    chain_sg->page_link = ((unsigned long) sgl | SG_CHAIN) & ~SG_END;
}

#define SG_CHAIN    0x01UL
#define SG_END        0x02UL


static inline void sg_chain(
    struct scatterlist *prv,
    unsigned int prv_nents,
    struct scatterlist *sgl)
{
    __sg_chain(&prv[prv_nents - 1], sgl);
}


static inline void __sg_chain(struct scatterlist *chain_sg,
                              struct scatterlist *sgl)
{
    /*
     * offset and length are unused for chain entry. Clear them.
     */
    chain_sg->offset = 0;
    chain_sg->length = 0;

    /*
     * Set lowest bit to indicate a link pointer, and make sure to clear
     * the termination bit if it happens to be set.
     */
    chain_sg->page_link = ((unsigned long) sgl | SG_CHAIN) & ~SG_END;
}

scatterlist chain
═══════════════════════════════════

┌────────────────────────────┐
│ sg #0                      │
│ page   = user page A       │
│ offset = 0x120             │
│ length = 0x300             │
└──────────────┬─────────────┘
               │ sg_next()
               ▼
┌────────────────────────────┐
│ sg #1                      │
│ page   = user page B       │
│ offset = 0x000             │
│ length = 0x700             │
└──────────────┬─────────────┘
               │ sg_next()
               ▼
┌────────────────────────────┐
│ sg #2                      │
│ page   = file-backed page  │
│ offset = 0x200             │
│ length = 0x010             │
└────────────────────────────┘

scatterlist chain
═══════════════════════════════════

┌────────────────────────────┐
│ sg #0                      │
│ page   = user page A       │
│ offset = 0x120             │
│ length = 0x300             │
└──────────────┬─────────────┘
               │ sg_next()
               ▼
┌────────────────────────────┐
│ sg #1                      │
│ page   = user page B       │
│ offset = 0x000             │
│ length = 0x700             │
└──────────────┬─────────────┘
               │ sg_next()
               ▼
┌────────────────────────────┐
│ sg #2                      │
│ page   = file-backed page  │
│ offset = 0x200             │
│ length = 0x010             │
└────────────────────────────┘

void scatterwalk_map_and_copy(void *buf, struct scatterlist *sg,
                      unsigned int start, unsigned int nbytes, int out)
{
    struct scatter_walk walk;
    struct scatterlist tmp[2];

    // jump to the sg entry containing logical offset "start" */
    sg = scatterwalk_ffwd(tmp, sg, start);

    // start walking from that entry 
    scatterwalk_start(&walk, sg);

    // copy nbytes across one or more sg entries
    scatterwalk_copychunks(buf, &walk, nbytes, out);

        scatterwalk_done(&walk, out, 0);
}

void scatterwalk_map_and_copy(void *buf, struct scatterlist *sg,
                      unsigned int start, unsigned int nbytes, int out)
{
    struct scatter_walk walk;
    struct scatterlist tmp[2];

    // jump to the sg entry containing logical offset "start" */
    sg = scatterwalk_ffwd(tmp, sg, start);

    // start walking from that entry 
    scatterwalk_start(&walk, sg);

    // copy nbytes across one or more sg entries
    scatterwalk_copychunks(buf, &walk, nbytes, out);

        scatterwalk_done(&walk, out, 0);
}

static inline void scatterwalk_done(struct scatter_walk *walk, int out,
                                    int more)
{
    /*
     * Finish the current page/chunk when:
     *
     * - there is no more data to copy, or
     * - the current scatterlist entry is exhausted, or
     * - the walk reached a page boundary
     */
    if (!more ||
        walk->offset >= walk->sg->offset + walk->sg->length ||
        !(walk->offset & (PAGE_SIZE - 1)))
        scatterwalk_pagedone(walk, out, more);
}

static inline void scatterwalk_pagedone(struct scatter_walk *walk, int out,
                                        unsigned int more)
{
    /*
     * If data was written into the backing page,
     * flush the data cache for coherency.
     */
    if (out) {
        struct page *page;

        page = sg_page(walk->sg) +
               ((walk->offset - 1) >> PAGE_SHIFT);

        flush_dcache_page(page);
    }

    /*
     * If more bytes remain and the current scatterlist entry
     * has been fully consumed, move to the next entry:
     *
     *     current sg entry -> sg_next(current sg entry)
     */
    if (more &&
        walk->offset >= walk->sg->offset + walk->sg->length)
        scatterwalk_start(walk, sg_next(walk->sg));  // [!]
}

static inline void scatterwalk_done(struct scatter_walk *walk, int out,
                                    int more)
{
    /*
     * Finish the current page/chunk when:
     *
     * - there is no more data to copy, or
     * - the current scatterlist entry is exhausted, or
     * - the walk reached a page boundary
     */
    if (!more ||
        walk->offset >= walk->sg->offset + walk->sg->length ||
        !(walk->offset & (PAGE_SIZE - 1)))
        scatterwalk_pagedone(walk, out, more);
}

static inline void scatterwalk_pagedone(struct scatter_walk *walk, int out,
                                        unsigned int more)
{
    /*
     * If data was written into the backing page,
     * flush the data cache for coherency.
     */
    if (out) {
        struct page *page;

        page = sg_page(walk->sg) +
               ((walk->offset - 1) >> PAGE_SHIFT);

        flush_dcache_page(page);
    }

    /*
     * If more bytes remain and the current scatterlist entry
     * has been fully consumed, move to the next entry:
     *
     *     current sg entry -> sg_next(current sg entry)
     */
    if (more &&
        walk->offset >= walk->sg->offset + walk->sg->length)
        scatterwalk_start(walk, sg_next(walk->sg));  // [!]
}

scatterwalk_start(walk, sg_next(walk->sg));

scatterwalk_start(walk, sg_next(walk->sg));

current sg entry
        │
        │ sg_next()
        ▼
next sg entry
        │
        │ scatterwalk_start()
        ▼
copy continues there

current sg entry
        │
        │ sg_next()
        ▼
next sg entry
        │
        │ scatterwalk_start()
        ▼
copy continues there

sg #0                         sg #1
┌────────────────────────────┐ ┌────────────────────────────┐
│ AAD || plaintext           │ │ tag / scratch / next bytes │
└────────────────────────────┘ └────────────────────────────┘
                             │
                             └── sg_next(sg #0) 

sg #0                         sg #1
┌────────────────────────────┐ ┌────────────────────────────┐
│ AAD || plaintext           │ │ tag / scratch / next bytes │
└────────────────────────────┘ └────────────────────────────┘
                             │
                             └── sg_next(sg #0) 

destination scatterlist
══════════════════════════════════════════════════════════════

valid output area                  sg_next() chained entry
┌────────────────────────────┐    ┌────────────────────────────┐
│ sg #0                      │    │ sg #1                      │
│ legitimate AEAD output     │───▶│ page-cache-backed file page│
│ AAD || plaintext           │    │ must not receive writes    │
└────────────────────────────┘    └────────────────────────────┘
                             ▲
                             │
                   valid output ends here

destination scatterlist
══════════════════════════════════════════════════════════════

valid output area                  sg_next() chained entry
┌────────────────────────────┐    ┌────────────────────────────┐
│ sg #0                      │    │ sg #1                      │
│ legitimate AEAD output     │───▶│ page-cache-backed file page│
│ AAD || plaintext           │    │ must not receive writes    │
└────────────────────────────┘    └────────────────────────────┘
                             ▲
                             │
                   valid output ends here

AAD length (assoclen) + plaintext length

AAD length (assoclen) + plaintext length

AAD || ciphertext || tag

AAD || ciphertext || tag

AAD || plaintext

AAD || plaintext

/*
 * ...
 *
 * The following concept of the memory management is used:
 *
 * The kernel maintains two SGLs, the TX SGL and the RX SGL. The TX SGL is
 * filled by user space with the data submitted via sendmsg (maybe with
 * MSG_SPLICE_PAGES).  Filling up the TX SGL does not cause a crypto operation
 * -- the data will only be tracked by the kernel. Upon receipt of one recvmsg
 * call, the caller must provide a buffer which is tracked with the RX SGL.
 *
 * During the processing of the recvmsg operation, the cipher request is
 * allocated and prepared. As part of the recvmsg operation, the processed
 * TX buffers are extracted from the TX SGL into a separate SGL.
 *
 * ...
 */

/*
 * ...
 *
 * The following concept of the memory management is used:
 *
 * The kernel maintains two SGLs, the TX SGL and the RX SGL. The TX SGL is
 * filled by user space with the data submitted via sendmsg (maybe with
 * MSG_SPLICE_PAGES).  Filling up the TX SGL does not cause a crypto operation
 * -- the data will only be tracked by the kernel. Upon receipt of one recvmsg
 * call, the caller must provide a buffer which is tracked with the RX SGL.
 *
 * During the processing of the recvmsg operation, the cipher request is
 * allocated and prepared. As part of the recvmsg operation, the processed
 * TX buffers are extracted from the TX SGL into a separate SGL.
 *
 * ...
 */

struct aead_request {
    struct crypto_async_request base;

    unsigned int assoclen;    /* AAD length */
    unsigned int cryptlen;    /* payload length */

    u8 *iv;                   /* IV / nonce */

    struct scatterlist *src;  /* input scatterlist */
    struct scatterlist *dst;  /* output scatterlist */

    void *__ctx[] CRYPTO_MINALIGN_ATTR;
};

struct aead_request {
    struct crypto_async_request base;

    unsigned int assoclen;    /* AAD length */
    unsigned int cryptlen;    /* payload length */

    u8 *iv;                   /* IV / nonce */

    struct scatterlist *src;  /* input scatterlist */
    struct scatterlist *dst;  /* output scatterlist */

    void *__ctx[] CRYPTO_MINALIGN_ATTR;
};

AF_ALG socket layer                  crypto request layer
───────────────────                  ────────────────────

TX SGL  ─────────────────────────▶   req->src
data submitted by sendmsg()          input scatterlist

RX SGL  ─────────────────────────▶   req->dst
buffer supplied to recvmsg()         output scatterlist

AF_ALG socket layer                  crypto request layer
───────────────────                  ────────────────────

TX SGL  ─────────────────────────▶   req->src
data submitted by sendmsg()          input scatterlist

RX SGL  ─────────────────────────▶   req->dst
buffer supplied to recvmsg()         output scatterlist

sendmsg() data    ─►  socket TX SGL  ─►  req->src
recvmsg() buffer  ─►  socket RX SGL  ─►  req->dst

sendmsg() data    ─►  socket TX SGL  ─►  req->src
recvmsg() buffer  ─►  socket RX SGL  ─►  req->dst

input  = AAD || plaintext  
output = AAD || ciphertext || tag

input  = AAD || plaintext  
output = AAD || ciphertext || tag

if (ctx->enc) {
    /*
     * Encryption operation - The in-place cipher operation is
     * achieved by the following operation:
     *
     * TX SGL: AAD  ||  PT
     *          |       |
     *          | copy  |
     *          v       v
     * RX SGL: AAD  ||  PT  ||  Tag
     */

    /*
     * Step 1:
     * Copy AAD || plaintext from TX into RX.
     *
     * At this point, RX contains the input material and has
     * enough space for the tag that will be produced later.
     */
    err = crypto_aead_copy_sgl(
        null_tfm,
        tsgl_src,                        // source: queued TX input
        areq->first_rsgl.sgl.sgt.sgl,    // destination: RX output
        processed                        // bytes copied into RX
    );

    ...

    /*
     * Step 2:
     * Consume the TX entries that were copied.
     */
    af_alg_pull_tsgl(sk, processed, NULL, 0);

} else ...

if (ctx->enc) {
    /*
     * Encryption operation - The in-place cipher operation is
     * achieved by the following operation:
     *
     * TX SGL: AAD  ||  PT
     *          |       |
     *          | copy  |
     *          v       v
     * RX SGL: AAD  ||  PT  ||  Tag
     */

    /*
     * Step 1:
     * Copy AAD || plaintext from TX into RX.
     *
     * At this point, RX contains the input material and has
     * enough space for the tag that will be produced later.
     */
    err = crypto_aead_copy_sgl(
        null_tfm,
        tsgl_src,                        // source: queued TX input
        areq->first_rsgl.sgl.sgt.sgl,    // destination: RX output
        processed                        // bytes copied into RX
    );

    ...

    /*
     * Step 2:
     * Consume the TX entries that were copied.
     */
    af_alg_pull_tsgl(sk, processed, NULL, 0);

} else ...

/*
 * Later:
 * submit the prepared AEAD request.
 *
 * In the encryption case, the algorithm transforms plaintext into
 * ciphertext and writes the authentication tag into the RX destination.
 */
crypto_aead_encrypt(&areq->cra_u.aead_req);

/*
 * Later:
 * submit the prepared AEAD request.
 *
 * In the encryption case, the algorithm transforms plaintext into
 * ciphertext and writes the authentication tag into the RX destination.
 */
crypto_aead_encrypt(&areq->cra_u.aead_req);

encryption sgl setup
══════════════════════════════════════════════════════

TX SGL                           RX SGL
┌─────────┬───────────┐  copy   ┌───────┬───────────┬───────────┐
│   AAD   │ plaintext │ ──────▶ │  AAD  │ plaintext │ tag space │
└─────────┴───────────┘         └───────┴───────────┴───────────┘
                                   │           later becomes
                                   │          │           │ 
                                   │          │           │ written by
                                   │          │ encrypt   │ crypto_aead_encrypt()
                                   │          ▼           ▼
                                   │         ciphertext  tag
                                   │                       ▲   
                                   ▼                       │
                                  AAD                      │
                                   │                       │
                                   └──── authenticated ────┘

encryption sgl setup
══════════════════════════════════════════════════════

TX SGL                           RX SGL
┌─────────┬───────────┐  copy   ┌───────┬───────────┬───────────┐
│   AAD   │ plaintext │ ──────▶ │  AAD  │ plaintext │ tag space │
└─────────┴───────────┘         └───────┴───────────┴───────────┘
                                   │           later becomes
                                   │          │           │ 
                                   │          │           │ written by
                                   │          │ encrypt   │ crypto_aead_encrypt()
                                   │          ▼           ▼
                                   │         ciphertext  tag
                                   │                       ▲   
                                   ▼                       │
                                  AAD                      │
                                   │                       │
                                   └──── authenticated ────┘