Programming Large Language Models With Azure

Open Ai Conversational Programming And Prompt

Engineering With Llms Developer Reference 1
Converted Esposito download
https://ebookbell.com/product/programming-large-language-models-
with-azure-open-ai-conversational-programming-and-prompt-
engineering-with-llms-developer-reference-1-converted-
esposito-56106874

Explore and download more ebooks at ebookbell.com

Here are some recommended products that we believe you will be
interested in. You can click the link to download.

Programming Large Language Models With Azure Open Ai Conversational

Programming And Prompt Engineering With Llms Francesco Esposito

https://ebookbell.com/product/programming-large-language-models-with-
azure-open-ai-conversational-programming-and-prompt-engineering-with-
llms-francesco-esposito-56144398

Matlab Programming Advanced Data Analysis Visualisation And Largescale

Applications For Research And Development Mastering Programming
Languages Series Edet

https://ebookbell.com/product/matlab-programming-advanced-data-
analysis-visualisation-and-largescale-applications-for-research-and-
development-mastering-programming-languages-series-edet-63151948

Programming In The Large With Design Patterns Burris Eddie

https://ebookbell.com/product/programming-in-the-large-with-design-
patterns-burris-eddie-10786708

Bonus Algorithm For Large Scale Stochastic Nonlinear Programming

Problems Urmila Diwekar

https://ebookbell.com/product/bonus-algorithm-for-large-scale-
stochastic-nonlinear-programming-problems-urmila-diwekar-5034436
Cohesive Subgraph Computation Over Large Sparse Graphs Algorithms Data
Structures And Programming Techniques 1st Ed Lijun Chang

https://ebookbell.com/product/cohesive-subgraph-computation-over-
large-sparse-graphs-algorithms-data-structures-and-programming-
techniques-1st-ed-lijun-chang-7321324

Programming Languages And Systems 31st European Symposium On

Programming Esop 2022 Held As Part Of The European Joint Conferences
On Theory And Practice Of Software Etaps 2022 Munich Germany April 27
2022 Proceedings Ilya Sergey
https://ebookbell.com/product/programming-languages-and-systems-31st-
european-symposium-on-programming-esop-2022-held-as-part-of-the-
european-joint-conferences-on-theory-and-practice-of-software-
etaps-2022-munich-germany-april-27-2022-proceedings-ilya-
sergey-44887738

Programming 101 Learn To Code Using The Processing Programming

Language 2nd Edition 2nd Jeanine Meyer

https://ebookbell.com/product/programming-101-learn-to-code-using-the-
processing-programming-language-2nd-edition-2nd-jeanine-meyer-46238180

Programming 101 The How And Why Of Programming Revealed Using The
Processing Programming Language Jeanine Meyer

https://ebookbell.com/product/programming-101-the-how-and-why-of-
programming-revealed-using-the-processing-programming-language-
jeanine-meyer-46318424

Programming And Gui Fundamentals Tcltk For Electronic Design

Automation Suman Lata Tripathi

https://ebookbell.com/product/programming-and-gui-fundamentals-tcltk-
for-electronic-design-automation-suman-lata-tripathi-46318712
 Programming Large Language
Models with Azure Open AI:
Conversational programming and
prompt engineering with LLMs

Francesco Esposito
Programming Large Language Models with Azure Open AI:
Conversational programming and prompt engineering with
LLMs
Published with the authorization of Microsoft Corporation by: Pearson
Education, Inc.

Copyright © 2024 by Francesco Esposito.

All rights reserved. This publication is protected by copyright, and
permission must be obtained from the publisher prior to any prohibited
reproduction, storage in a retrieval system, or transmission in any form or by
any means, electronic, mechanical, photocopying, recording, or likewise. For
information regarding permissions, request forms, and the appropriate
contacts within the Pearson Education Global Rights & Permissions
Department, please visit www.pearson.com/permissions.
No patent liability is assumed with respect to the use of the information
contained herein. Although every precaution has been taken in the
preparation of this book, the publisher and author assume no responsibility
for errors or omissions. Nor is any liability assumed for damages resulting
from the use of the information contained herein.
ISBN-13: 978-0-13-828037-6
ISBN-10: 0-13-828037-1
Library of Congress Control Number: 2024931423
$PrintCode

Trademarks
Microsoft and the trademarks listed at http://www.microsoft.com on the
“Trademarks” webpage are trademarks of the Microsoft group of companies.
All other marks are property of their respective owners.

Warning and Disclaimer

Every effort has been made to make this book as complete and as accurate as
possible, but no warranty or fitness is implied. The information provided is
on an “as is” basis. The author, the publisher, and Microsoft Corporation
shall have neither liability nor responsibility to any person or entity with
respect to any loss or damages arising from the information contained in this
book or from the use of the programs accompanying it.

Special Sales
For information about buying this title in bulk quantities, or for special sales
opportunities (which may include electronic versions; custom cover designs;
and content particular to your business, training goals, marketing focus, or
branding interests), please contact our corporate sales department at
[email protected] or (800) 382-3419.
For government sales inquiries, please contact
[email protected].
For questions about sales outside the U.S., please contact
[email protected].

Editor-in-Chief
Brett Bartow

Executive Editor
Loretta Yates

Associate Editor
Shourav Bose

Development Editor
Kate Shoup

Managing Editor
Sandra Schroeder

Senior Project Editor

Tracey Croom

Copy Editor
Dan Foster

Indexer
Timothy Wright

Proofreader
Donna E. Mulder

Technical Editor
Dino Esposito

Editorial Assistant
Cindy Teeters

Cover Designer
Twist Creative, Seattle

Compositor
codeMantra

Graphics
codeMantra

Figure Credits
Figure 4.1: LangChain, Inc
Figures 7.1, 7.2, 7.4: Snowflake, Inc
Figure 8.2: SmartBear Software
Figure 8.3: Postman, Inc
Dedication

A I.
Perché non dedicarti un libro sarebbe stato un sacrilegio.
Contents at a Glance

Introduction

CHAPTER 1 The genesis and an analysis of large language models

CHAPTER 2 Core prompt learning techniques
CHAPTER 3 Engineering advanced learning prompts
CHAPTER 4 Mastering language frameworks
CHAPTER 5 Security, privacy, and accuracy concerns
CHAPTER 6 Building a personal assistant
CHAPTER 7 Chat with your data
CHAPTER 8 Conversational UI

Appendix: Inner functioning of LLMs

Index
Contents

Acknowledgments
Introduction

Chapter 1 The genesis and an analysis of large language models

LLMs at a glance
History of LLMs
Functioning basics
Business use cases
Facts of conversational programming
The emerging power of natural language
LLM topology
Future perspective
Summary

Chapter 2 Core prompt learning techniques

What is prompt engineering?
Prompts at a glance
Alternative ways to alter output
Setting up for code execution
Basic techniques
Zero-shot scenarios
Few-shot scenarios
Chain-of-thought scenarios
Fundamental use cases
Chatbots
 Translating
LLM limitations
Summary

Chapter 3 Engineering advanced learning prompts

What’s beyond prompt engineering?
Combining pieces
Fine-tuning
Function calling
Homemade-style
OpenAI-style
Talking to (separated) data
Connecting data to LLMs
Embeddings
Vector store
Retrieval augmented generation
Summary

Chapter 4 Mastering language frameworks

The need for an orchestrator
Cross-framework concepts
Points to consider
LangChain
Models, prompt templates, and chains
Agents
Data connection
Microsoft Semantic Kernel
Plug-ins
Data and planners
Microsoft Guidance
Configuration
 Main features
Summary

Chapter 5 Security, privacy, and accuracy concerns

Overview
Responsible AI
Red teaming
Abuse and content filtering
Hallucination and performances
Bias and fairness
Security and privacy
Security
Privacy
Evaluation and content filtering
Evaluation
Content filtering
Summary

Chapter 6 Building a personal assistant

Overview of the chatbot web application
Scope
Tech stack
The project
Setting up the LLM
Setting up the project
Integrating the LLM
Possible extensions
Summary

Chapter 7 Chat with your data

Overview
 Scope
Tech stack
What is Streamlit?
A brief introduction to Streamlit
Main UI features
Pros and cons in production
The project
Setting up the project and base UI
Data preparation
LLM integration
Progressing further
Retrieval augmented generation versus fine-tuning
Possible extensions
Summary

Chapter 8 Conversational UI
Overview
Scope
Tech stack
The project
Minimal API setup
OpenAPI
LLM integration
Possible extensions
Summary

Appendix: Inner functioning of LLMs

Index
Acknowledgments

In the spring of 2023, when I told my dad how cool Azure OpenAI was
becoming, his reply was kind of a shock: “Why don’t you write a book about
it?” He said it so naturally that it hit me as if he really thought I could do it.
In fact, he added, “Are you up for it?” Then there was no need to say more.
Loretta Yates at Microsoft Press enthusiastically accepted my proposal, and
the story of this book began in June 2023.
AI has been a hot topic for the better part of a decade, but the emergence
of new-generation large language models (LLMs) has propelled it into the
mainstream. The increasing number of people using them translates to more
ideas, more opportunities, and new developments. And this makes all the
difference.
Hence, the book you hold in your hands can’t be the ultimate and
definitive guide to AI and LLMs because the speed at which AI and LLMs
evolve is impressive and because—by design—every book is an act of
approximation, a snapshot of knowledge taken at a specific moment in time.
Approximation inevitably leads to some form of dissatisfaction, and
dissatisfaction leads us to take on new challenges. In this regard, I wish for
myself decades of dissatisfaction. And a few more years of being on the stage
presenting books written for a prestigious publisher—it does wonders for my
ego.
First, I feel somewhat indebted to all my first dates since May because
they had to endure monologues lasting at least 30 minutes on LLMs and
some weird new approach to transformers.
True thanks are a private matter, but publicly I want to thank Martina first,
who cowrote the appendix with me and always knows what to say to make
me better. My gratitude to her is keeping a promise she knows. Thank you,
Martina, for being an extraordinary human being.
 To Gianfranco, who taught me the importance of discussing and
expressing, even loudly, when something doesn’t please us, and taught me to
always ask, because the worst thing that can happen is hearing a no. Every
time I engage in a discussion, I will think of you.
I also want to thank Matteo, Luciano, Gabriele, Filippo, Daniele,
Riccardo, Marco, Jacopo, Simone, Francesco, and Alessia, who worked with
me and supported me during my (hopefully not too frequent) crises. I also
have warm thoughts for Alessandro, Antonino, Sara, Andrea, and Cristian
who tolerated me whenever we weren’t like 25-year-old youngsters because I
had to study and work on this book.
To Mom and Michela, who put up with me before the book and probably
will continue after. To my grandmas. To Giorgio, Gaetano, Vito, and Roberto
for helping me to grow every day. To Elio, who taught me how to dress and
see myself in more colors.
As for my dad, Dino, he never stops teaching me new things—for
example, how to get paid for doing things you would just love to do, like
being the technical editor of this book. Thank you, both as a father and as an
editor. You bring to my mind a song you well know: “Figlio, figlio, figlio.”
Beyond Loretta, if this book came to life, it was also because of the hard
work of Shourav, Kate, and Dan. Thank you for your patience and for
trusting me so much.
This book is my best until the next one!
Introduction

This is my third book on artificial intelligence (AI), and the first I wrote on
my own, without the collaboration of a coauthor. The sequence in which my
three books have been published reflects my own learning path, motivated by
a genuine thirst to understand AI for far more than mere business
considerations. The first book, published in 2020, introduced the
mathematical concepts behind machine learning (ML) that make it possible to
classify data and make timely predictions. The second book, which focused
on the Microsoft ML.NET framework, was about concrete applications—in
other words, how to make fancy algorithms work effectively on amounts of
data hiding their complexity behind the charts and tables of a familiar web
front end.
Then came ChatGPT.
The technology behind astonishing applications like ChatGPT is called a
large language model (LLM), and LLMs are the subject of this third book.
LLMs add a crucial capability to AI: the ability to generate content in
addition to classifying and predicting. LLMs represent a paradigm shift,
raising the bar of communication between humans and computers and
opening the floodgates to new applications that for decades we could only
dream of.
And for decades, we did dream of these applications. Literature and
movies presented various supercomputers capable of crunching any sort of
data to produce human-intelligible results. An extremely popular example
was HAL 9000—the computer that governed the spaceship Discovery in the
movie 2001: A Space Odyssey (1968). Another famous one was JARVIS
(Just A Rather Very Intelligent System), the computer that served Tony
Stark’s home assistant in Iron Man and other movies in the Marvel Comics
universe.
 Often, all that the human characters in such books and movies do is
simply “load data into the machine,” whether in the form of paper
documents, digital files, or media content. Next, the machine autonomously
figures out the content, learns from it, and communicates back to humans
using natural language. But of course, those supercomputers were conceived
by authors; they were only science fiction. Today, with LLMs, it is possible
to devise and build concrete applications that not only make human–
computer interaction smooth and natural, but also turn the old dream of
simply “loading data into the machine” into a dazzling reality.
This book shows you how to build software applications using the same
type of engine that fuels ChatGPT to autonomously communicate with users
and orchestrate business tasks driven by plain textual prompts. No more, no
less—and as easy and striking as it sounds!

Who should read this book

Software architects, lead developers, and individuals with a background in
programming—particularly those familiar with languages like Python and
possibly C# (for ASP.NET Core)—will find the content in this book
accessible and valuable. In the vast realm of software professionals who
might find the book useful, I’d call out those who have an interest in ML,
especially in the context of LLMs. I’d also list cloud and IT professionals
with an interest in using cloud services (specifically Microsoft Azure) or in
sophisticated, real-world applications of human-like language in software.
While this book focuses primarily on the services available on the Microsoft
Azure platform, the concepts covered are easily applicable to analogous
platforms. At the end of the day, using an LLM involves little more than
calling a bunch of API endpoints, and, by design, APIs are completely
independent of the underlying platform.
In summary, this book caters to a diverse audience, including
programmers, ML enthusiasts, cloud-computing professionals, and those
interested in natural language processing, with a specific emphasis on
leveraging Azure services to program LLMs.
Assumptions

To fully grasp the value of a programming book on LLMs, there are a couple
of prerequisites, including proficiency in foundational programming concepts
and a familiarity with ML fundamentals. Beyond these, a working knowledge
of relevant programming languages and frameworks, such as Python and
possibly ASP.NET Core, is helpful, as is an appreciation for the significance
of classic natural language processing in the context of business domains.
Overall, a blend of programming expertise, ML awareness, and linguistic
understanding is recommended for a comprehensive grasp of the book’s
content.

This book might not be for you if…

This book might not be for you if you’re just seeking a reference book to find
out in detail how to use a particular pattern or framework. Although the book
discusses advanced aspects of popular frameworks (for example, LangChain
and Semantic Kernel) and APIs (such as OpenAI and Azure OpenAI), it does
not qualify as a programming reference on any of these. The focus of the
book is on using LLMs to build useful applications in the business domains
where LLMs really fit well.

Organization of this book

This book explores the practical application of existing LLMs in developing

versatile business domain applications. In essence, an LLM is an ML model
trained on extensive text data, enabling it to comprehend and generate
human-like language. To convey knowledge about these models, this book
focuses on three key aspects:
The first three chapters delve into scenarios for which an LLM is
effective and introduce essential tools for crafting sophisticated
solutions. These chapters provide insights into conversational
programming and prompting as a new, advanced, yet structured,
approach to coding.
 The next two chapters emphasize patterns, frameworks, and techniques
for unlocking the potential of conversational programming. This
involves using natural language in code to define workflows, with the
LLM-based application orchestrating existing APIs.
The final three chapters present concrete, end-to-end demo examples
featuring Python and ASP.NET Core. These demos showcase
progressively advanced interactions between logic, data, and existing
business processes. In the first demo, you learn how to take text from an
email and craft a fitting draft for a reply. In the second demo, you apply
a retrieval augmented generation (RAG) pattern to formulate responses
to questions based on document content. Finally, in the third demo, you
learn how to build a hotel booking application with a chatbot that uses a
conversational interface to ascertain the user’s needs (dates, room
preferences, budget) and seamlessly places (or denies) reservations
according to the underlying system’s state, without using fixed user
interface elements or formatted data input controls.

Downloads: notebooks and samples

Python and Polyglot notebooks containing the code featured in the initial part
of the book, as well as the complete codebases for the examples tackled in the
latter part of the book, can be accessed on GitHub at:
https://github.com/Youbiquitous/programming-llm

Errata, updates, & book support

We’ve made every effort to ensure the accuracy of this book and its
companion content. You can access updates to this book—in the form of a
list of submitted errata and their related corrections—at:
MicrosoftPressStore.com/LLMAzureAI/errata
If you discover an error that is not already listed, please submit it to us at
the same page.
 For additional book support and information, please visit
MicrosoftPressStore.com/Support.
Please note that product support for Microsoft software and hardware is
not offered through the previous addresses. For help with Microsoft software
or hardware, go to http://support.microsoft.com.

Stay in touch
Let’s keep the conversation going! We’re on X / Twitter:
http://twitter.com/MicrosoftPress.
Chapter 1

The genesis and an analysis of large

language models

Luring someone into reading a book is never a small feat. If it’s a novel, you
must convince them that it’s a beautiful story, and if it’s a technical book,
you must assure them that they’ll learn something. In this case, we’ll try to
learn something.
Over the past two years, generative AI has become a prominent buzzword.
It refers to a field of artificial intelligence (AI) focused on creating systems
that can generate new, original content autonomously. Large language
models (LLMs) like GPT-3 and GPT-4 are notable examples of generative
AI, capable of producing human-like text based on given input.
The rapid adoption of LLMs is leading to a paradigm shift in
programming. This chapter discusses this shift, the reasons for it, and its
prospects. Its prospects include conversational programming, in which you
explain with words—rather than with code—what you want to achieve. This
type of programming will likely become very prevalent in the future.
No promises, though. As you’ll soon see, explaining with words what you
want to achieve is often as difficult as writing code.
This chapter covers topics that didn’t find a place elsewhere in this book.
It’s not necessary to read every section or follow a strict order. Take and read
what you find necessary or interesting. I expect you will come back to read
certain parts of this chapter after you finish the last one.
LLMs at a glance

To navigate the realm of LLMs as a developer or manager, it’s essential to

comprehend the origins of generative AI and to discern its distinctions from
predictive AI. This chapter has one key goal: to provide insights into the
training and business relevance of LLMs, reserving the intricate mathematical
details for the appendix.
Our journey will span from the historical roots of AI to the fundamentals
of LLMs, including their training, inference, and the emergence of
multimodal models. Delving into the business landscape, we’ll also spotlight
current popular use cases of generative AI and textual models.
This introduction doesn’t aim to cover every detail. Rather, it intends to
equip you with sufficient information to address and cover any potential gaps
in knowledge, while working toward demystifying the intricacies surrounding
the evolution and implementation of LLMs.

History of LLMs
The evolution of LLMs intersects with both the history of conventional AI
(often referred to as predictive AI) and the domain of natural language
processing (NLP). NLP encompasses natural language understanding (NLU),
which attempts to reduce human speech into a structured ontology, and
natural language generation (NLG), which aims to produce text that is
understandable by humans.
LLMs are a subtype of generative AI focused on producing text based on
some kind of input, usually in the form of written text (referred to as a
prompt) but now expanding to multimodal inputs, including images, video,
and audio. At a glance, most LLMs can be seen as a very advanced form of
autocomplete, as they generate the next word. Although they specifically
generate text, LLMs do so in a manner that simulates human reasoning,
enabling them to perform a variety of intricate tasks. These tasks include
sentiment analysis, summarization, translation, entity and intent recognition,
structured information extraction, document generation, and so on.
 LLMs represent a natural extension of the age-old human aspiration to
construct automatons (ancestors to contemporary robots) and imbue them
with a degree of reasoning and language. They can be seen as a brain for such
automatons, able to respond to an external input.

AI beginnings
Modern software—and AI as a vibrant part of it—represents the culmination
of an embryonic vision that has traversed the minds of great thinkers since
the 17th century. Various mathematicians, philosophers, and scientists, in
diverse ways and at varying levels of abstraction, envisioned a universal
language capable of mechanizing the acquisition and sharing of knowledge.
Gottfried Leibniz (1646–1716), in particular, contemplated the idea that at
least a portion of human reasoning could be mechanized.
The modern conceptualization of intelligent machinery took shape in the
mid-20th century, courtesy of renowned mathematicians Alan Turing and
Alonzo Church. Turing’s exploration of “intelligent machinery” in 1947,
coupled with his groundbreaking 1950 paper, “Computing Machinery and
Intelligence,” laid the cornerstone for the Turing test—a pivotal concept in
AI. This test challenged machines to exhibit human behavior
(indistinguishable by a human judge), ushering in the era of AI as a scientific
discipline.

Note
Considering recent advancements, a reevaluation of the original
Turing test may be warranted to incorporate a more precise
definition of human and rational behavior.

NLP
NLP is an interdisciplinary field within AI that aims to bridge the interaction
between computers and human language. While historically rooted in
linguistic approaches, distinguishing itself from the contemporary sense of
AI, NLP has perennially been a branch of AI in a broader sense. In fact, the
overarching goal has consistently been to artificially replicate an expression
of human intelligence—specifically, language.
The primary goal of NLP is to enable machines to understand, interpret,
and generate human-like language in a way that is both meaningful and
contextually relevant. This interdisciplinary field draws from linguistics,
computer science, and cognitive psychology to develop algorithms and
models that facilitate seamless interaction between humans and machines
through natural language.
The history of NLP spans several decades, evolving from rule-based
systems in the early stages to contemporary deep-learning approaches,
marking significant strides in the understanding and processing of human
language by computers.
Originating in the 1950s, early efforts, such as the Georgetown-IBM
experiment in 1954, aimed at machine translation from Russian to English,
laying the foundation for NLP. However, these initial endeavors were
primarily linguistic in nature. Subsequent decades witnessed the influence of
Chomskyan linguistics, shaping the field’s focus on syntactic and
grammatical structures.
The 1980s brought a shift toward statistical methods, like n-grams, using
co-occurrence frequencies of words to make predictions. An example was
IBM’s Candide system for speech recognition. However, rule-based
approaches struggled with the complexity of natural language. The 1990s saw
a resurgence of statistical approaches and the advent of machine learning
(ML) techniques such as hidden Markov models (HMMs) and statistical
language models. The introduction of the Penn Treebank, a 7-million word
dataset of part-of-speech tagged text, and statistical machine translation
systems marked significant milestones during this period.
In the 2000s, the rise of data-driven approaches and the availability of
extensive textual data on the internet rejuvenated the field. Probabilistic
models, including maximum-entropy models and conditional random fields,
gained prominence. Begun in the 1980s but finalized years later, the
development of WordNet, a semantical-lexical database of English (with its
groups of synonyms, or synonym set, and their relations), contributed to a
deeper understanding of word semantics.
The landscape transformed in the 2010s with the emergence of deep
learning made possible by a new generation of graphics processing units
(GPUs) and increased computing power. Neural network architectures—
particularly transformers like Bidirectional Encoder Representations from
Transformers (BERT) and Generative Pretrained Transformer (GPT)—
revolutionized NLP by capturing intricate language patterns and contextual
information. The focus shifted to data-driven and pretrained language
models, allowing for fine-tuning of specific tasks.

Predictive AI versus generative AI

Predictive AI and generative AI represent two distinct paradigms, each
deeply entwined with advancements in neural networks and deep-learning
architectures.
Predictive AI, often associated with supervised learning, traces its roots
back to classical ML approaches that emerged in the mid-20th century. Early
models, such as perceptrons, paved the way for the resurgence of neural
networks in the 1980s. However, it wasn’t until the advent of deep learning in
the 21st century—with the development of deep neural networks,
convolutional neural networks (CNNs) for image recognition, and recurrent
neural networks (RNNs) for sequential data—that predictive AI witnessed a
transformative resurgence. The introduction of long short-term memory
(LSTM) units enabled more effective modeling of sequential dependencies in
data.
Generative AI, on the other hand, has seen remarkable progress, propelled
by advancements in unsupervised learning and sophisticated neural network
architectures (the same used for predictive AI). The concept of generative
models dates to the 1990s, but the breakthrough came with the introduction
of generative adversarial networks (GANs) in 2014, showcasing the power of
adversarial training. GANs, which feature a generator for creating data and a
discriminator to distinguish between real and generated data, play a pivotal
role. The discriminator, discerning the authenticity of the generated data
during the training, contributes to the refinement of the generator, fostering
continuous enhancement in generating more realistic data, spanning from
lifelike images to coherent text.
Table 1-1 provides a recap of the main types of learning processes.

TABLE 1-1 Main types of learning processes

Type Definition Training Use Cases

Supervised Trained on Adjusts Classification,
labeled data parameters to regression
where each input minimize the
has a prediction error
corresponding
label
Self- Unsupervised Learns to fill in NLP, computer
supervised learning where the blank (predict vision
the model parts of input data
generates its own from other parts)
labels
Semi- Combines Uses labeled data Scenarios with
supervised labeled and for supervised limited labeled
unlabeled data tasks, unlabeled data—for
for training data for example, image
generalizations classification
Unsupervised Trained on data Identifies inherent Clustering,
without explicit structures or dimensionality
supervision relationships in reduction,
the data generative
modeling

The historical trajectory of predictive and generative AI underscores the

symbiotic relationship with neural networks and deep learning. Predictive AI
leverages deep-learning architectures like CNNs for image processing and
RNNs/LSTMs for sequential data, achieving state-of-the-art results in tasks
ranging from image recognition to natural language understanding.
Generative AI, fueled by the capabilities of GANs and large-scale language
models, showcases the creative potential of neural networks in generating
novel content.

LLMs
An LLM, exemplified by OpenAI’s GPT series, is a generative AI system
built on advanced deep-learning architectures like the transformer (more on
this in the appendix).
These models operate on the principle of unsupervised and self-supervised
learning, training on vast text corpora to comprehend and generate coherent
and contextually relevant text. They output sequences of text (that can be in
the form of proper text but also can be protein structures, code, SVG, JSON,
XML, and so on), demonstrating a remarkable ability to continue and expand
on given prompts in a manner that emulates human language.
The architecture of these models, particularly the transformer architecture,
enables them to capture long-range dependencies and intricate patterns in
data. The concept of word embeddings, a crucial precursor, represents words
as continuous vectors (Mikolov et al. in 2013 through Word2Vec),
contributing to the model’s understanding of semantic relationships between
words. Word embeddings is the first “layer” of an LLM.
The generative nature of the latest models enables them to be versatile in
output, allowing for tasks such as text completion, summarization, and
creative text generation. Users can prompt the model with various queries or
partial sentences, and the model autonomously generates coherent and
contextually relevant completions, demonstrating its ability to understand and
mimic human-like language patterns.
The journey began with the introduction of word embeddings in 2013,
notably with Mikolov et al.’s Word2Vec model, revolutionizing semantic
representation. RNNs and LSTM architectures followed, addressing
challenges in sequence processing and long-range dependencies. The
transformative shift arrived with the introduction of the transformer
architecture in 2017, allowing for parallel processing and significantly
improving training times.
 In 2018, Google researchers Devlin et al. introduced BERT. BERT
adopted a bidirectional context prediction approach. During pretraining,
BERT is exposed to a masked language modeling task in which a random
subset of words in a sentence is masked and the model predicts those masked
words based on both left and right context. This bidirectional training allows
BERT to capture more nuanced contextual relationships between words. This
makes it particularly effective in tasks requiring a deep understanding of
context, such as question answering and sentiment analysis.
During the same period, OpenAI’s GPT series marked a paradigm shift in
NLP, starting with GPT in 2018 and progressing through GPT-2 in 2019, to
GPT-3 in 2020, and GPT-3.5-turbo, GPT-4, and GPT-4-turbo-visio (with
multimodal inputs) in 2023. As autoregressive models, these predict the next
token (which is an atomic element of natural language as it is elaborated by
machines) or word in a sequence based on the preceding context. GPT’s
autoregressive approach, predicting one token at a time, allows it to generate
coherent and contextually relevant text, showcasing versatility and language
understanding. The size of this model is huge, however. For example, GPT-3
has a massive scale of 175 billion parameters. (Detailed information about
GPT-3.5-turbo and GPT-4 are not available at the time of this writing.) The
fact is, these models can scale and generalize, thus reducing the need for task-
specific fine-tuning.

Functioning basics
The core principle guiding the functionality of most LLMs is autoregressive
language modeling, wherein the model takes input text and systematically
predicts the subsequent token or word (more on the difference between these
two terms shortly) in the sequence. This token-by-token prediction process is
crucial for generating coherent and contextually relevant text. However, as
emphasized by Yann LeCun, this approach can accumulate errors; if the N-th
token is incorrect, the model may persist in assuming its correctness,
potentially leading to inaccuracies in the generated text.
Until 2020, fine-tuning was the predominant method for tailoring models
to specific tasks. Recent advancements, however—particularly exemplified
by larger models like GPT-3—have introduced prompt engineering. This
allows these models to achieve task-specific outcomes without conventional
fine-tuning, relying instead on precise instructions provided as prompts.
Models such as those found in the GPT series are intricately crafted to
assimilate comprehensive knowledge about the syntax, semantics, and
underlying ontology inherent in human language corpora. While proficient at
capturing valuable linguistic information, it is imperative to acknowledge that
these models may also inherit inaccuracies and biases present in their training
corpora.

Different training approaches

An LLM can be trained with different goals, each requiring a different
approach. The three prominent methods are as follows:
Causal language modeling (CLM) This autoregressive method is used
in models like OpenAI’s GPT series. CLM trains the model to predict
the next token in a sequence based on preceding tokens. Although
effective for tasks like text generation and summarization, CLM models
possess a unidirectional context, only considering past context during
predictions. We will focus on this kind of model, as it is the most used
architecture at the moment.
Masked language modeling (MLM) This method is employed in
models like BERT, where a percentage of tokens in the input sequence
are randomly masked and the model predicts the original tokens based
on the surrounding context. This bidirectional approach is advantageous
for tasks such as text classification, sentiment analysis, and named entity
recognition. It is not suitable for pure text-generation tasks because in
those cases the model should rely only on the past, or “left part,” of the
input, without looking at the “right part,” or the future.
Sequence-to-sequence (Seq2Seq) These models, which feature an
encoder-decoder architecture, are used in tasks like machine translation
and summarization. The encoder processes the input sequence,
generating a latent representation used by the decoder to produce the
output sequence. This approach excels in handling complex tasks
involving input-output transformations, which are commonly used for
tasks where the input and output have a clear alignment during training,
 such as translation tasks.
The key disparities lie in their objectives, architectures, and suitability for
specific tasks. CLM focuses on predicting the next token and excels in text
generation, MLM specializes in (bidirectional) context understanding, and
Seq2Seq is adept at generating coherent output text in the form of sequences.
And while CLM models are suitable for autoregressive tasks, MLM models
understand and embed the context, and Seq2Seq models handle input-output
transformations. Models may also be pretrained on auxiliary tasks, like next
sentence prediction (NSP), which tests their understanding of data
distribution.

The transformer model

The transformer architecture forms the foundation for modern LLMs.
Vaswani et al. presented the transformer model in a paper, “Attention Is All
You Need,” released in December 2017. Since then, NLP has been
completely revolutionized. Unlike previous models, which rely on sequential
processing, transformers employ an attention mechanism that allows for
parallelization and captures long-range dependencies.
The original model consists of an encoder and decoder, both articulated in
multiple self-attention processing layers. Self-attention processing means that
each word is determined by examining and considering its contextual
information.
In the encoder, input sequences are embedded and processed in parallel
through the layers, thus capturing intricate relationships between words. The
decoder generates output sequences, using the encoder’s contextual
information. Throughout the training process, the decoder learns to predict
the next word by analyzing the preceding words.
The transformer incorporates multiple layers of decoders to enhance its
capacity for language generation. The transformer’s design includes a context
window, which determines the length of the sequence the model considers
during inference and training. Larger context windows offer a broader scope
but incur higher computational costs, while smaller windows risk missing
crucial long-range dependencies. The real “brain” that allows transformers to
understand context and excel in tasks like translation and summarization is
the self-attention mechanism. There’s nothing like conscience or neuronal
learning in today’s LLM.
The self-attention mechanism allows the LLM to selectively focus on
different parts of the input sequence instead of treating the entire input in the
same way. Because of this, it needs fewer parameters to model long-term
dependencies and can capture relationships between words placed far away
from each other in the sequence. It’s simply a matter of guessing the next
words on a statistical basis, although it really seems smart and human.
While the original transformer architecture was a Seq2Seq model,
converting entire sequences from a source to a target format, nowadays the
current approach for text generation is an autoregressive approach.
Deviating from the original architecture, some models, including GPTs,
don’t include an explicit encoder part, relying only on the decoder. In this
architecture, the input is fed directly to the decoder. The decoder has more
self-attention heads and has been trained with a massive amount of data in an
unsupervised manner, just predicting the next word of existing texts.
Different models, like BERT, include only the encoder part that produces the
so-called embeddings.

Tokens and tokenization

Tokens, the elemental components in advanced language models like GPTs,
are central to the intricate process of language understanding and generation.
Unlike traditional linguistic units like words or characters, a token
encapsulates the essence of a single word, character, or subword unit. This
finer granularity is paramount for capturing the subtleties and intricacies
inherent in language.
The process of tokenization is a key facet. It involves breaking down texts
into smaller, manageable units, or tokens, which are then subjected to the
model’s analysis. The choice of tokens over words is deliberate, allowing for
a more nuanced representation of language.
OpenAI and Azure OpenAI employ a subword tokenization technique
called byte-pair encoding (BPE). In BPE, frequently occurring pairs of
characters are amalgamated into single tokens, contributing to a more
compact and consistent representation of textual data. With BPE, a single
token results in approximately four characters in English or three-quarters of
a word; equivalently, 100 tokens equal roughly 75 words. To provide an
example, the sentence “Many words map to one token, but some don’t:
indivisible”, would be split into [“Many”, “ words”, “ map”, “ to”, “ one”, “
token”, “,”, “ but”, “ some”, “ don”, “'t”, “:”, “ indiv”, “isible”], which,
mapped to token IDs, would be [8607, 4339, 2472, 311, 832, 4037, 11, 719,
1063, 1541, 956, 25, 3687, 23936].
Tokenization serves a multitude of purposes, influencing both the
computational dynamics and the qualitative aspects of the generated text. The
computational cost of running an LLM is intricately tied to tokenization
methods, vocabulary size (usually 30,000–50,000 different tokens are used
for a single language vocabulary), and the length and complexity of input and
output texts.
The deliberate choice of tokens over words in LLMs is driven by various
considerations:
Tokens facilitate a more granular representation of language, allowing
models to discern subtle meanings and handle out-of-vocabulary or rare
words effectively. This level of granularity is particularly crucial when
dealing with languages that exhibit rich morphological structures.
Tokens help address the challenge of handling ambiguity and polysemy
in language, with a more compositional approach.
Subword tokenization enables LLMs to represent words as combinations
of subword tokens, allowing them to capture different senses of a word
more effectively based on preceding or following characters. For
instance, the suffix of a word can have two different token
representations, depending on the prefix of the next word.
Although the tokenization algorithm is run over a given language
(usually English), which makes token splitting sub-optimal for different
languages, it natively extends support for any language with the same
character set.
 The use of tokens significantly aids in efficient memory use. By
breaking down text into smaller units, LLMs can manage memory more
effectively, processing and storing a larger vocabulary without imposing
impractical demands on memory resources.
In summary, tokens and tokenization represent the foundational elements
that shape the processing and understanding of language in LLMs. From their
role in providing granularity and managing memory to addressing linguistic
challenges, tokens are indispensable in optimizing the performance and
efficiency of LLMs.

Embeddings
Tokenization and embeddings are closely related concepts in NLP.
Tokenization involves breaking down a sequence of text into smaller
units. These tokens are converted into IDs and serve as the basic building
blocks for the model to process textual information. Embeddings, on the
other hand, refer to the numerical and dense representations of these tokens in
a high-dimensional vector space, usually 1000+ dimensions.
Embeddings are generated through an embedding layer in the model, and
they encode semantic relationships and contextual information about the
tokens. The embedding layer essentially learns, during training, a distributed
representation for each token, enabling the model to understand the
relationships and similarities between words or subwords based on their
contextual usage.
Semantic search is made simple through embeddings: We can embed
different sentences and measure their distances in this 1000+ dimensional
space. The shorter the sentence is and the larger this high-dimensional space
is, the more accurate the semantic representation is. The inner goal of
embedding is to have words like queen and king close in the embedding
space, with woman being quite close to queen as well.
Embeddings can work on a word level, like Word2Vec (2013), or on a
sentence level, like OpenAI’s text-ada-002 (with its latest version released in
2022).
 If an embedding model (a model that takes some text as input and outputs
a dense numerical vector) is usually the output of the encoding part of a
transformer model, for GPTs models it’s a different story. In fact, GPT-4 has
some inner embedding layers (word and positional) inside the attention
heads, while the proper embedding model (text-ada-002) is trained separately
and not directly used within GPT-4. Text-ada-002 is available just like the
text-generation model and is used for similarity search and similar use cases
(discussed later).
In summary, tokenization serves as the initial step in preparing textual
data for ML models, and embeddings enhance this process by creating
meaningful numerical representations that capture the semantic nuances and
contextual information of the tokens.

Training steps
The training of GPT-like language models involves several key phases, each
contributing to the model’s development and proficiency:
1. Initial training on crawl data
2. Supervised fine-tuning (SFT)
3. Reward modeling
4. Reinforcement learning from human feedback (RLHF)

Initial training on crawl data

In the initial phase, the language model is pretrained on a vast dataset
collected from internet crawl data and/or private datasets. This initial training
set for future models likely includes LLM-generated text.
During this phase, the model learns the patterns, structure, and
representations of language by predicting the next word in a sequence given
the context. This is achieved using a language modeling objective.
Tokenization is a crucial preprocessing step during which words or
subwords are converted into tokens and then into numerical tokens. Using
tokens instead of words enables the model to capture more nuanced
relationships and dependencies within the language because tokens can
represent subword units, characters, or even parts of words.
The model is trained to predict the next token in a sequence based on the
preceding tokens. This training objective is typically implemented using a
loss function, such as cross-entropy loss, which measures the dissimilarity
between the predicted probability distribution over tokens and the actual
distribution.
Models coming out of this phase are usually referred to as base models or
pretrained models.

Supervised fine-tuning (SFT)

Following initial training, the model undergoes supervised fine-tuning (SFT).
In this phase, prompts and completions are fed into the model to further
refine its base. The model learns from labeled data, adjusting its parameters
to improve performance on specific tasks.
Some small open-source models use outputs from bigger models for this
fine-tuning phase. Even if this is a clever way to save money when training, it
can lead to misleading models that claim to have higher capabilities than they
do.

Reward modeling
Once the model is fine-tuned with SFT, a reward model is created. Human
evaluators review and rate different model outputs based on quality,
relevance, accuracy, and other criteria. These ratings are used to create a
reward model that predicts the “reward” or rating for various outputs.

Reinforcement learning from human feedback (RLHF)

With the reward model in place, RLHF is employed to guide the model in
generating better outputs. The model receives feedback on its output from the
reward model and adjusts its parameters to maximize the predicted reward.
This reinforcement learning process enhances the model’s precision and
communication skills. Closed source models, like GPT-4, are RLHF models
(with the base models behind it not yet released).
 It is crucial to acknowledge the distinct nature of prompting a base model
compared to an RLHF or SFT model. When presented with a prompt such as
“write me a song about love,” a base model is likely to produce something
akin to “write me a poem about loyalty” rather than a song about love. This
tendency arises from the training dataset, where the phrase “write me a song
about love” might precede other similar instructions, leading the model to
generate responses aligned with those patterns. To guide a base model toward
generating a love song, a nuanced approach to prompt engineering becomes
essential. For instance, crafting a prompt like “here is a love song: I love you
since the day we met” allows the model to build on the provided context and
generate the desired output.

Inference
The inferring process is an autoregressive generation process that involves
iteratively calling the model with its own generated outputs, employing initial
inputs. During causal language modeling, a sequence of text tokens is taken
as input, and the model returns the probability distribution for the next token.
The non-deterministic aspect arises when selecting the next token from
this distribution, often achieved through sampling. However, some models
provide a seed option for deterministic outcomes.
The selection process can range from simple (choosing the most likely
token) to complex (involving various transformations). Parameters like
temperature influence the model’s creativity, with high temperatures yielding
a flatter probability distribution.
The iterative process continues until a stopping condition—ideally
determined by the model or a predefined maximum length—is reached.
When the model generates incorrect, nonsensical, or even false
information, it is called hallucination. When LLMs generate text, they
operate as prompt-based extrapolators, lacking the citation of specific training
data sources, as they are not designed as databases or search engines. The
process of abstraction—transforming both the prompt and training data—can
contribute to hallucination due to limited contextual understanding, leading to
potential information loss.
 Despite being trained on trillions of tokens, as seen in the case of GPT-3
with nearly 1 TB of data, the weights of these models—determining their size
—are often 20% to 40% less than the original. Here, quantization is
employed to try to reduce weight size, truncating precision for weights.
However, LLMs are not engineered as proper lossless compressors, resulting
in information loss at some point; this is a possible heuristic explanation for
hallucination.
One more reason is an intrinsic limitation of LLMs as autoregressive
predictors. In fact, during the prediction of the next token, LLMs rely heavily
on the tokens within their context window belonging to the dataset
distribution, which is primarily composed of text written by humans. As we
execute LLMs and sample tokens from them, each sampled token
incrementally shifts the model slightly outside the distribution it was initially
trained on. The model’s actual input is generated partially by itself, and as we
extend the length of the sequence we aim to predict, we progressively move
the model beyond the familiar distribution it has learned.

Note
Hallucinations can be considered a feature in LLMs, especially
when seeking creativity and diversity. For instance, when
requesting a fantasy story plot from ChatGPT or other LLMs, the
objective is not replication but the generation of entirely new
characters, scenes, and storylines. This creative aspect relies on the
models not directly referencing the data on which they were trained,
allowing for imaginative and diverse outputs.

Fine-tuning, prompting, and other techniques

To optimize responses from an LLM, various techniques such as prompt
engineering and fine-tuning are employed.
Prompt engineering involves crafting carefully phrased and specific user
queries to guide and shape the model’s responses. This specialized skill aims
to improve output by creating more meaningful inputs and often requires a
deep understanding of the model’s architecture. Prompt engineering works
because it leverages the capabilities of newer and larger language models that
have learned general internal representations of language. These advanced
models, often developed through techniques like unsupervised pretraining on
vast datasets, possess a deep understanding of linguistic structures, context,
and semantics. As a result, they can generate meaningful responses based on
the input they receive.
When prompt engineers craft carefully phrased and specific queries, they
tap into the model’s ability to interpret and generate language in a
contextually relevant manner. By providing the model with more detailed and
effective inputs, prompt engineering guides the model to produce desired
outputs. Essentially, prompt engineering aligns with the model’s inherent
capacity to comprehend and generate language, allowing users to influence
and optimize its responses through well-crafted prompts.
In contrast, fine-tuning is a training technique that adapts the LLM to
specific tasks or knowledge domains by applying new, often custom,
datasets. This process involves training the model’s weights with additional
data, resulting in improved performance and relevance.
Prompt engineering and fine-tuning serve different optimization purposes.
Prompt engineering focuses on eliciting better output by refining inputs,
while fine-tuning aims to enhance the model’s performance on specific tasks
by training on new datasets. And prompt engineering offers precise control
over the LLM’s actions, while fine-tuning adds depth to relevant topic areas.
Both techniques can be complementary, improving overall model behavior
and output.
There are specific tasks that, in essence, cannot be addressed by any LLM,
at least not without leveraging external tools or supplementary software. An
illustration of such a task is generating a response to the user’s input
'calculate 12*6372’, particularly if the LLM has not previously encountered a
continuation of this calculation in its training dataset. For this, an older-style
option is the usage of plug-ins as extensions to allow the LLM to access
external tools or data, broadening its capabilities. For instance, ChatGPT
supports plug-ins for services like Wolfram Alpha, Bing Search, and so on.
 Pushing prompt engineering forward, one can also encourage self-
reflection in LLMs involving techniques like chain-of-thought prompts,
guiding models to explain their thinking. Constrained prompting (such as
templated prompts, interleave generation, and logical control) is another
technique recommended to improve accuracy and safety in model outputs.
In summary, optimizing responses from LLMs is a multifaceted process
that involves a combination of prompt engineering, fine-tuning, and plug-in
integration, all tailored to the specific requirements of the desired tasks and
domains.

Multimodal models
Most ML models are trained and operate in a unimodal way, using a single
type of data—text, image, or audio. Multimodal models amalgamate
information from diverse modalities, encompassing elements like images and
text. Like humans, they can seamlessly navigate different data modes. They
are usually subject to a slightly different training process.
There are different types of multimodalities:
Multimodal input This includes the following:
Text and image input Multimodal input systems process both text and
image inputs. This configuration is beneficial for tasks like visual
question answering, where the model answers questions based on
combined text and image information.
Audio and text input Systems that consider both audio and text inputs
are valuable in applications like speech-to-text and multimodal
chatbots.
Multimodal output This includes the following:
Text and image output Some models generate both text and image
outputs simultaneously. This can be observed in tasks like text-to-
image synthesis or image captioning.
Audio and text output In scenarios where both audio and text outputs
are required, such as generating spoken responses based on textual
 input, multimodal output models come into play.
Multimodal input and output This includes the following:
Text, image, and audio input Comprehensive multimodal systems
process text, image, and audio inputs collectively, enabling a broader
understanding of diverse data sources.
Text, image, and audio output Models that produce outputs in
multiple modalities offer versatile responses—for instance, generating
textual descriptions, images, and spoken content in response to a user
query.
The shift to multimodal models is exemplified by pioneering models like
DeepMind’s Flamingo, Salesforce’s BLIP, and Google’s PaLM-E. Now
OpenAI’s GPT-4-visio, a multimodal input model, has entered the market.
Given the current landscape, multimodal output (but input as well) can be
achieved by engineering existing systems and leveraging the integration
between different models. For instance, one can call OpenAI’s DALL-E for
generating an image based on a description from OpenAI GPT-4 or apply the
speech-to-text function from OpenAI Whisper and pass the result to GPT-4.

Note
Beyond enhancing user interaction, multimodal capabilities hold
promise for aiding visually impaired individuals in navigating both
the digital realm and the physical world.

Business use cases

LLMs reshape the landscape of business applications and their interfaces.
Their transformative potential spans various domains, offering a spectrum of
capabilities akin to human reasoning.
For instance, some standard NLP tasks—such as language translation,
summarization, intent extraction, and sentiment analysis—become seamless
with LLMs. They provide businesses with powerful tools for effective
communication and market understanding, along with chatbot applications
for customer services. Whereas historically, when chatting with a chatbot,
people thought, “Please, let me speak to a human,” now it could be the
opposite, as chatbots based on LLMs understand and act in a very human and
effective way.
Conversational UI, facilitated by chatbots based on LLMs, can replace
traditional user interfaces, offering a more interactive and intuitive
experience. This can be particularly beneficial for intricate platforms like
reporting systems.
Beyond specific applications, the true strength of LLMs lies in their
adaptability. They exhibit a human-like reasoning ability, making them
suitable for a diverse array of tasks that demand nuanced understanding and
problem-solving. Think about checking and grouping reviews for some kind
of product sold online. Their capacity to learn from examples (what we will
later call few-shot prompting) adds a layer of flexibility.
This adaptability extends to any kind of content creation, where LLMs can
generate human-like text for marketing materials and product descriptions,
optimizing efficiency in information dissemination. In data analysis, LLMs
derive and extract valuable insights from vast text datasets, empowering
businesses to make informed decisions.
From improving search engines to contributing to fraud detection,
enhancing cybersecurity, and even assisting in medical diagnoses, LLMs
emerge as indispensable tools, capable of emulating human-like reasoning
and learning from examples. However, amidst this technological marvel,
ethical considerations regarding bias, privacy, and responsible data use
remain paramount, underlining the importance of a thoughtful and
considerate integration of LLMs in the business landscape. In essence, LLMs
signify not just a leap in technological prowess but also a profound shift in
how businesses approach problem-solving and information processing.

Facts of conversational programming

In a world of fast data and AI-powered applications, natural language
emerges as a versatile force, serving dual roles as a programming medium
(English as a new programming language) and a user interface. This marks
the advent of Software 3.0. Referring to Andrej Karpathy’s analogies, if
Software 1.0 was “plain and old” code, and Software 2.0 was the neural-
network stack, then Software 3.0 is the era of conversational programming
and software. This trend is expected to intensify as AI becomes readily
available as a product.

The emerging power of natural language

The influence of natural language is multifaceted, serving both as a means of
programming LLMs (usually through prompt engineering) and as a user
interface (usually in chat scenarios).
Natural language assumes the role of a declarative programming
language, employed by developers to articulate the functionalities of the
application and by users to express their desired outcomes. This convergence
of natural language as both an input method for programming and a
communication medium for users exemplifies the evolving power and
versatility of LLMs, where linguistic expressions bridge the gap between
programming intricacies and user interactions.

Natural language as a (new) presentation layer

Natural language in software has evolved beyond its traditional role as a
communication tool and is now emerging as a powerful presentation layer in
various applications.
Instead of relying on graphical interfaces, users can interact with systems
and applications using everyday language. This paradigm shift, thanks to
LLMs, simplifies user interactions, making technology more accessible to a
broader audience and allowing users to engage with applications in an
intuitive and accessible manner.
By leveraging LLMs, developers can create conversational interfaces,
turning complex tasks into fairly simple conversations. To some extent, in
simple software and in specific use cases where, for instance, security is
handled separately, a normal UI is no longer needed. The whole back-end
API can be called through a chat in Microsoft Teams or WhatsApp or
Telegram.

AI engineering
Natural language programming, usually called prompt engineering,
represents a pivotal discipline in maximizing the capabilities of LLMs,
emphasizing the creation of effective prompts to guide LLMs in generating
desired outputs. For instance, when asking a model to “return a JSON list of
the cities mentioned in the following text,” a prompt engineer should know
how to rephrase the prompt (or know which tools and frameworks might
help) if the model starts returning introductory text before the proper JSON.
In the same way, a prompt engineer should know what prompts to use when
dealing with a base model versus an RLHF model.
With the introduction of OpenAI’s GPTs and the associated store, there’s
a perception that anyone can effortlessly develop an app powered by LLMs.
But is this perception accurate? If it were true, the resulting apps would likely
have little to no value, making them challenging to monetize. Fortunately, the
reality is that constructing a genuinely effective LLM-powered app entails
much more than simply crafting a single creative prompt.
Sometimes prompt engineering (which does not necessarily involve
crafting a single prompt, but rather several different prompts) itself isn’t
enough, and a more holistic view is needed. This helps explain why the
advent of LLMs-as-a-product has given rise to a new professional role
integral to unlocking the full potential of these models. Often called an AI
engineer, this role extends beyond mere prompting of models. It encompasses
the comprehensive design and implementation of infrastructure and glue code
essential for the seamless functioning of LLMs.
Specifically, it must deal with two key differences with respect to the
“simple” prompt engineering:
Explaining in detail to an LLM what one wants to achieve is roughly as
complex as writing traditional code, at least if one aims to maintain
 control over the LLM’s behavior.
An application based on an LLM is, above all, an application. It is a
piece of traditional software executed on some infrastructure (mostly on
the cloud with microservices and all that cool stuff) and interacting with
other pieces of software (presumably APIs) that someone (perhaps
ourselves) has written. Moreover, most of the time, it is not a single
LLM that crafts the answer, but multiple LLMs, orchestrated with
different strategies (like agents in LangChain/Semantic Kernel or in an
AutoGen, multi-agent, style).
The connections between the various components of an LLM often
require “traditional” code. Even when things are facilitated for us (as with
assistants launched by OpenAI) and are low-code, we still need a precise
understanding of how the software functions to know how to write it.
Just because the success of an AI engineer doesn’t hinge on direct
experience in neural networks trainings, and an AI engineer can excel by
concentrating on the design, optimization, and orchestration of LLM-related
workflows, this doesn’t mean the AI engineer doesn’t need some knowledge
of the inner mechanisms and mathematics. However, it is true that the role is
more accessible to individuals with diverse skill sets.

LLM topology
In our exploration of language models and their applications, we now shift
our focus to the practical tools and platforms through which these models are
physically and technically used. The question arises: What form do these
models take? Do we need to download them onto the machines we use, or do
they exist in the form of APIs?
Before delving into the selection of a specific model, it’s crucial to
consider the type of model required for the use case: a basic model (and if so,
what kind—masked, causal, Seq2Seq), RLHF models, or custom fine-tuned
models. Generally, unless there are highly specific task or budgetary
requirements, larger RLHF models like GPT-4-turbo (as well as 4 and 3.5-
turbo) are suitable, as they have demonstrated remarkable versatility across
various tasks due to their robust generalization during training.
 In this book, we will use OpenAI’s GPT models (from 3.5-turbo onward)
via Microsoft Azure. However, alternative options exist, and I’ll briefly touch
on them here.

OpenAI and Azure OpenAI

Both of OpenAI’s GPT models, OpenAI and Azure OpenAI, stem from the
same foundational technology. However, each product offers different service
level parameters such as reliability and rate limits.
OpenAI has developed breakthrough models like the GPT series, Codex,
and DALL-E. Azure OpenAI—a collaboration between Microsoft Azure and
OpenAI—combines the latter’s powerful AI models with Azure’s secure and
scalable infrastructure. Microsoft Azure OpenAI also supports models
beyond the GPT series, including embedding models (like text-embedding-
ada-002), audio models (like Whisper), and DALL-E. In addition, Azure
OpenAI offers superior security capabilities and support for VNETs and
private endpoints—features not available in OpenAI. Furthermore, Azure
OpenAI comes with Azure Cognitive Services SLA, while OpenAI currently
provides only a status page. However, Azure OpenAI is available only in
limited regions, while OpenAI has broader accessibility globally.

Note
Data submitted to the Azure OpenAI service remains under the
governance of Microsoft Azure, with automatic encryption for all
persisted data. This ensures compliance with organizational security
requirements.

Users can interact with OpenAI and Azure OpenAI’s models through
REST APIs and through the Python SDK for both OpenAI and Azure
OpenAI. Both offer a web-based interface too: Playground for OpenAI and
the Azure OpenAI Studio. ChatGPT and Bing Chat are based on models
hosted by OpenAI and Microsoft Azure OpenAI, respectively.
 Note
Azure OpenAI is set for GPT-3+ models. However, one can use
another Microsoft product, Azure Machine Learning Studio, to
create models from several sources (like Azure ML and Hugging
Face, with more than 200,000 open-source models) and import
custom and fine-tuned models.

Hugging Face and more

Hugging Face is a platform for members of the ML community to collaborate
on models, datasets, and applications. The organization is committed to
democratizing access to NLP and deep learning and plays an important role
in NLP.
Known for its transformers library, Hugging Face provides a unified API
for pretrained language models like Transformers, Diffusion, and Timm. The
platform empowers users with tools for model sharing, education through the
Hugging Face Course, and diverse ML implementations. Libraries support
model fine-tuning, quantization, and dataset sharing, emphasizing
collaborative research.
Hugging Face’s Enterprise Hub facilitates private work with transformers,
datasets, and open-source libraries. For quick insights, the Free Inference
widget allows code-free predictions, while the Free Inference API supports
HTTP requests for model predictions. In production, Inference Endpoints
offer secure and scalable deployments, and Spaces facilitate model
deployment on a user-friendly UI, supporting hardware upgrades.

Note
Notable alternatives to Hugging Face include Google Cloud AI,
Mosaic, CognitiveScale, NVIDIA’s pretrained models, Cohere for
enterprise, and task-specific solutions like Amazon Lex and
Comprehend, aligning with Azure’s Cognitive Services.
The current LLM stack
LLMs can be used as a software development tool (think GitHub Copilot,
based on Codex models) or as a tool to integrate in applications. When used
as a tool for applications, LLMs make it possible to develop applications that
would be unthinkable without them.
Currently, an LLM-based application follows a fairly standard workflow.
This workflow, however, is different from that of traditional software
applications. Moreover, the technology stack is still being defined and may
look different within a matter of a few months.
In any case, the workflow is as follows:
1. Test the simple flow and prompts. This is usually via Azure OpenAI
Studio in the Prompt Flow section, or via Humanloop, Nat.dev, or the
native OpenAI Playground.
2. Conceive a real-world LLM application to work with the user in
response to its queries. Versel, Streamlit, and Steamship are common
frameworks for application hosting. However, the application hosting is
merely a web front end, so any web UI framework will do, including
React and ASP.NET.
3. When the user’s query leaves the browser (or WhatsApp, Telegram, or
whatever), a data-filter tool ensures that no unauthorized data makes it to
the LLM engine. A layer that monitors for abuse may also be involved,
even if Azure OpenAI has a default shield for it.
4. The combined action of the prompt and of orchestrators such as
LangChain and Semantic Kernel (or a custom-made piece of software)
builds the actual business logic. This orchestration block is the core of
an LLM application. This process usually involves augmenting the
available data using data pipelines like Databricks and Airflow; other
tools like LlamaIndex (which can be used as an orchestrator too); and
vector databases like Chroma, Pinecone, Qdrant, and Weaviate—all
working with an embedding model to deal with unstructured or semi-
 structured data.
5. The orchestrator may need to call into an external, proprietary API,
OpenAPI documented feeds, and/or ad hoc data services, including
native queries to databases (SQL or NoSQL). As the data is passed
around, the use of some cache is helpful. Frequently used libraries
include GPTCache and Redis.
6. The output generated by the LLM engine can be further checked to
ensure that unwanted data is not presented to the user interface and/or a
specific output format is obtained. This is usually performed via
Guardrails, LMQL, or Microsoft Guidance.
7. The full pipeline is logged to LangSmith, MLFlow, Helicone,
Humanloop, or Azure AppInsights. Some of these tools offer a
streamlined UI to evaluate production models. For this purpose, the
Weight & Biases AI platform is another viable option.

Future perspective
The earliest LLMs were a pipeline of three simpler neural networks like
RNNs, CNNs, and LSTMs. Although they offered several advantages over
traditional rule-based systems, they were far inferior to today’s LLMs in
terms of power. The significant advancement came with the introduction of
the transformer model in 2017.
Companies and research centers seem eager to build and release more and
more advanced models, and in the eyes of many, the point of technological
singularity is just around the corner.
As you may know, technological singularity describes a time in some
hypothetical future when technology becomes uncontrollable, leading to
unforeseeable changes in human life. Singularity is often associated with the
development of some artificial superintelligence that surpasses human
intelligence across all domains. Are LLMs the first (decisive) step toward this
kind of abyss? To answer this question about our future, it is necessary to
first gain some understanding of our present.
Current developments
In the pre-ChatGPT landscape, LLMs were primarily considered research
endeavors, characterized by rough edges in terms of ease of use and cost
scaling. The emergence of ChatGPT, however, has revealed a nuanced
understanding of LLMs, acknowledging a diverse range of capabilities in
costs, inference, prediction, and control. Open-source development is a
prominent player, aiming to create LLMs more capable for specific needs,
albeit less cumulatively capable. Open-source models differ significantly
from proprietary models due to different starting points, datasets, evaluations,
and team structures. The decentralized nature of open source, with numerous
small teams reproducing ideas, fosters diversity and experimentation.
However, challenges such as production scalability exist.
Development paths have taken an interesting turn, emphasizing the
significance of base models as the reset point for wide trees of open models.
This approach offers open-source opportunities to advance, despite
challenges in cumulative capabilities compared to proprietary models like
GPT-4-turbo. In fact, different starting points, datasets, evaluation methods,
and team structures contribute to diversity in open-source LLMs. Open-
source models aim to beat GPT-4 on specific targets rather than replicating its
giant scorecard.
Big tech, both vertical and horizontal, plays a crucial role. Vertical big
tech, like OpenAI, tends to keep development within a walled garden, while
horizontal big tech encourages the proliferation of open source. In terms of
specific tech organizations, Meta is a horizontal player. It has aggressively
pursued a “semi” open-source strategy. That is, although Llama 2 is free, the
license is still limited and, as of today, does not meet all the requirements of
the Open Source Initiative.
Other big tech players are pursuing commercially licensed models, with
Apple investing in its Ajax, Google in its Gemini, PaLMs and Flan-T5, and
Amazon in Olympus and Lex. Of course, beyond the specific LLMs backing
their applications, they’re all actively working on incorporating AI into
productivity tools, as Microsoft quickly did with Bing (integrated with
OpenAI’s GPTs) and all its products.
 Microsoft’s approach stands out, leveraging its investment in OpenAI to
focus more on generative AI applications rather than building base models.
Microsoft’s efforts extend to creating software pieces and architecture around
LLMs—such as Semantic Kernel for orchestration, Guidance for model
guidance, and AutoGen for multi-agent conversations—showcasing a holistic
engineering perspective in optimizing LLMs. Microsoft also stands out in
developing “small” models, sometimes called small language models
(SLMs), like Phi-2.
Indeed, engineering plays a crucial role in the overall development and
optimization process, extending beyond the realm of pure models. While
direct comparisons between full production pieces and base models might not
be entirely accurate due to their distinct functionalities and the engineering
involved in crafting products, it remains essential to strive to maximize the
potential of these models within one’s means in terms of affordability. In this
context, OpenAI’s strategy to lower prices, announced along with GPT-4-
turbo in November 2023, plays a key role.
The academic sector is also influential, contributing new ways of
maximizing LLM performance. Academic contributions to LLMs include
developing new methods to extract more value from limited resources and
pushing the performance ceiling higher. However, the landscape is changing,
and there has been a shift toward collaboration with industry. Academia often
engages in partnerships with big tech companies, contributing to joint
projects and research initiatives. New and revolutionary ideas—perhaps
needed for proper artificial general intelligence (AGI)—often come from
there.
Mentioning specific models is challenging and pointless, as new open-
source models are released on a weekly basis, and even big tech companies
announce significant updates every quarter. The evolving dynamics suggest
that the development paths of LLMs will continue to unfold, with big tech,
open source, and academia playing distinctive roles in shaping the future of
these models.

What might be next?

OpenAI’s GPT stands out as the most prominent example of LLMs, but it is
not alone in this category. Numerous proprietary and open-source alternatives
exist, including Google’s Gemini, PaLM 2, Meta’s Llama 2, Microsoft Phi-2,
Anthropic’s Claude 2, Vicuna, and so on. These diverse models represent the
state of the art and ongoing development in the field.
Extensively trained on diverse datasets, GPT establishes itself as a potent
tool for NLP and boasts multimodal capabilities. Gemini demonstrates
enhanced reasoning skills and proficiency in tackling mathematical
challenges. At the same time, Claude 2 excels at identifying and reacting to
emotions in text. Finally, LLaMA is great in coding tasks.
Three factors may condition and determine the future of LLMs as we
know them today:
Fragmentation of functions No model is great at everything, and each
is already trained based on billions of parameters.
Ethical concerns As models aggregate more functions and become
more powerful, the need for rules related to their use will certainly arise.
Cost of training Ongoing research is directed toward reducing the
computational demands to enhance accessibility.
The future of LLMs seems to be gearing toward increasingly more
efficient transformers, increasingly more input parameters, and larger and
larger datasets. It is a brute-force approach to building models with improved
capabilities of reasoning, understanding the context, and handling different
input types, leveraging more data or higher-quality data.
Beyond the model, prompt engineering is on the rise, as are techniques
involving vector database orchestrators like LangChain and Semantic Kernel
and autonomous agents powered by those orchestrators. This signals a
maturation of novel approaches in the field. Future LLM challenges have,
though, a dual nature: the need for technical advancements to enhance
capabilities and the growing importance of addressing ethical considerations
in the development and deployment of these models.

Speed of adoption
Considering that ChatGPT counted more than 100 million active users within
two months of its launch, the rapid adoption of LLMs is evident. As
highlighted by various surveys during 2023, more than half of data scientists
and engineers plan to deploy LLM applications into production in the next
months. This surge in adoption reflects the transformative potential of LLMs,
exemplified by models like OpenAI’s GPT-4, which show sparks of AGI.
Despite concerns about potential pitfalls, such as biases and hallucinations, a
flash poll conducted in April 2023 revealed that 8.3% of ML teams have
already deployed LLM applications into production since the launch of
ChatGPT in November 2022.
However, adopting an LLM solution in an enterprise is more problematic
than it may seem at first. We all experienced the immediacy of ChatGPT and,
sooner or later, we all started dreaming of having some analogous chatbot
trained on our own data and documents. This is a relatively common
scenario, and not even the most complex one. Nonetheless, adopting an LLM
requires a streamlined and efficient workflow, prompt engineering,
deployment, and fine-tuning, not to mention an organizational and technical
effort to create and store needed embeddings. In other words, adopting an
LLM is a business project that needs adequate planning and resources, not a
quick plug-in to some existing platform.
With LLMs exhibiting a tendency to hallucinate, reliability remains a
significant concern, necessitating human-in-the-loop solutions for
verification. Privacy attacks and biases in LLM outputs raise ethical
considerations, emphasizing the importance of diverse training datasets and
continuous monitoring. Mitigating misinformation requires clean and
accurate data, temperature setting adjustments, and robust foundational
models.
Additionally, the cost of inference and model training poses financial
challenges, although these are expected to decrease over time. Generally, the
use of LLM models requires some type of hosting cloud via API or an
executor, which may be an issue for some corporations. However, hosting or
executing in-house may be costly and less effective.
The adoption of LLMs is comparable to the adoption of web technologies
25 years ago. The more companies moved to the web, the faster technologies
Random documents with unrelated
content Scribd suggests to you:
 Poa fertilis.

*Polygonatum multiflorum (Solomon’s Seal).—This, one of the most

graceful of our native plants, is too distinct and pleasing in aspect to be
omitted from an enumeration of ornamental subjects. It is best suited for a
shady position under trees, or the fringes of shrubberies, or groups of tall
and widely-branching plants, where its elegantly arching stems and pretty
pendent flowers would attain greatest perfection, and exhibit a very marked
contrast to the surrounding types of vegetation. A well-drained, sandy, and
peaty soil will be found to suit it best, and it is well adapted for the wild and
semi-wild parts of the pleasure-ground. Easily multiplied by division of the
rootstocks.
*Polygonum cuspidatum (Sieboldi).—Forms large and noble tufts of
lively green, which increase in beauty from year to year. It grows to a
height of 3½ ft. to 6 ft. and more, the stems being at first erect and simple,
then becoming much branched, the branches arching, and spreading nearly
horizontally at the top. The white flowers, which are disposed in clusters
forming close panicles, are succeeded by handsome rosy-white fruit. When
planted singly, and away from other subjects, its head assumes a rather
peculiar and pretty arching character; and therefore it is not quite fit for
forming centres or using in groups, so much as for planting singly on the
turf, there leaving it to take care of itself and come up year after year. In this
way it would be particularly useful in the pleasure-ground or diversified
English flower-garden. It is also good for any position in which a bold and
distinct type of vegetation is desired, while of course, when we come to
have fine groups of hardy “foliage-plants” in our gardens, its use will be
much extended. The deeper and better the soil, the finer will its
development prove. You cannot make the soil too deep and good if you
want the plant to assume a very striking character. It runs very much at the
root in all directions horizontally just below the surface of the soil. By
cutting away the runners, and thus concentrating the sap in the central
stems, tufts have been obtained from 9 ft. to 13 ft. high, and as much across.
Japan.
Polymnia grandis. (See Montagnæa heracleifolia.)
Polymnia pyramidalis.—A free, bold, and tall plant, with somewhat of
a sunflower habit, but withal very fine, and making a tall green growth by
the end of July, before many other things used in this way begin to push.
The leaves are not so large as those of the other species, and differ in shape,
being nearly cordate; but the growth is vigorous, and the habit distinct. It
pushes up a narrow pyramidal head of foliage to a height of nearly 10 ft. in
Paris gardens, and will be found to do well in the south of England. Easily
multiplied in spring by division or cuttings from plants placed in heat in
January, or from seed sown in a pot. The species Uvedalia and maculata are
rather coarse herbaceous plants, fitted for rough places in warm positions,
and deep, rich soil. New Granada.
*Pontederia cordata.—One of the handsomest water-plants in
cultivation, combining gracefulness of habit and leaf with beauty of flower.
It forms thick tufts of almost arrow-shaped, erect, long-stalked leaves from
1½ ft. to more than 2 ft. high, crowned with the handsome blue flower-
spikes, which issue from the leafstalks just below the base of the leaves. It
should be planted in shallow pieces of water. Multiplied by division of the
tufts at any season. N. America.
Pothos acaulis.—A noble plant of the Arum family, with huge simple
dark-green leaves, forming a magnificent rosette. It requires stove treatment
in winter and spring, and having made its growth and been hardened off
under cover, it may be placed out in the open air in sheltered warm places in
the southern counties, from the middle of June to the end of September. It,
however, sometimes suffers from cold, and is on the whole only likely to be
of very partial use. Among the other large Aroids which have been tried in
Battersea Park, the best are Philodendron macrophyllum, P. Simsii, and
Anthurium Hookeri. Being rather tender they all require a very warm and
well-sheltered position. W. Indies.
*Rhaponticum cynaroides.—A hardy perennial from the Pyrenees, 3 ft.
or more in height, with a rigid, simple, furrowed stem, and lobed or entire
oblong radical leaves, covered underneath with silvery down. Flowers
solitary, purple, in very large heads. It thrives in a deep, substantial, moist,
but well-drained and free soil. This and, to a smaller extent, the following
species are worthy of a place in full collections of hardy fine-leaved plants,
for borders, the margins of groups, and, in the case of R. cynaroides, for
isolation. Both are easily increased by division.
*Rhaponticum pulchrum.—A hardy perennial from the Caucasus, with
numerous simple stems 2 ft. or more in height, and much-divided,
undulating, toothed leaves of an ashy or sea-green colour on the upper side,
and whitish underneath. The flowers are borne in small solitary purplish
heads late in summer. A very suitable subject for embellishing dry, arid,
rocky positions. R. scariosum is another kind, useful for the same purposes
as the preceding species.

*THE RHUBARBS.
The Rhubarbs, from their vigour and picturesqueness, are well worthy of
cultivation among hardy, fine-leaved plants. They are so hardy that they
may be planted in any soil, and afterwards left to take care of themselves.
Their fine leaves and bold habit make them valuable ornaments for the
margins of shrubberies (the best way is to plant one singly a few feet from
the margin of the shrubbery, so that when they die down in autumn no blank
may be seen), and for semi-wild places where a very free and luxuriant type
of vegetation is desired. Though not particular as to soil, they enjoy it when
it is deep and rich, and the more it is made so the better they will grow.
Rheum Emodi is undoubtedly the handsomest and most distinct of the
genus in cultivation. The figure conveys an accurate idea of the outline of
its leaves, and of its aspect when in flower. The large leaves have their veins
red, which distinguishes it from any other species. It has a large and deep-
feeding root, black on the outside, and yellow within. The flowers are very
small, of a yellowish white. It comes up somewhat later than the common
kinds, and is not by any means common, though it may be found in botanic
gardens and nurseries where collections of herbaceous plants are formed. It
may, like all the species, be increased by division, but a young plant should
not be disturbed for several years after being planted. It is a native of
Nepaul.
 The palmated rhubarb, Rheum palmatum, is immediately distinguished
from its cultivated fellows by its leaves

RHEUM EMODI.
Hardy herbaceous fine-foliaged Type.

being deeply cut into lobes. It is scarcely so ornamental or imposing as the

fuller-leaved kinds, but is an interesting plant. I have seen it grown in some
Irish gardens for culinary purposes, but do not remember to have noticed it
in English kitchen-gardens. When well grown in deep and rather light and
well-drained earth, the flowering-stem of this species attains a height of
about nine feet. It is a native of Tartary, and well deserves a place. Rheum
Ribes is a somewhat delicate species, a native of Southern Persia, which
may be seen in one or two of our botanic gardens, and more frequently in
those of France; but we fear it is not hardy enough to thrive well in these
islands. The most remarkable known species is the Himalayan (R. nobile),
which has its flower-stems beautifully clothed with large straw-coloured
and pink-edged bracts, so as to form what may be termed a pyramid of
leaves; but, as this species is not in cultivation, we need not describe it
further. It would prove a very welcome addition to our collection of hardy
plants.
The common rhubarbs are said by some of our authorities to have
chiefly sprung from Rheum Rhaponticum, and some of them have also
come from the wavy-leaved Rheum (R. undulatum). In any case, some of
the garden varieties of rhubarb are worth planting for ornamental purposes.
They have been so planted in Hyde Park, but in masses—not the proper
way to employ them. Kinds deserving of notice are R. australe, R.
compactum, R. rugosum, R. hybridum, Victoria rhubarb (a garden variety,
with very large leaves and long red stalks), Myatt’s Linnæus, and Prince
Albert (also garden varieties, and splendid ornamental plants). Mr. Shirley
Hibberd says he has found Scott’s Monarch to be the most imposing and
ornamental of all the garden varieties.
*Rhus Cotinus (Venetian Sumach).—A bushy shrub, about 6½ ft. high,
with simple, smooth, shining green leaves, and a very remarkable feathery
inflorescence of a deep red colour. It requires a dry, gravelly, warm soil, and
will grow in the most arid positions. Where it thrives the effect of its
peculiar inflorescence is very fine and distinct. It is used with most effect as
an isolated specimen, though it would group very well with such plants as
Lindley’s Spiræa. S. Europe.
*Rhus glabra laciniata.—This variety of the smooth or scarlet Sumach
is a small shrub with compound leaves, growing from 4 ft. to 7 ft. high, a
native of North America, with finely-cut and elegant leaves, the strongest
being about a foot long when the plants have been established a year or two.
When seen on an established plant, these leaves combine the beauty of
those of the finest Grevillea with that of a fern frond, while the youngest
and unfolding leaves remind one of the aspect of a finely-cut umbelliferous
plant in spring. The variety observable in the shape, size, and aspect of the
foliage makes the plant charming to look upon, while the midribs of the
fully-grown leaves are red, and in autumn the whole glow off into bright
colour after the fashion of American shrubs and trees. During the entire
season it is presentable, and there is no fear of any vicissitude of weather
injuring it. Its great merit is that, in addition to being so elegant in foliage, it
has a very dwarf habit, and is thoroughly hardy. Plants three years old and
undisturbed for the last two years are not more
 RHUS GLABRA LACINIATA.
Hardy deciduous Shrub Section.

than eighteen inches high. The heads of some are branched, but these are
not less elegant than when in a simple-stemmed state, so that here we have
clearly a subject that will afford a charming fern-like effect in the full sun,
and add graceful verdure and distinction to the flower-garden. When the
flowers show after the plant is a few years old, they may be pinched off; but
this need only be practised in the case of permanent groups or plantings of
it. To produce the effect of a Grevillea or a fern on a small scale, we should
of course keep this graceful Rhus small and propagate it like a bedding-
plant. Like most other shrubs, it has a tendency to branch; but to fully enjoy
the beauty of the leaves it is best to cut down the plants yearly, as then the
leaves given off from the simple erect stem are much larger and more
graceful. It will, however, be necessary to allow it to become established
before treating it in this way, as it is at present comparatively new to our
gardens. The figure, sketched early in August, represents a young plant little
more than a foot high, which had been cut down to the ground during the
spring of the past year, and proves that its full beauty may be enjoyed in a
very small state. It may be most tastefully used in association with bedding-
plants, or on banks in or near the rock-garden or hardy fernery, planting it in
light sandy loam. The graceful mixtures and bouquet-*like beds that might
be made with the aid of such plants need not be suggested here, while of
course an established plant, or groups of three, might well form the centre
of a bed. Planting a very small bed or group separately in the flower-garden,
and many other uses which cannot be enumerated here, will occur to those
who have once tried it. Some hardy plants of fine foliage are either so
rampant or so topheavy that they cannot be wisely associated with bedding-
plants. This is, on the contrary, as tidy and tractable a grower as the most
fastidious could desire. It would be a mistake to put such a pretty plant
under or near rough trees and shrubs. Give it the full sun, and good free
soil.
*Rhus vernicifera is distinct from the preceding, and has fine leaves. It
is a native of Japan, and the source of the best Japan varnish according to
Thunberg. Useful for grouping with the preceding or other hardy shrubs of
like character.
*Ricinus communis (Castor-oil Plant).—When well grown in the open
air, there is not in the whole range of cultivated plants a more imposing
subject than this. It may have been seen nearly 12 ft. high in the London
parks of late years, and with leaves nearly 1 yd. wide. It is true we require a
bed of very rich deep earth under it to make it attain such dimensions and
beauty; but in all parts, and with ordinary attention, it grows well. In warm
countries, in which the plant is very widely cultivated, it becomes a small
tree, but is much prettier in the state in which it is seen with us—i.e., with
an unbranched stem clothed from top to bottom with noble leaves. Soon
after it betrays, a tendency to develope side-shoots the cold autumn comes
and puts an end to all further progress; and so much the better, because it is
much handsomer in a simple-stemmed state than any other. The same is true
of not a few other large-leaved plants—once they break into a number of
side-shoots their leaf beauty is to a great extent lost. It is as easily raised
from seed as the common bean, requiring, however, to be raised in heat. It
should be sown about the middle of February, and the plants gradually
hardened off so as to be fit to put out by the middle of May. The Ricinus is a
grand plant for making bold and noble beds near those of the more brilliant
flowers, and tends to vary the flower-garden finely. It is not well to
associate it closely with bedding-plants, in consequence of the strong
growth and shading power of the leaves, so to speak. A good plan is to
make a compact group of the plant in the centre of some wide circular bed
and surround it with a band of a dwarfer subject, say the Aralia or
Caladium, and then finish with whatever arrangement of the flowering
plants may be most admired. A bold and striking centre may be obtained,
while the effect of the flowers is much enhanced, especially if the planting
be nicely graduated and tastefully done. For such groups the varieties of the
Castor-oil plant are not likely to be surpassed. East Indies.
The most notable varieties are R. c. sanguineus, the stem, leaf-stalks,
young leaves, and fruit of which are of a blood-red colour; R. c.
borboniensis, which in southern climates often attains the extraordinary
height of 26 ft. in one year; R. c. giganteus, a very tall kind from the
Philippine Islands.
Other kinds in cultivation are R. Belot Desfougerès (a very tall and
branching kind), R. viridis (of a uniform lively green colour), R. insignis, R.
africanus, R. africanus albidus, R. minor, R. hybridus, R. microcarpus.
The better and richer the soil, and the warmer the position, the more
vigorous will be the growth of any of the above. Copious watering in
summer is indispensable.
*Rumex Hydrolapathum.—A very large native water-plant of a size
and habit sufficiently striking to entitle it to a place amongst ornamental
subjects by the water-side. The radical long-stalked leaves, which are
sometimes 2 ft. or more in length, form erect tufts of a very imposing
character. The flowering-stem is frequently 6 ft. in height, and bears a very
large, dense, pyramidal panicle of a reddish or olive-fawn colour. The plant
is most effective in autumn, when the leaves change to a lurid red colour,
which they retain for some time.
*Saccharum ægyptiacum.—A vigorous perennial grass, forming ample
tufts of reed-like downy stems 6½ ft. to 13 ft. high, and clothed with very
graceful foliage, well adapted for ornamenting the margins of pieces of
water, the slopes and other parts of pleasure-grounds, etc., in a warm
position. In our climate it does not flower, but even without its fine feathery
plumes it is a pretty plant from its foliage and habit alone. Easily and
quickly multiplied by division in spring; the offsets to be started in a frame
or pit. When established they may be planted out in May or June. N. Africa.
*Sagittaria sagittifolia.—A British water-plant, affording the most
remarkable example of the arrow-shaped leaf to be met with among hardy
plants. These leaves stand erect, from 1 ft. to 1½ ft. above the water, and
from the middle of the tuft the flowering-stem rises in August to the height
of 1½ ft. to 2½ ft. The flowers are of a pale rosy-white colour. There is a
variety with double flowers (S. sagittifolia flore pleno), which resemble the
flowers of the double Rocket. Both the double and single kinds should have
a place among water or bog plants.

SEAFORTHIA ELEGANS.
Conservatory Palm; standing well in the open air in summer.

*Salvia argentea (S. patula).—A handsome biennial about 3½ ft. high,

forming broad spreading rosettes of large, oval, heart-shaped leaves,
densely covered with long silky hairs of a silvery whiteness, and bearing
large panicles of white flowers. An excellent subject for grouping on grass-
plats or the uneven parts of pleasure-grounds. Where the effect of the
foliage only is desired the flower-stems should be pinched off as soon as
they appear; the leaves will then preserve their freshness and silvery colour
throughout the year. A light, sandy, or gravelly soil is the most suitable.
Easily propagated by sowing in autumn, and keeping the seedlings in a cold
frame through the winter, giving them air as often as possible, and watering
very moderately. This plant is most effective during its first summer, and
before it makes an attempt to flower. S. Europe.
*Scirpus lacustris (Bulrush).—This giant rush sends up numerous
smooth green stems as thick as the finger at the base, and from 3 ft. to 8 ft.
high. In still water the bases of these are covered with leafless sheaths, but
in running water the uppermost sheath produces at its extremity a leaf of
several inches in length, in addition to which numerous barren tufts of
leaves, often of great length, and resembling those of Sparganium affine,
are also produced. When in flower, the stems are crowned with short,
umbel-like, chestnut-coloured panicles. It is very effective on the margins
of lakes or streams, associated with other tall and imposing aquatic plants.
Seaforthia elegans.—One of the most beautiful of the Palm family,
from the northern parts of Australia, where it attains a height of about 30 ft.,
but in this country seldom arrives at more than half its full size. The leaves
are from 2 ft. to 10 ft. in length, and are divided into numerous narrow
leaflets from 1 ft. to 1½ ft. long, and of a dark green colour. The whole
plant is perfectly smooth, and is one of the finest subjects in cultivation for
the conservatory, greenhouse, or subtropical garden. It may be placed in the
open air from the middle or end of May until the beginning of October. It is
too scarce as yet to be procurable by horticulturists generally, but should be
looked for by all who take an interest in these matters and have a house in
which to grow it. It stands well in the conservatory during the winter,
though generally kept in the stove, where of course it grows beautifully.
There are hardier kinds—the dwarf Fan-palm for example—but on the
whole none of them are so valuable as this.
Senecio Ghiesbreghtii.—A handsome Mexican plant with stout, round,
spotted stems, 3 to 4 ft. high, and large oval-oblong, thick, coarsely-toothed
leaves of a light green colour, and slightly rolled down at the margin.
Flowers small, yellow, very numerous, in corymbose clusters of enormous
size. A useful plant for isolation on grass-plats, or for beds, etc. Young
plants are to be preferred for this purpose, as the old ones are apt to become
bare and ragged-looking at the base. Plant in a mixture of peat soil and free
loam at the end of May. Multiplied by cuttings in winter, struck under glass
in a temperate heat in early spring.
Senecio Petasites (Cineraria platanifolia).—Another Mexican species,
nearly 3½ ft. high, with a stout, half-shrubby, slightly-branching stem, and
large, dark-green, roughish, lobed leaves. It requires a substantial, but free
and cool soil, and may be multiplied at almost any season by cuttings. It
requires greenhouse protection in winter, grows freely in the open air
planted out in early summer, and is suitable for beds or groups associated
with the medium-sized subjects.
*Seseli gummiferum. (Silvery Seseli).—An umbelliferous plant with
elegantly-divided leaves of a peculiarly pleasing glaucous or almost silvery
tone. I am not sure whether this plant is perennial or not, and it is not hardy
on cold soils, having perished during several of our most severe winters, but
it is so unique in its way that some persons might like to grow it, and if so
the best position is on dry and sunny banks, or raised beds or borders. It is
one of the few subjects that are at once fern-like and silvery, and if plentiful
enough might be used to form charming edgings. It is not difficult to raise
from seed, which should be sown soon after being gathered. As it is liable
to perish in winter, perhaps the best way to deal with it would be to put it
out annually at the end of spring, raising it and keeping it in frames for this
purpose; and to secure seeds a few plants might be left in 10 in. or 12 in.
pots, so that the seed might be gathered under cover. Doubtless, however,
continental raisers can supply it. S. Russia.
*Silphium integrifolium.—This composite has a rough, vigorous, rigid,
slightly four-angled, grooved stem, from 2 ft. to 4 ft. high, and leaves from
3 ins. to 5 ins. long, all opposite, lanceolate-ovate, entire, tapering to a sharp
point from a roundish heart-shaped and partly-clasping base, and covered
with a roughish down. The flower-heads are of a greenish-yellow, on short
stalks, in a close, forking corymb. A variety (S. læve) has the leaves and
stem smooth, or nearly so. This and the following kinds are only suited for
the rougher parts of the pleasure-ground, and by wood-walks, etc.; they will
be seen to greatest advantage in rich and deep soil, but will grow in any
kind. Division. N. America.
*Silphium laciniatum (Compass-plant).—A vigorous perennial with a
stout, round stem, often upwards of 8 ft. in height. The leaves, which are
collected chiefly about the base of the plant, are large, wrinkled, and
deeply-divided into lance-shaped, pointed segments, and fringed with white
hairs. The stem-leaves are few, much smaller, and opposite. Flowers of a
fine yellow with a brownish centre, in large, few, solitary, short-stalked,
horizontal or drooping heads, which have the peculiarity of facing to the
east. Division. N. America.
*Silphium perfoliatum (Cup-plant).—A robust-growing North
American perennial from 4 ft. to 8 ft. in height, with a square stem and
broad, opposite, oval, lance-shaped, coarsely toothed leaves, 6 ins. to 15
ins. long, rough on both sides, the upper ones united at their bases; the
lower ones abruptly narrowed into winged leaf-stalks, which are also united
at their bases. Flower-heads about 2 ins. across, with a greenish-yellow disk
and a yellow ray. Does best in a deep, free, well-drained, sandy soil, but
will grow well when the ground is slightly moist, especially in warm, but
not too shady, positions. Division.
*Silphium terebinthinaceum (Prairie-Dock).—A large kind with
smooth slender stems from 4 ft. to 10 ft. high, panicled at the summit, and
bearing many small heads of light yellow flowers. The leaves are ovate-
oblong, thick and rough, especially beneath, and from 1 ft. to 2 ft. long, on
slender stalks. A variety (pinnatifidum) has the leaves deeply cut or
pinnatifid. This species is remarkable for its strong turpentine odour.
Division. N. America.
*Silphium trifoliatum.—This species has a smooth, often glaucous,
rather slender stem, from 4 ft. to 6 ft. high, branching above. Leaves lance-
shaped, pointed, entire or scarcely serrate, short-stalked, in whorls of three
or four, the uppermost ones opposite. Flower-heads yellow, more than 2 ins.
across, on long stalks, and forming loose panicles. Division. N. America.
*Silybum marianum (Milk-thistle).—A very robust and vigorous-
growing native biennial, 5 ft. or more in height, of strikingly handsome
appearance, and well deserving to be associated with other large fine-
foliaged plants. Its leaves are of very great size, variously cut and
undulated, tipped and margined with scattered spines, and of a bright
glistening green colour marbled and variegated with broad white veins.
Easily raised from seed, and thrives in almost any kind of well-drained soil.
Additional vigour and development may be thrown into the foliage by
pinching off the flower-stems on their first appearance. If a few plants are
raised in the garden and planted out in rough and somewhat bare places or
banks, etc., this will soon establish itself permanently.
Silybum eburneum is a more tender species, very closely resembling
the above, but with spines which appear as if made of ivory. It is also more
constantly biennial, and in consequence its leaves are almost always in the
rosette stage throughout the first year. It is somewhat tenderer than S.
marianum. Algeria.
The Solanums.—This family, so wonderfully varied, affords numerous
species that look graceful and imposing in leaf when in a young and free-
growing state. In selecting examples from this great genus we must be
careful, as our climate is a shade too cold for some of the kinds grown on
the continent, and many of them are of too ragged an aspect to be tolerated
in a tasteful garden. Half a dozen species or so are indispensable, but there
is quite a crowd of narrow-leaved and ignoble ones which may well be
passed over.
Most of these plants may be raised from seed, while they are also freely
grown from cuttings, which struck in February will make good plants by
May. All the kinds named are suitable for association with the larger-leaved
plants, though they do not as a rule attain such height and vigorous
development as those of the first rank, like the Ricinus. As a rule,
temperate-house treatment in winter is required, and they should be planted
out about the middle or end of May, in rich fight soil, a warm position, and
perfect shelter. S. marginatum, planted in a very dwarf and young state,
furnishes a most distinct and charming effect: it should be planted rather
thinly, so that the leaves of one plant may not brush against those of
another. If some very dwarf plants are used as a groundwork, so much the
better; but the downy and silvery leaves of this plant are sure to please
without this aid. It is very much better when thus grown than when
permitted to assume the bush form.
Solanum betaceum.—A small tree from South
 SOLANUM ROBUSTUM.
Tender Section; making vigorous growth during the summer months.

America, which in our climate attains a height of nearly 10 ft. if taken up

in autumn and kept through the winter in a house. The stems are stout,
smooth, and fleshy. The leaves, which resemble those of the Beet, are of an
oval, pointed shape, and of a deep green colour, tinged with violet in the
variety purpureum. The flowers are small, rose-colour, in pendent cyme-
like clusters, and are succeeded by fruit of the shape and size of a fowl’s
egg, which become of a fine deep scarlet colour during the winter. Some
varieties have flowers tinged with purple and fruit striped with brown. May
be placed to great advantage in groups in round beds with dwarfer plants or
shrubs at the base, or with climbing plants ascending the stems, but is much
better isolated on slopes, etc. It is a vigorous grower, and should have rich
soil.
Solanum crinitipes.—A slow-growing woody species with undivided
oval leaves somewhat more than a foot long: the young stems and flower-
stalks being densely covered with chaffy hairs somewhat like those of a
fern. This I have not seen thrive so well in England as the preceding kind,
but it is well worthy of trial in full collections in the southern counties. S.
America.
 Solanum crinitum.—A vigorous-growing species from Guiana, 5 ft. or
more in height, with stout stems, set with short strong spines and dense long
hairs. It has very large, soft, hairy, spreading, roundish leaves, which in
good soil attain a length of 2½ ft.: the upper surface of a tender green colour
with violet veinings set with spines, as are also the leaf-stalks; the under
side whitish and more thickly furnished with spines. The hairs and bark on
the upper portion of each petiole are of a purplish hue, and, on the lower
part, of a light pale green, by which the plant may be readily recognised.
The flowers are very large and white. Berries roundish, villose, and twice or
thrice as large as a cherry. This I have seen attain a very remarkable
development in sheltered warm spots in the south of England. It is fine in
medium-sized groups.
Solanum hyporhodium.—A fine branching kind from Venezuela, with
a stout stem about 5 ft. high, and branches armed with short thinly-scattered
spines. The leaves, which attain a length of nearly 2½ ft., are oval, with
angular sinuated lobes, the upper surface being of a fine green colour with
white veins, and the under side of a violet-red and downy. When young the
hue of the leaves is exceedingly lively. The flowers are borne in almost
lateral cymes and are of a rosy-white colour with yellow stamens. This
plant is sometimes sold as S. discolor and S. purpureum, but is quite distinct
from them.
Solanum Karstenii.—This, which is more commonly known as S.
callicarpum, is a robust, slightly branching, arborescent shrub about 5 ft.
high, covered with long hairs interspersed with spines and of a general
variable greyish-violet hue. The leaves are oval, broad, angular, heart-
shaped at the base, and 2 ft. or more in length. The flowers are large, of a
fine delicate violet colour, and borne in crowded, almost one-sided clusters.
This plant is best isolated, as when placed in close groups the leaves of the
associated subjects are apt to tear it. Venezuela.
Solanum lanceolatum.—This is the best kind for blooming qualities.
The foliage, which is somewhat fluffy and willow-like, possesses no
marked character, but the mauve-coloured flowers are borne abundantly in
clusters, each containing 20 or more blooms: the stamens, being of an
orange colour, add to the effect. There are a dozen or more species that
flower freely but have little beauty of leaf: among the best of these is S.
Rantonnettii, which has very pretty dark-purple flowers, more than an inch
across, with an orange centre. It forms a neat bush, and flowers freely in the
southern counties, in warm sunny spots and on light soils. Mexico.
Solanum macranthum.—A fine species from Brazil, confessedly one
of the best kinds in cultivation, and somewhat resembling Polymnia
grandis. It grows nearly 7 ft. high in one year, with a stout, simple, spiny
stem of a deep shining green with grayish spots, and sparsely armed with
very strong shortish spines. The leaves are elegant and deeply cut, some of
them over 2½ ft. long, falling gracefully earthwards, of a light green on the
upper surface, with red veinings, the under side having a reddish hue. The
flowers, seldom seen with us, are of a fine violet colour, and grow in
corymbs. It will not attain its full character and large dimensions in cold
places, and should therefore have as warm positions as possible. Increased
by cuttings struck in February: they are fit to plant out in May.
Solanum marginatum.—A vigorous-growing, erect, branching and
bushy species from Abyssinia, 3 ft., or more, in height. The leaves are
somewhat oval, with a bluntly sinuated margin; the upper surface smooth,
of a brilliant green with a white silvery border, and the under side covered
with a white satiny down. The flowers are white, with orange stamens,
pendulous, very numerous, in clusters. For the positions suited to this plant
see the introductory remarks on the genus.
Solanum Quitoense.—A half-shrubby native of Peru and the
neighbourhood of Quito, seldom growing higher than 3¼ ft. in cultivation.
The stem is spineless, covered with a soft down, and of a delicate green
colour suffused with violet, which exhibits iridescent changes. The leaves
are broad, stalked, obcordate, with toothed angles, and of a fine green
colour, with violet downy veins. Flowers rather large, white, tinged with
lilac on the top when in bud, in short clusters. It requires a warm position
and a warm season to bring out its best qualities.
Solanum robustum.—A Brazilian species with a vigorous much-
branching stem more than 3 ft. high, and furnished with very sharp and
strong spines and densely-set, long, reddish, viscous hairs. The leaves,
which are very large, are of a rich brown colour on the upper surface and
oval-elliptical in form, with 8 or 9 oval-acute lobes, the upper ones nearly
triangular; and the midrib and principal veins, which are of a brown colour,
are closely set with spines similar to those on the stem. The flowers are
white, with orange stamens, and are borne in unilateral clusters. The berries
are round, of a brown colour, and the size of a small cherry. As a foliage-
plant this is a subject of considerable merit, and one of those most suitable
for our climate. It requires a warm sunny aspect in a position which will be
at the same time airy and sheltered from strong winds.
Solanum Sieglingii.—A large and handsome kind, which forms a small
tree about 13 ft. high after some

SOLANUM WARSCEWICZII.
Tender Section; making noble leaves in the open garden in summer.

years’ growth. The foliage is of a light-green colour, tinged here and there
with rose, and sparsely armed with spines; the young unfolded leaves are
slightly tinged with violet. Flowers numerous, small and white, appearing
when the plant is two or three years old. A good kind which has been little
tried in England. Venezuela.
Solanum Warscewiczii.—A very fine and ornamental kind, resembling
S. macranthum, but with a lower and more thickset habit, and branching
more at the base. The leaf-stalks also, and upper branches, are of a red
colour, glandular, and scaly; and the flowers are white and small. The stem
is armed with strong slightly recurved spines, and both the stems and the
petioles of the leaves are covered with a very dense crop of short stiff
brown hairs scarcely rising above the skin. This is one of the handsomest
and best kinds we have.
 Sonchus laciniatus.—A very graceful composite plant, from Madeira,
with a stout stem, growing to a height of more than 5 ft., and large deeply-
cut leaves with linear-lance-shaped segments. Flower-heads yellow. When
grouped on grass-plats, or open spaces in pleasure-grounds, the fine foliage
of this plant is seen to very great advantage; but being so slender and
delicate the plants must be placed where they may be seen. It should be
planted out at the end of May, and thrives best in rich, substantial soil, in a
warm sunny position. Very numerous varieties, with the leaves variously
divided and of various shades of green, have been advertised in catalogues
under specific names, as S. lyratus, S. gummiferus, etc., etc. Many of these
are quite as charming as the type, and are well adapted for the same uses.
*Sorghum halepense.—A handsome hardy grass from S. Europe, N.
Africa, and Syria, with an erect stem about 3½ ft. high, and broad flat
leaves more than 1 ft. long, chiefly collected round the base of the plant. It
is most attractive when in flower in the end of summer, the inflorescence
consisting of a dense panicle of purplish awned flowers. Suitable for
isolation, groups, or borders.
Sparmannia africana.—A beautiful flowering stove-shrub from 3 ft. to
12 ft. high, very much resembling a Malva in habit, with long-stalked,
heart-shaped, lobed leaves, clothed with soft down, and numerous pretty
white flowers produced in stalked umbels. It thrives freely in the open air in
the south of England, from the end of May to October, if planted in rich
light soil and in warm positions. Cape of Good Hope.
*Spiræa Aruncus.—This is a remarkably handsome and effective plant,
from 3½ ft. to 5 ft. high, with elegantly-divided leaves, which bear some
resemblance to the fronds of certain ferns. The flowers are white, and are
disposed above the foliage in graceful, airy plumes. A cool, peaty soil, and a
slightly-shaded position, are best suited for this plant, and it may be placed
with advantage on slopes with a north aspect, the banks of streams or pieces
of water, in glades, and thinly-planted shrubberies, etc. Division. Siberia.
*Spiræa Filipendula.—A hardy, native perennial, with elegant foliage
and handsome flowers. The leaves are mostly radical, very finely cut, and
form a loosely-spreading rosette. The flower-stems rise to a height of 1½ ft.
to 2 ft., and are terminated by dense panicles of rosy-white flowers. There is
a fine variety with double flowers. This plant is included here only in
consequence of the resemblance of its leaves to a pinnate-leaved fern. By
pinching off the flowers it may be used with good effect as a green, fern-
like edging plant, and it is pretty in borders. Division in winter or spring.
*Spiræa (Hoteia) japonica.—A handsome, herbaceous perennial,
forming rich tufts of dark shining green much-divided leaves, which have a
somewhat fern-like appearance. These tufts are usually from a foot to 16
ins. high. The flowers are very freely produced in graceful panicles, of
which the bracts, little flower-stems, and all the ramifications are, like the
flowers, white. It is particularly fond of a sandy peat, or very sandy loam, a
sheltered position, and moist soil. Multiplied by division of the tufts in
spring or the end of summer. Japan.
*Spiræa Lindleyana.—A graceful shrub, with erect stems, from 6½ ft.
to nearly 10 ft. high, and large compound leaves, with finely-toothed
leaflets. Flowers late in summer, white, in very large and handsome
terminal panicles. This well-known plant is second to none for its grace and
distinctness, both of foliage and flower. It is a native of the Himalayas, and
easily procured in our nurseries; it should receive far more attention than
the majority of our shrubs do, and should be employed both in a young and
fully-grown state in and near the flower-garden. Few things, tender or
hardy, known in our gardens, afford a better effect than may be obtained
from this.
It is probably one of those plants which would look exceedingly
effective if trained to a single stem and cut down every year, as
recommended for the Ailantus and the Paulownia; but I have had no
experience of it in this way, and its natural habit is sufficiently graceful.
Stadmannia Jonghei.—A tall and stately foliage-plant from Australia,
where it attains the dimensions of a small tree, with dark shining green
pinnate leaves; the divisions oblong-pointed, with serrated margins, and of a
paler colour underneath. Bears the open air of the southern counties in
summer well, if placed in sunny and sheltered spots.
*Statice latifolia.—A hardy and very ornamental herbaceous perennial
from Russia, with broad leaves, which form a rosette or tuft more or less
spreading. The flower-stem is more than 2 ft. high, and very much
branched; the branches commencing at from 4 ins. to 8 ins. above the
ground, and forming a large and exceedingly handsome panicle of flowers
of a light-blue colour, tinged with the greyish hue of the numerous
membranous bracts and thin dry calyces. A well-drained, sandy soil, in an
open sunny position, is the best for this plant, which, however, grows in any
ordinary garden-soil, and is admirably adapted for naturalisation or
grouping with the acanthuses, tritomas, etc., the effect of the inflorescence
being very remarkable.
*Stipa pennata (Feather-grass).—This plant, which at other times is
hardly to be distinguished from a strong, stiff tuft of common grass,
presents, in May and June, a very different appearance, the tuft being then
surmounted by numerous flower-stems, nearly 2 ft. high, gracefully
arching, and densely covered, for a considerable part of their upper
extremity, with long, twisted, feathery awns. It loves a deep, sandy loam,
and may be used with fair effect in groups of small plants, or isolated; but
its flowers continue too short a time in bloom to make it very valuable away
from borders.
*Struthiopteris germanica.—One of the most elegant hardy ferns, with
fronds resembling ostrich-plumes in shape, nearly 3 ft. long, and arranged
in a somewhat erect, vase-like rosette. It is particularly suited for the
embellishment of the slopes of pleasure-grounds, cascades, grottoes, and
rough rockwork, the margins of streams and pieces of water, and will thrive
in moist and deep sandy soil, either in the full sunshine or in the shade. S.
pennsylvanica very closely resembles S. germanica, the chief point of
difference being the narrowness of the fertile fronds of the former species.
Both kinds will prove very effective in adding beauty of form to a garden,
and should by no means be confined to the fernery proper. Central Europe.
*Tamarix.—These very elegant hardy shrubs may be used with
excellent effect in the flower-garden and pleasure-ground, though they are
at present seldom employed in these places. T. gallica or anglica is found
apparently wild in several parts of the south of England, and other kinds,
such as germanica, parviflora, tetrandra, spectabilis, and indica, are also in
cultivation. In the neighbourhood of Paris T. indica thrives very freely, and
forms beautiful hedges, but is cut down by frost during some winters. It
would probably do better in the south of England. The plants have minute
leaves and very elegantly-panicled branches, which gives them a feathery
effect, somewhat like that of the most graceful conifers, and, if possible,
more elegant: the roseate panicles of small flowers are also very pretty. A
finer effect would be obtained from these shrubs by isolating them on the
grass than in any other way.
 *Tanacetum vulgare var. crispum.—A very elegant variety of the
common tansy, much dwarfer in stature, and with smaller emerald-green
leaves, which are very elegantly cut, and have a crisped or frizzled
appearance. It is quite hardy, and forms an effective ornament on the
margins of shrubberies, near rockwork, etc. It does best fully exposed, and
probably the only way in which it can be benefited after planting—in deep
and rather moist soil it does best, but will grow “anywhere”—is by thinning
out the shoots in spring, so that each remaining one shall have free room to
suspend its exquisite leaves; thinned thus, it looks much better than when
the stems are crowded, and of course, if it is done in time, they individually
attain more strength and dignity. The flowers should be pinched off before
they open. Britain.
Thalia dealbata.—This is one of the finest aquatic plants which we can
employ in the embellishment of pieces of water, streams, etc. In a warm and
sheltered position, and on a substantial and rich bottom, it grows
vigorously, sometimes attaining a height of 6 ft. The best mode of growing
it is in pots or tubs pierced with holes, in a mixture of stiff peat and clayey
soil, with a portion of river-mud and sand. In winter these pots or tubs may
be submerged to a greater depth, and the plants be thus effectually
protected. It would not attain the above size out of doors except in warm
places in the southern counties, in which it might be planted out directly
without taking the precautions above described. It is generally grown in the
stove in this country. N. America.
*Thalictrum minus.—One of the most elegant-leaved of our native
plants, forming compact, roundish bushes, from a foot to 18 ins. high, very
symmetrical, and of a slightly glaucous hue. It may be grown in any soil,
and requires only one little attention, namely, to pinch off the slender
flower-stems that appear in May and June. Not alone in its aspect, as a little
bushy tuft, does it resemble the “Maidenhair Fern,” as Adiantum cuneatum
is often called, but the leaves are almost pretty enough to pass, when
mingled with flowers, for those of the fern; they are also stiffer and more
lasting than fern-leaves, and are well suited for mingling with vases of
flowers, etc. There are probably several “forms” or varieties of this plant. It
would look very pretty isolated in large tufts as an edging, or in borders, or
in groups of dwarf subjects. Easily increased by division.
*The Tritomas.—So hardy, so magnificent in colouring, and so fine in
form are these plants, that we can no more dispense with their use in the
garden where beauty of form as well as colour is to prevail, than we can
with the noble Pampas grass. They are more conspicuously beautiful, when
other things begin to succumb before the gusts and heavy rains of autumn,
than any plants which flower in the bright days of midsummer. It is not
alone as component parts of large ribbon-borders and in such positions that
these grand plants are useful, but in almost any part of the garden.
Springing up as a bold, close group on the green turf, and away from
brilliant surroundings, they are more effective than when associated with
bedding plants; and of course many such spots may be found for them near
the margins of the shrubberies in most pleasure-grounds. It is in an isolated
group, flaming up amid the verdure of trees and shrubs and grass, that their
dignified aspect and brilliant colour are seen to best advantage. However,
tastefully disposed in the flower-garden, they will prove generally useful,
and particularly for association with the finer autumn-flowering herbaceous
plants. A most satisfactory result may be produced by associating the
Tritomas with the Pampas grass and the two Arundos, the large Statice
latifolia, and the strong and beautiful autumn-flowering Anemone japonica
alba, which is peculiarly suited for association with hardy herbaceous
plants of fine habit, and should be in every garden where a hardy flower is
valued.
The Tritomas are not fastidious as to soil, and with a little preparation of
the ground may be grown almost anywhere. They thrive with extraordinary
vigour and freedom where the soil is very sandy as well as rich and deep,
and are readily multiplied by division.
As every garden should be embellished by well-developed specimens or
groups of these fine plants, those who have very poor and thin, or pure clay
soils, would do well to excavate the ground to the depth of 2 ft. or 3 ft., and
fill in with good rich loam. When the soil is deep, no watering will be
required.
*Tritoma Burchelli.—This kind is distinguished by the lighter green of
its leaves, by its black-spotted flower-stem, and especially by the colour of
its flowers, which are crimson at the base, passing into carmine in the
middle, and pale-yellow or greenish at the tips. There is a variety which has
the leaves variegated or striped with white, but it is somewhat tender and
rare.
 *Tritoma glauca.—A dwarfer kind than T. Uvaria, with leaves of a sea-
green colour, and very large spikes of scarlet-and-yellow flowers, which,
when in bud, are hidden by long, sea-green bracts, streaked and rayed with
white. There is a scarce variety with recurved leaves (T. g. recurvata),
which has somewhat of the habit of a Bromelia. S. Africa.
*Tritoma præcox.—A recently-introduced, handsome, hardy perennial,
with very much the habit of T. Uvaria. The flower-stem grows from 20 ins.
to 2 ft. high, and the flowers, which are produced about the middle of May,
are of a bright-red colour when exposed to the full sun, and of a bright-
yellow when grown in the shade. The leaves are fully 2 ft. long, sharply
keeled, and with toothed edges. S. Africa.
*Tritoma Uvaria.—A very ornamental and well-known kind from S.
Africa, forming thick tufts of linear, erect leaves. It is a vigorous grower,
and small specimens have been known in three years to form tufts from 3 ft.
to 4 ft. through, bearing from 50 to 100 flower-spikes. The flowering-stems
are about 3¼ ft. in height, and the flowers are borne in dense conical
clusters at the top. The upper part of the cluster, containing the young
flowers, is of a coral-red colour, the lower part yellow, all the flowers
gradually changing to this colour. Other varieties in cultivation are—T. U.
grandis or grandiflora, which is much taller than the preceding kind, with
stouter stems and larger flower-spikes; T. U. Rooperi, which only differs
from the type in being somewhat dwarfer in habit and having softish or
flaccid leaves, frequently falling forward; it also flowers later; and T. U.
Lindleyana, which has erect, very rigid leaves, and more deeply-coloured
flowers than the type.
Tupidanthus calyptratus.—A noble subtropical plant from Bengal,
standing in the open air from the beginning of June till October without the
slightest injury. The leaves are large, deeply-divided, and of a dark shining
green colour. It requires stove treatment in winter and spring, and is suitable
for beds or planting singly.
*Typha latifolia (Reed-Mace).—A native aquatic plant, growing in tufts
of 2-rowed flat leaves from 1½ ft. to 2 ft. long, and 1 in. or 1½ in. wide.
From the centre of each tuft springs a stem 6 ft. or 7 ft. high, which in the
flowering season is terminated by a close cylindrical spike 9 ins. long, and
of a dark-olive colour, changing to a brownish-black as it ripens. This is one
of the most striking and ornamental of our British water-plants, and may be
used with excellent effect grouped with such subjects as the Great Water-
Dock.
*Typha angustifolia resembles the preceding species in all respects
except in the size of its leaves and spike. The leaves are about ½ in. wide
and the spike about ½ in. in diameter, and something shorter than that of T.
latifolia. Of the two it is perhaps the more graceful in aspect.
Uhdea bipinnatifida.—This is one of the most useful plants in its class,
producing a rich mass of handsome leaves, with somewhat the aspect of
those of the great cow-parsnips, but of a more refined type. The foliage has
a slightly silvery tone, and the plant continues to grow fresh and vigorously
till late in autumn. It is well suited for forming rich masses of foliage, not so
tall, however, as those formed by such things as Ricinus or Ferdinanda. It is
freely propagated by cuttings taken from old plants kept in a cool stove,
greenhouse, or pit during the winter months, and placed in heat to afford
cuttings freely in early spring. Under ordinary cutting treatment on hotbeds
or in a moist warm propagating house, it grows as freely as could be
desired, and may be planted out at the end of May or the beginning of June.
Mexico.

Uhdea bipinnatifida.
 Uhdea pyramidata.—This kind has been less cultivated in England
than the preceding, from which it is distinct in appearance. It is of a lighter
and fresher green, and inclined to grow larger in habit, having more of the
aspect of a Malva in foliage. Useful for the same purposes as the preceding
kind, but not so valuable.
*Veratrum album (White Hellebore).—A handsome, erect perennial of
pyramidal habit, 3½ ft. to 5 ft. high, with curiously plaited leaves 1 ft. long
and 6 ins. to 8 ins. broad, regularly alternating on the stem and overlapping
each other at the base. The flowers, of a yellowish-white colour, are borne
in numerous dense spikes on the top of the stem, forming a large panicle.
The leaves being handsome, it is worth a place in full collections of fine-
foliaged hardy herbaceous plants, and would look to best advantage in
small groups in the rougher parts of the pleasure-ground and by wood-
walks. Thrives best in peaty soil, and is best multiplied by division, as the
seed is very slow and capricious in germinating, sometimes not starting
until the second year, and it is some years before the seedlings are strong
enough to flower. The root of this plant is exceedingly poisonous. V. nigrum
differs from V. album, in having more slender stems, narrower leaves, and
blackish-purple flowers. V. viridiflorum resembles V. album in every
respect, except that its flowers are of a lively green colour. France.
*Verbascum Chaixii.—Most of us know how very distinct and
imposing are the larger Verbascums, and those who have attempted their
culture must soon have found out what far-seeding things they are. Of a
biennial character, their culture is most unsatisfactory: they either migrate
into the adjoining shrubbery or disappear altogether. The possession of a
fine perennial species must therefore be a desideratum, and such a plant will
be found in Verbascum Chaixii. This is fine in leaf and stature, and
produces abundance of flowers. The lower leaves grow 18 ins. or 20 ins.
long, and the plant when in flower reaches a height of 7 ft. or 8 ft., or even
more when in good soil. It is a truly distinct subject, and may, it is to be
hoped, ere long be found common in our gardens and nurseries. Like the
preceding, but grown under the name V. vernale, is a kind I saw in the
Jardin des Plantes at Paris, and introduced into cultivation in England; but it
is as yet scarce.
Verbesina gigantea.—An ornamental shrub from Jamaica, about 6½ ft.
high, forming, when young, a very pleasing subject for decorative purposes,
its round green stems being covered with large, winged, pinnate leaves of a
Welcome to our website – the perfect destination for book lovers and
knowledge seekers. We believe that every book holds a new world,
offering opportunities for learning, discovery, and personal growth.
That’s why we are dedicated to bringing you a diverse collection of
books, ranging from classic literature and specialized publications to
self-development guides and children's books.

More than just a book-buying platform, we strive to be a bridge

connecting you with timeless cultural and intellectual values. With an
elegant, user-friendly interface and a smart search system, you can
quickly find the books that best suit your interests. Additionally,
our special promotions and home delivery services help you save time
and fully enjoy the joy of reading.

Join us on a journey of knowledge exploration, passion nurturing, and

personal growth every day!

ebookbell.com