AI terms: Machine Learning (ML), Reinforcement Learning (RL) and Deep Learning (DL)
An article by ScienceInsight
Article AI130
AI vs. ML
Artificial Intelligence (AI) is the overarching term that deals with machines capable of exhibiting human-like intelligence. Machine Learning, on the other hand, is not. The goal of machine learning is to teach a machine to perform a specific task and deliver precise results by identifying patterns. Machine Learning (ML) is a specific subfield of AI that focuses on developing algorithms and models that can learn from data and recognize patterns. Thus, AI is a much broader field than ML. ML is one specific method for achieving AI.
Difference Learning:
AI can attain the desired intelligence using various strategies and technologies, while ML is explicitly based on learning from data.
Difference Applications:
AI applications can include expert systems, process automation, and deep learning, while ML is employed in many AI applications such as image recognition and speech recognition. An AI system could be, for example, an autonomous vehicle, whereas an ML system is an algorithm that learns from image data to recognize objects.
Generative AI
Generative AI is a form of artificial intelligence that is trained to create new content such as text, images, music, audio, and videos. It operates through machine learning on large datasets and, based on algorithms and pre-trained foundational models, independently produces original results.
How generative AI works
Training data:
Generative AI models are trained on vast amounts of data to learn patterns and structures within it
Foundational models:
These models serve as base frameworks, are multitask-capable, and can be fine-tuned for specific use cases
Machine learning:
By leveraging machine learning algorithms and techniques, the AI can generate new content that mimics or extends existing patterns
User interaction:
Users input a “prompt,” and the AI then generates a response, such as text, an image, or a song, based on its learned knowledge
Examples of Applications
Content creation:
Generating texts, emails, reports, and presentations
Creative fields:
Producing images (AI art), music, and videos
Development:
Assisting in programming, creating 3D models and prototypes
Healthcare:
Analyzing medical images to aid diagnosis and drug development
Service:
Providing chatbots for customer support and auto mated responses
ML vs. RL vs. DL
Machine Learning (ML), Reinforcement Learning (RL), and Deep Learning (DL) are not the same. Reinforcement Learning and Deep Learning are specific types of Machine Learning. Machine Learning is a broader field that encompasses various learning methods, including Reinforcement Learning, Deep Learning, Supervised Learning, and Unsupervised Learning.
Machine Learning (ML)
Is a branch of artificial intelligence concerned with developing algorithms that can learn from data without being explicitly programmed.
Reinforcement Learning (RL)
RL is a specific type of Machine Learning where an agent learns to make decisions in a dynamic environment to achieve maximum utility or reward. Learning occurs through interaction with the environment, with the agent receiving feedback in the form of rewards or penalties. Need for simulations:
Training Reinforcement Learning models often requires simulations of the environment to replicate interactions and rewards.
Deep Learning (DL)
Deep Learning is a specific type of Machine Learning based on artificial neural networks with many layers (deep neural networks) and is particularly well-suited for tasks involving large datasets and complex patterns (a method for pattern recognition and prediction). While Machine Learning includes a broader range of algorithms, Deep Learning primarily uses neural networks to detect complex patterns in large datasets. Deep Learning can be utilized as a tool within Reinforcement Learning, for example, to improve state recognition or action selection.
Learning paradigm: Deep Learning is often a form of supervised learning, where models are trained on labeled data, but it can also be employed in unsupervised learning.
The main distinguishing feature between Reinforcement Learning and Deep Learning is that in Deep Learning, the learning process is based on a training dataset, which is then applied to a new dataset. Reinforcement Learning, on the other hand, learns dynamically by adjusting actions based on continuous feedback to maximize rewards. Reinforcement Learning represents a learning paradigm in which an agent learns to make optimal decisions through interaction with its environment.
KI fields
Machine Learning (ML)
Reinforcement Learning (RL)
Deep Learning (DL)
Definiton
ML is a method for creating AI systems that can learn from data and recognize patterns. It encompasses a broad range of different learning methods.
A specific learning method aimed at making decisions in dynamic environments to maximize utility.
A specific learning method that focuses on deep neural networks to identify and simulate complex patterns in data.
Learning process
The learning process can be carried out through various methods, such as Supervised Learning and Unsupervised Learning.
Learns through interaction with the environment (trial-and-error interaction) and receives feedback in the form of rewards/penalties. An agent performs actions and receives a reward or penalty based on the environment's response. It learns an optimal strategy through interaction with its environment to accomplish a specific task.
Learns from large amounts of data by using deep neural networks to identify structures and relationships within the data. Builds computational models composed of multiple layers, enabling the creation of abstraction levels from the data. Encompasses algorithms that learn from data to make predictions or decisions.
Algorithmen
Utilizes a variety of algorithms, such as decision trees, support vector machines, or linear models.
Commonly used algorithms include Q-learning, SARSA, policy gradient methods, and Deep Q-Networks (DQN). The choice of the appropriate algorithm depends on the specific application and the characteristics of the problem, such as the type of environment, the complexity of states and actions, and the available resources.
Relies on artificial neural networks with many layers (deep neural networks).
Datenmenge und Rechenleistung
Machine Learning can be more efficient with smaller datasets.
RL is typically resource-intensive, especially when deep learning is employed or real-world systems are being controlled. The required amounts of data and computational power are often orders of magnitude higher than in classic supervised learning.
Deep learning models often require very large amounts of data to train effectively and need a powerful computational infrastructure.
Feature Engineering*
With ML models, manual feature engineering is often required, meaning the creation of relevant input features for the model.
Enables efficient learning. Construction of a meaningful and compact state representation (s) from observations.
DL models can often extract features autonomously from the raw data.
Datentypen
Can use various data types, e.g., structured data.
Learns from collected experiences in a dynamic environment.
Can handle both structured and unstructured data, such as images, text, and speech.
Komplexität
Less complex than RL or DL
RL models are generally more complex and can also recognize more intricate patterns and structures in data that are difficult for standard machine learning models to capture.
DL models are typically more complex and can recognize even more intricate patterns and structures in data that are difficult for standard machine learning models to capture.
Anwendungen
Various fields of application, e.g., image recognition, speech processing
Robotics, game development (e.g., chess, Go), autonomous vehicles, process optimization
Deep Learning is often used for tasks that require high accuracy and the processing of large amounts of data, such as image recognition, speech recognition, pattern recognition, natural language processing, and autonomous driving.
*) What is meant by 'Feature Engineering', also referred to as "data preprocessing". It describes the processes involved in transforming raw data into features that can be used for developing machine learning models. In other words: The transformation and enrichment of data with the goal of improving the performance of machine learning algorithms used to train models with this data.
Terms in Reinforcement Learning
| Term | Meaning / Usage | Origin / Library |
|---|---|---|
obs |
Stands for observation – the current state of the environment as seen by the agent | gym / gymnasium (OpenAI) |
.step() |
Environment function: executes an action and returns obs, reward, done, info |
gym / gymnasium |
env |
Stands for environment – the RL environment in which the agent operates | gym, also used by stable-baselines3 |
reward |
Feedback after an action indicating how good or bad it was | gym, but used by all RL frameworks |
done |
Indicates whether an episode has ended | gym / gymnasium |
| OpenAI Gym | Library for defining RL environments | Developed by OpenAI |
| stable-baselines3 | High-level RL framework for easy use of algorithms such as PPO, DQN, etc. | Community project, based on gym |
model.learn() |
Training method for the RL model | stable-baselines3 |
.predict() |
Makes predictions based on the trained model object | stable-baselines3 |
OpenAI Gym
OpenAI originally developed Gym. OpenAI Gym is a free Python toolkit and API that provides a collection of simulated environments for training and testing reinforcement learning (RL) agents. It serves as an industry standard for developing, comparing, and evaluating RL algorithms by offering a unified interface for communication between learning algorithms and various environments.
Stable_baselines3
Stable-baselines3 is a framework for training agents. It consists of a set of improved implementations of reinforcement learning (RL) algorithms based on Farama Foundation (former OpenAI Baselines). Stable-baselines3 is not directly from former OpenAI, but rather an open-source project built upon it.
Best-known and practically relevant RL frameworks
| Framework | Type | Language | Short Description & Special Features |
|---|---|---|---|
| Gym / Gymnasium | Low-Level Env | Python | OpenAI's standard API for RL environments. No algorithms included! |
| Stable-Baselines3 | High-Level | Python | Easy to use, many algorithms (PPO, DQN, A2C, etc.), actively maintained |
| RLlib (Ray) | High-Level | Python | Scalable, distributed RL, ideal for production environments |
| CleanRL | Medium-Level | Python | Minimalistic implementations, ideal for learning |
| Tianshou | High-Level | Python | Highly modular, supports many algorithms |
| Acme (by DeepMind) | High-Level | Python | Research-oriented but complex. Usable with JAX |
| PettingZoo | Multi-Agent | Python | Gym-like, but for multi-agent environments |
| SBX (Stable Baselines X) | High-Level (experimental) | Python | Extension of SB3 with more features, still in development |
| Keras-RL2 | Medium-Level | Python | Uses Keras/TensorFlow, but less actively maintained |
| Dopamine (by Google) | Research | Python | Lightweight, specifically for DQN variants, but not very modular |
| Coach (by Intel) | High-Level | Python | Many algorithms, but no longer actively developed |
| Unity ML-Agents | Env + Training | C# / Python | Particularly suited for 3D simulations with Unity |
| TF-Agents | High-Level | Python | TensorFlow-based, official from Google |
Current RL learning frameworks
| Framework | Active? | Backend | Special Features / Focus |
|---|---|---|---|
| Stable-Baselines3 (SB3) | active | PyTorch | Simple, modular, popular, many algorithms |
| CleanRL | active | PyTorch | Extremely minimalist, one file per algorithm |
| Tianshou | active | PyTorch | Highly modular, many algorithms, performant |
| RLlib (Ray) | active | PyTorch / TensorFlow | Distributed RL, scales to clusters |
| Acme (DeepMind) | less active | JAX | Research-oriented, structured, complex |
| SBX (Stable Baselines X) | experimental | PyTorch | Extension of SB3 with features like HER |
| Keras-RL2 | less active | TensorFlow | Keras-based, but rather outdated |
| TF-Agents (Google) | less active | TensorFlow | Complex, heavyweight, for TF users |
| Dopamine (Google) | discontinued | TensorFlow | Research focus on DQN, less modular |
| Coach (Intel) | discontinued | TensorFlow | Many algorithms, difficult to use |
| Horizon (Meta) | barely maintained | PyTorch | Meta's RL for production, high effort |
Python libraries for RL
numpy:
NumPy is a powerful Python library essential for numerical calculations and scientific computing. It provides efficient data structures such as multidimensional arrays (N-dimensional arrays, or ndarrays for short) and offers a wide variety of functions for working with these arrays. NumPy is a cornerstone for many other Python libraries in the fields of data science, machine learning, and scientific programming.
pandas:
The Pandas library is a powerful open-source Python library specifically designed for data analysis and manipulation. It provides flexible data structures and functions that facilitate working with tabular and time-series data.
seaborn:
Seaborn is a Python library used for creating statistical graphics. It is built on top of Matplotlib and is tightly integrated with Pandas data structures. Seaborn simplifies the creation of complex diagrams and enables efficient data analysis and presentation by providing a user-friendly interface for statistical visualizations.
sklearn:
Scikit-learn, also known as sklearn, is a Python library for machine learning, supporting both supervised and unsupervised learning. It also provides capabilities for analyzing models using statistical methods. It is an open-source project for machine learning. This data science toolkit optimizes artificial intelligence (AI), ML, and statistical modeling with a consistent interface.
