SimonBlanke/Gradient-Free-Optimizers — Gitpedia

<h3 align="center"> Lightweight optimization with local, global, population-based and sequential techniques across mixed search spaces </h3> <a href="https://github.com/SimonBlanke/Gradient-Free-Optimizers/actions"><img src="https://img.shields.io/github/actions/workflow/status/SimonBlanke/Gradient-Free-Optimizers/ci.yml?style=for-the-badge&logo=githubactions&logoColor=white&label=tests" alt="Tests"></a> <a href="https://app.codecov.io/gh/SimonBlanke/Gradient-Free-Optimizers"><img src="https://img.shields.io/codecov/c/github/SimonBlanke/Gradient-Free-Optimizers?style=for-the-badge&logo=codecov&logoColor=white" alt="Coverage"></a> <table align="center"> <tr> <td align="right">Documentation</td> <td align="center">▸</td> <td> <a href="https://gradient-free-optimizers.readthedocs.io/en/latest/">Homepage</a> · <a href="https://gradient-free-optimizers.readthedocs.io/en/latest/user_guide/optimizers/index.html">Optimizers</a> · <a href="https://gradient-free-optimizers.readthedocs.io/en/latest/api_reference.html">API Reference</a> · <a href="https://gradient-free-optimizers.readthedocs.io/en/latest/examples.html">Examples</a> </td> </tr> <tr> <td align="right">On this page</td> <td align="center">▸</td> <td> <a href="#key-features">Features</a> · <a href="#examples">Examples</a> · <a href="#core-concepts">Concepts</a> · <a href="#citation">Citation</a> </td> </tr> </table>

Gradient-Free-Optimizers is a Python library for gradient-free optimization of black-box functions. It provides a unified interface to 23 optimization algorithms, from simple hill climbing to Bayesian optimization, all operating on mixed search spaces that combine continuous ranges, discrete grids, categorical choices, and SciPy distribution-backed dimensions.

Designed for hyperparameter tuning, simulation optimization, feature selection, engineering design, and any scenario where gradients are unavailable or impractical. The library prioritizes simplicity: define your objective function, specify the search space, and run. All algorithms share one consistent API, so switching from hill climbing to Bayesian optimization is a one-line change. SciPy is optional; GFO only requires NumPy and pandas, making it suitable as an optimization backend or for minimal environments, containers, and embedded systems.

Installation

bash
pip install gradient-free-optimizers

<a href="https://pypi.org/project/gradient-free-optimizers/"><img src="https://img.shields.io/pypi/v/gradient-free-optimizers?style=flat-square&color=blue" alt="PyPI"></a> <a href="https://pypi.org/project/gradient-free-optimizers/"><img src="https://img.shields.io/pypi/pyversions/gradient-free-optimizers?style=flat-square" alt="Python"></a> <a href="https://pepy.tech/project/gradient-free-optimizers"><img src="https://img.shields.io/pepy/dt/gradient-free-optimizers?style=flat-square&color=green" alt="Total Downloads"></a> <details> <summary>Optional dependencies</summary>

bash
pip install gradient-free-optimizers[progress]  # Progress bar with tqdm
pip install gradient-free-optimizers[sklearn]   # scikit-learn for surrogate models
pip install gradient-free-optimizers[full]      # All optional dependencies

</details>

Key Features

<table> <tr> <td width="33%"> <a href="https://gradient-free-optimizers.readthedocs.io/en/latest/user_guide/optimizers/index.html">23 Optimization Algorithms</a> Local, global, population-based, and sequential model-based optimizers. Switch algorithms with one line of code. </td> <td width="33%"> <a href="https://gradient-free-optimizers.readthedocs.io/en/latest/api_reference.html">Zero Configuration</a> Sensible defaults for all parameters. Start optimizing immediately without tuning the optimizer itself. </td> <td width="33%"> <a href="https://gradient-free-optimizers.readthedocs.io/en/latest/user_guide/memory.html">Memory System</a> Built-in caching prevents redundant evaluations. Critical for expensive objective functions like ML models. </td> </tr> <tr> <td width="33%"> <a href="https://gradient-free-optimizers.readthedocs.io/en/latest/user_guide/search_spaces.html">Mixed Search Spaces</a> Combine continuous ranges, discrete grids, categorical choices, and SciPy distributions in a single search space. </td> <td width="33%"> <a href="https://gradient-free-optimizers.readthedocs.io/en/latest/user_guide/constraints.html">Constraints Support</a> Define constraint functions to restrict the search space. Invalid regions are automatically avoided. </td> <td width="33%"> <a href="https://github.com/SimonBlanke/Gradient-Free-Optimizers">Minimal Dependencies</a> Only NumPy and pandas required. Optional integrations for progress bars (tqdm), SciPy distributions, and surrogate models (scikit-learn). </td> </tr> </table>

Quick Start

python
import numpy as np
from gradient_free_optimizers import HillClimbingOptimizer

def objective(params):
    x, y = params["x"], params["y"]
    return -(x**2 + y**2)

search_space = {
    "x": (-5.0, 5.0),            # continuous range
    "y": np.arange(-5, 5, 0.1),  # discrete grid
}

opt = HillClimbingOptimizer(search_space)
opt.search(objective, n_iter=1000)

print(f"Best score: {opt.best_score}")
print(f"Best params: {opt.best_para}")

Core Concepts

mermaid
flowchart LR
    O["Optimizer
    ━━━━━━━━━━
    23 algorithms"]

    S["Search Space
    ━━━━━━━━━━━━
    mixed dimensions"]

    F["Objective
    ━━━━━━━━━━
    f(params) → score"]

    D[("Search Data
    ━━━━━━━━━━━
    history")]

    O -->|propose| S
    S -->|params| F
    F -->|score| O

    O -.-> D
    D -.->|warm start| O

Optimizer: Implements the search strategy. Choose from 23 algorithms across four categories: local search, global search, population-based, and sequential model-based.

Search Space: Defines valid parameter ranges and choices. Each key is a parameter name, each value is a tuple (min, max) for continuous, a NumPy array for discrete, a list for categorical, or a SciPy distribution.

Objective Function: Your function to maximize. Takes a dictionary of parameters, returns a score. Use negation to minimize.

Search Data: Complete history of all evaluations accessible via opt.search_data for analysis and warm-starting future searches.

Examples

<details open> <summary>Hyperparameter Optimization</summary>

python
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.model_selection import cross_val_score
from sklearn.datasets import load_wine
import numpy as np

from gradient_free_optimizers import BayesianOptimizer

X, y = load_wine(return_X_y=True)

def objective(params):
    model = GradientBoostingClassifier(
        n_estimators=params["n_estimators"],
        max_depth=params["max_depth"],
        learning_rate=params["learning_rate"],
    )
    return cross_val_score(model, X, y, cv=5).mean()

search_space = {
    "n_estimators": np.arange(50, 300, 10),
    "max_depth": np.arange(2, 10),
    "learning_rate": np.logspace(-3, 0, 20),
}

opt = BayesianOptimizer(search_space)
opt.search(objective, n_iter=50)

</details> <details> <summary>Bayesian Optimization</summary>

python
import numpy as np
from gradient_free_optimizers import BayesianOptimizer

def ackley(params):
    x, y = params["x"], params["y"]
    return -(
        -20 * np.exp(-0.2 * np.sqrt(0.5 * (x**2 + y**2)))
        - np.exp(0.5 * (np.cos(2 * np.pi * x) + np.cos(2 * np.pi * y)))
        + np.e + 20
    )

search_space = {
    "x": np.arange(-5, 5, 0.01),
    "y": np.arange(-5, 5, 0.01),
}

opt = BayesianOptimizer(search_space)
opt.search(ackley, n_iter=100)

</details> <details> <summary>Particle Swarm Optimization</summary>

python
import numpy as np
from gradient_free_optimizers import ParticleSwarmOptimizer

def rastrigin(params):
    A = 10
    values = [params[f"x{i}"] for i in range(5)]
    return -sum(v**2 - A * np.cos(2 * np.pi * v) + A for v in values)

search_space = {f"x{i}": np.arange(-5.12, 5.12, 0.1) for i in range(5)}

opt = ParticleSwarmOptimizer(search_space, population=20)
opt.search(rastrigin, n_iter=500)

</details> <details> <summary>Simulated Annealing</summary>

python
import numpy as np
from gradient_free_optimizers import SimulatedAnnealingOptimizer

def sphere(params):
    return -(params["x"]**2 + params["y"]**2)

search_space = {
    "x": np.arange(-10, 10, 0.1),
    "y": np.arange(-10, 10, 0.1),
}

opt = SimulatedAnnealingOptimizer(
    search_space,
    start_temp=1.2,
    annealing_rate=0.99,
)
opt.search(sphere, n_iter=1000)

</details> <details> <summary>Constrained Optimization</summary>

python
import numpy as np
from gradient_free_optimizers import RandomSearchOptimizer

def objective(params):
    return params["x"] + params["y"]

def constraint(params):
    # Only positions where x + y < 5 are valid
    return params["x"] + params["y"] < 5

search_space = {
    "x": np.arange(0, 10, 0.1),
    "y": np.arange(0, 10, 0.1),
}

opt = RandomSearchOptimizer(search_space, constraints=[constraint])
opt.search(objective, n_iter=1000)

</details> <details> <summary>Mixed Search Space</summary>

python
import numpy as np
from scipy import stats
from gradient_free_optimizers import BayesianOptimizer

def objective(params):
    x = params["x"]
    n_layers = params["n_layers"]
    lr = params["learning_rate"]
    activation_scores = {"relu": 0.0, "tanh": 0.1, "gelu": 0.3}
    return -(x**2) - 0.1 * n_layers + activation_scores[params["activation"]] - abs(lr - 0.001)

search_space = {
    "x": (-5.0, 5.0),                          # continuous
    "n_layers": np.arange(1, 6),                # discrete
    "activation": ["relu", "tanh", "gelu"],     # categorical
    "learning_rate": stats.loguniform(1e-5, 1),  # distribution
}

opt = BayesianOptimizer(search_space)
opt.search(objective, n_iter=100)

</details> <details> <summary>Memory and Warm Starting</summary>

python
import numpy as np
from gradient_free_optimizers import HillClimbingOptimizer

def expensive_function(params):
    # Simulating an expensive computation
    return -(params["x"]**2 + params["y"]**2)

search_space = {
    "x": np.arange(-10, 10, 0.1),
    "y": np.arange(-10, 10, 0.1),
}

# First search
opt1 = HillClimbingOptimizer(search_space)
opt1.search(expensive_function, n_iter=100, memory=True)

# Continue with warm start using previous search data
opt2 = HillClimbingOptimizer(search_space)
opt2.search(expensive_function, n_iter=100, memory_warm_start=opt1.search_data)

</details> <details> <summary>Ask/Tell Interface</summary>

python
import numpy as np
from gradient_free_optimizers import BayesianOptimizer

def objective(params):
    return -(params["x"]**2 + params["y"]**2)

search_space = {
    "x": np.arange(-10, 10, 0.1),
    "y": np.arange(-10, 10, 0.1),
}

# Manual control over the optimization loop
opt = BayesianOptimizer(search_space)
opt.setup_search(objective, n_iter=100)

for _ in range(100):
    params = opt.ask()           # Get next parameters to evaluate
    score = objective(params)
    opt.tell(params, score)      # Report result back

</details> <details> <summary>Early Stopping</summary>

python
import numpy as np
from gradient_free_optimizers import BayesianOptimizer

def objective(params):
    return -(params["x"]**2 + params["y"]**2)

search_space = {
    "x": np.arange(-10, 10, 0.1),
    "y": np.arange(-10, 10, 0.1),
}

opt = BayesianOptimizer(search_space)
opt.search(
    objective,
    n_iter=1000,
    max_time=60,           # Stop after 60 seconds
    max_score=-0.01,       # Stop when score reaches -0.01
    early_stopping={       # Stop if no improvement for 50 iterations
        "n_iter_no_change": 50,
    },
)

</details>

Ecosystem

GFO is used as the optimization engine in other packages and integrates with the broader Python optimization ecosystem. For updates, follow on GitHub.

Package	Description
Hyperactive	High-level hyperparameter optimization framework, uses GFO as its optimization backend
Surfaces	Test functions and benchmark surfaces for optimization algorithm evaluation

Documentation

Resource	Description
User Guide	Comprehensive tutorials and explanations
API Reference	Complete API documentation
Optimizers	Detailed description of all 23 algorithms
Examples	Code examples for various use cases

Contributing

Contributions welcome! See CONTRIBUTING.md for guidelines.

Bug reports: GitHub Issues
Feature requests: GitHub Discussions
Questions: GitHub Issues

Citation

If you use this software in your research, please cite:

bibtex
@software{gradient_free_optimizers,
  author = {Simon Blanke},
  title = {Gradient-Free-Optimizers: Simple and reliable optimization with local, global, population-based and sequential techniques in mixed search spaces},
  year = {2020},
  url = {https://github.com/SimonBlanke/Gradient-Free-Optimizers},
}