What’s new

Next release

• In accordance with the Upstream version policy:

  • message_ix_models supports and is tested against ixmp and message_ix_models version 3.10.0 (PR #299).

  • Support for ixmp and message_ix versions 3.4, 3.5, and 3.6 is dropped (PR #288, PR #289). The minimum supported version of both packages is 3.7.0.

• Update IEA_EWEB to support transform="B" / transform_B() (GH #230, PR #259).

• New utility sdmx.AnnotationsMixIn (PR #259).

• Drop obsolete series_of_pint_quantity() (PR #289).

By topic:

SSP 2023–2025 update/ScenarioMIP

Improve ssp.transport:

• Add prepare_method_B() and make this the default (PR #259).

• Add process_df() (PR #303).

• Adapt to revised ‘variable’ codes (PR #309, GH #304).

Transport

Update MESSAGEix-Transport (model.transport) (PR #259, PR #289).

• Adjust constraints on t="conm_ar".

• Recompute minimum_activity for transport technologies.

• Improve freight representation:

  • Adjust freight activity, freight and passenger mode shares for some regions.

  • Add dynamic constraints on activity of freight technologies.

  • Fix alignment of freight technology outputs with demand \(l\).

• Implement LED override using exogenous passenger activity data from pdt-cap.csv → P activity:scenario-n-t-y:exo.

• Drop base-scenario-url.json; store base scenario URLs in Code list CL_TRANSPORT_SCENARIO.

• Generate SDMX-ML structural metadata, including data flow definitions, and SDMX-{CSV,ML} data outputs for certain reported quantities.

• Expand use of fixed/shared keys from transport.key.

• Simplify and consolidate tests.

• Improve simulated_solution() to load ‘simulated’ solution data from file to reduce test durations.

Documentation

• New HOWTO guides documentation sub-tree (PR #291).

• New guide on HOWTO Run mix-models on UnICC (PR #279) and supporting command mix-models sbatch in util.slurm (PR #291).

• New summary pages for projects ALPS, Carbon Direct CDR, Climate Financial Risk, ECEMF, ELEVATE (JustMIP), GEIDCO, GENIE, GUIDE, HYway, NEWPATHWAYS, NextGen Carbon, NGFS, PRISMA, SPARCCLE, and UPTAKE (PR #282).

• New utility sdmx.AnnotationsMixIn (PR #259).

• Bug fix: adjust or guard some Python usage that was not compatible with Python 3.9—the earliest version supported by message_ix_models (PR #295, GH #294).

v2025.1.10

• message_ix_models is tested and compatible with Python 3.13 (PR #250).

• Support for Python 3.8 is dropped (PR #250), as it has reached end-of-life.

• Expand Reproducibility with sections on Documentation and Versioning and naming, including a list of external model names and ‘versions’ like “MESSAGEix-GLOBIOM 2.0” (GH #224, PR #226).

• New code list Emission species (emission.yaml) (PR #225).

• Designate Code owners for certain files (PR #272) and document development practices.

By topic:

Materials

• Improve and extend MESSAGEix-Materials (model.material) (PR #218, PR #253).

• Release of MESSAGEix-Materials 1.1.0 (Version 1.1.0 (August 27, 2024)).

Transport

• Update MESSAGEix-Transport (model.transport) (PR #213, PR #225).

• Rework freight, ldv, and non_ldv to use genno consistently.

• Adopt consistent terms “F RAIL” and “F ROAD” for freight service/modes.

• New technologies: “f rail {electr,lightoil}”, “f road electr”.

• Extend unit annotations to all transport commodities.

• New input files pdt-cap.csv and load-factor-ldv.csv.

• Add mix-models ssp transport CLI command to postprocess aviation emissions data.

• New SDMX Codelist IIASA:CL_TRANSPORT_SCENARIO to distinguish “Low energy demand (LED)” and EDITS (project.edits) scenarios from Shared Socioeconomic Pathways (project.ssp) baseline scenarios.

• Adjust R12 baseline settings:

  • mode-share/default.csv: adjust R12_NAM values for AIR and LDV.

  • pdt-cap-ref.csv: adjust R12_NAM value.

• Implement LED scenarios via load-factor-ldv.csv and pdt-cap.csv.

• Use y=2019 data from IEA EWEB 2024 edition to align MESSAGEix-Transport with base model calibration.

• New genno operators: broadcast(), broadcast_wildcard(), broadcast_t_c_l(), freight_usage_output().

• New utility methods get_cl_scenario(), wildcard().

• Replace DataSourceConfig.LDV setting with Config.dummy_LDV.

Water/Nexus

• Update water availability data and major code editing to allow a new test suite for doc:/water/index (PR #106).

• Fix the nexus/cooling function and add test for checking some input data (PR #236).

• Make setup of constraints for cooling technologies flexible and update solar CSP technology name (PR #242).

• Connect model.water to tools.costs for cooling technologies (PR #245).

• Introduce an SSP configuration for cooling technologies with share constraints (PR #256).

• Change SSP parameters and add missing CSP data (PR #262).

Investment and fixed costs

• Fix naming of GDP and population columns in SSP data aggregation for Investment and fixed costs (tools.costs) (PR #219).

• Edit inputs for storage, CSP, hydrogen, and industry technologies (PR #206).

• Replace solar and wind technologies with new ones (PR #206).

• Reorganize input files and incorporate first_year.csv data into tech_map.csv (PR #221).

• Reconfigure use and implementation of technology variants/modules to be more agnostic (PR #221).

• Change cost decay to reach reduction percentage specified on the year 2100 (PR #227).

• Add cooling technology variant/module (PR #222).

• Add functionality to specify cost reduction values and cost reduction scenarios in a module (GH #251, PR #255).

• Fix technology mapping of ammonia CCS technologies in materials module (PR #258).

• Remove unused technology_type column from tool (PR #269).

Others

• Add “LED”, “SSP4”, and “SSP5” as values for the --ssp=… option in common_params() (PR #233).

• Add CircEUlar (project.circeular) project code and documentation (PR #232).

• iea.web handles the 2024 edition and fixed-width file format published by the IEA directly (PR #225).

  • Code lists for the COUNTRY, FLOW, and PRODUCT concepts are included with message_ix_models.

  • Add COUNTRY_NAME to map particular labels appearing in these data.

  • Fuzzed data and tests for this functionality.

• Add .types.ParameterData and .types.MutableParameterData to type the common internal data structure in which a dict maps from MESSAGE parameter names to message_ix-structured pandas.DataFrame (PR #225).

• message_ix_models.Config—the “core” configuration class—gains methods specific to its settings (PR #225). These were formerly on Context.

• Context is no longer a subclass of dict (PR #225). This avoids its mishandling by dask version 2024.11.0 or later when Context is used with ixmp.Reporter.

  • Add MODULE_WITH_CONFIG_DATACLASS; add attributes such as Context.model that are typed to the respective class such as model.Config.

  • Add Context.asdict() for serialization.

  • Former methods such as Context.get_scenario() are aliased to their new locations, e.g. Config.get_scenario().

• Improve ScenarioInfo (PR #225):

  • Implement the | (logical OR/union) operator: si_a | si_b is a new ScenarioInfo instance with the union of the contents of the operands.

  • New method substitute_codes() to replace string codes (for instance, determined from a Scenario object) with Codes from the corresponding code list, including all annotations.

• MappingAdapter skips missing labels in the input data without raising an exception (PR #225).

• Workflow.visualize() displays in left-to-right rank direction by default (PR #225).

• convert_units() can handle MESSAGE-scheme pandas.DataFrame (PR #225).

• util.sdmx.make_enum() uses URNLookupEnum by default (PR #225). This allows to call, for instance, SSP_2024.by_urn("…Code=ICONICS:SSP(2024).2") to retrieve an enumeration number.

• make_matched_dfs() accepts dict for its par_value arg, allowing replacement of values for particular dimensions of output data frames (PR #225).

• New reporting operator quantity_from_iamc() (PR #225).

• same_node() and same_time() can handle .ParameterData (PR #225).

• report.register() becomes report.Config.register() which populates report.Config.callback (PR #225). The set of callback functions used to configure a class:.Reporter instance is thus now particular to a report.Config instance attached to a Context instance. This allows better isolation of code/tests that use different sets of callbacks.

v2024.8.6

• Add MESSAGEix-Materials (model.material) (PR #188, PR #189).

• Update MESSAGEix-Materials (model.material) (PR #201).

• Add MESSAGEix-Transport (model.transport) (PR #207, PR #208, GH #210, PR #212).

• Add EDITS (project.edits) project code and documentation (PR #204).

• Reduce log verbosity of apply_spec() (PR #202).

• Fix and update Investment and fixed costs (tools.costs) (PR #186, PR #187, PR #190, PR #195).

  • Fix jumps in cost projections for technologies with first technology year that’s after than the first model year (PR #186).

  • Change the use of base_year to mean the year to start modeling cost changes (PR #186).

  • Update cost assumptions for certain CCS technologies (PR #186).

  • Change the default fixed O&M reduction rate to 0 (PR #186).

  • Modify to use 2023 release of IEA WEO data and to use 2022 historic data for the base year (PR #187).

  • Change the default final year to 2110 (PR #190).

  • Add use_vintages to control whether vintages are used in computing fixed O&M costs (PR #195).

v2024.4.22

• Incorporate the MESSAGEix-GLOBIOM global model documentation (PR #107, PR #110). This documentation formerly lived at https://docs.messageix.org/global/ and in a separate repository at iiasa/message_doc.

• Improve tool for Migrate individual modules using git filter-repo (PR #174); expand documentation.

• New module for Investment and fixed costs (tools.costs) (PR #99).

• Update investment cost assumptions in Investment and fixed costs (tools.costs) for wind and solar technologies (PR #176).

  • Remove manually specified base year reference region costs for solar_pv_ppl and solar_pv_RC so that 2021 IEA WEO costs are used for these technologies.

  • Fix the manually specified base year reference region cost for wind_ppf.

  • Update cost reduction rates and scenarios for wind_ppf so that it follows the same narratives as wind_ppl.

• Convert Intratec data for Investment and fixed costs (tools.costs) from Excel to a simpler CSV format. (PR #167).

• Migrate ‘Legacy’ reporting (report.legacy) to provide post-processing functionality for the global model snapshot (PR #159).

• Migrate and improve code for four sources of exogenous data (PR #162): project.gea, project.shape, tools.gfei, and tools.iea.eei.

• Add a quickstart (PR #157).

• Expand Data, metadata, and configuration (PR #161).

• Add an explicit Upstream version policy (PR #162).

v2024.4.2

• The SSPUpdate data provider pulls data from the SSP 2024 “Release 3.0” data files, and handles both the earlier and current structures (PR #156).

• Improve ExoDataSource with raise_on_extra_kw() utility method, automatic copy of source keyword arguments (PR #156).

• Expose node.nodes_ex_world() for use as a genno (reporting) operator.

• Raise DeprecationWarning from util.sdmx.eval_anno(); remove internal usage of this deprecated method (PR #156).

• Reduce verbosity when using the mix-models CLI when message_data is not installed (GH #37, PR #156).

• Improve logging (PR #156).

  • Use multi-threaded logging for better performance. Logging to stdout and file is on a separate thread and does not block operations on the main thread.

  • Add automatic file logging. Log versions of packages to file when using workflow.make_click_command().

  • New CLI command mix-models last-log to retrieve the location of the latest log file.

• Update Command-line interface (PR #156).

• Improve performance in disutility.data_conversion() (PR #156).

• Use platformdirs.user_cache_path() in more places; remove cache-path handling code (PR #156).

• Add util.datetime_now_with_tz() (PR #156).

• Add util.show_versions(), wrapping ixmp.util.show_versions() and returning its output as str (PR #156).

• util.private_data_path() returns an alternate, local data path if message_data is not installed (PR #156).

• Annotate c="transport" in the commodity code list with associated IEA (E)WEB flows (PR #153).

v2024.1.29

• Add (Extended) World Energy Balances (tools.iea.web) for handling data from the International Energy Agency (IEA) Extended World Energy Balances (GH #25, PR #75).

• Add World Bank structures (tools.wb) and assign_income_groups() to assign MESSAGE regions to World Bank income groups (PR #144).

• Adjust report.compat for genno version 1.22 (GH #141, PR #142).

• Raise informative exception from ScenarioInfo.io_units() (PR #151).

v2023.11.24

Migration notes

Update code that imports from the following modules:

• message_ix_models.report.computations → use message_ix_models.report.operator.

Code that imports from the old locations will continue to work, but will raise DeprecationWarning.

Data for MESSAGEix-Nexus (model.water) is no longer included in the PyPI distributions for message_ix_models. This reduces the package size from >20 MB to <5 MB. To automatically download and unpack these data into a local directory, use mix-models fetch MESSAGEix-Nexus.

All changes

• Improve ExoDataSource (PR #137):

  • New attributes name, extra_dims.

  • New method transform() that can be overridden by subclasses.

  • New arguments archive_member, non_iso_3166 to iamc_like_data_for_query().

• New provider for exogenous data from the ADVANCE project (PR #137). This module, project.advance, supersedes tools.advance and its idiosyncratic API, which are deprecated.

• New CLI commands (PR #137):

  • mix-models testing fuzz-private-data, superseding mix-models ssp make-test-data.

  • mix-models fetch, superseding mix-models snapshot fetch.

• New utility functions (PR #137).

  • tools.iamc.describe() to generate SDMX code lists that describe the structure of particular IAMC-format data (PR #137).

  • workflow.make_click_command() to generate click commands for any Workflow.

  • util.minimum_version() to ensure compatibility with upstream packages and aid test writing.

  • util.iter_keys() to generate keys for chains of genno computations.

• Add message_ix_models.report.compat for emulating legacy reporting (PR #134).

• Rename message_ix_models.report.operator (PR #137).

• Deprecate iter_parameters() in favour of ixmp.Scenario.par_list() with indexed_by=... argument from ixmp v3.8.0 (PR #137).

v2023.10.16

• New providers for exogenous data from the SSPOriginal and SSPUpdate (PR #125) sources.

• Improved ScenarioInfo (PR #125):

  • New attributes model, scenario, version, and (settable) url; class method from_url() to allow storing Scenario identifiers on ScenarioInfo objects.

  • New property path, giving a valid path name for scenario-specific file I/O.

• Improvements to report (PR #125):

  • New report.Config class collecting recognized settings for the module.

  • context["report"] always exists as an instance of report.Config.

  • New submodule report.plot with base class and 5 plots of time-series data stored on Scenarios.

  • Submodule report.sim provides add_simulated_solution() for testing reporting configuration.

  • New operator filter_ts().

• New reusable command-line option --urls-from-file in util.click (PR #125).

• Add pyarrow to dependencies (PR #125).

v2023.9.12

All changes

• New module project.ssp (PR #122) to generate SDMX codelists for the 2017/original SSPs and the 2024 update, and provide these as Enum to other code.

• New module tools.exo_data to retrieve exogenous data for, among others, population and GDP (PR #122). This module has a general API that can be implemented by provider classes.

• New function model.emissions.get_emission_factors() and associated data file to provide data from this table in the MESSAGEix-GLOBIOM documentation (PR #122).

• New functions in util.sdmx (PR #122):

  • read(), write() to retrieve/store package data in SDMX-ML.

  • make_enum() to make pure-Python Enum (or subclass) data structures based on SDMX code lists.

• same_node() also fills “node_shares”, “node_loc”, and “node”, as appropriate (PR #122).

Deprecations

• eval_anno() is deprecated; code should instead use sdmx.model.common.AnnotableArtefact.eval_annotation(), which provides the same functionality.

v2023.9.2

• New module message_ix_models.report for reporting (PR #116). Use of this module requires ixmp and message_ix version 3.6.0 or greater.

• Add documentation on Migrate individual modules using git filter-repo using git filter-repo and helper scripts (PR #89).

v2023.7.26

• Add code and CLI commands to fetch and load MESSAGEix-GLOBIOM snapshots (PR #102). Use of this module requires ixmp and message_ix version 3.5.0 or greater.

• Add util.pooch.fetch(), a thin wrapper for using Pooch (PR #102).

• New module message_ix_models.model.macro with utilities for calibrating message_ix.macro (PR #104).

• New method Workflow.guess_target() (PR #104).

• Change in behaviour of Workflow.add_step(): the method now returns the name of the newly-added workflow step, rather than the WorkflowStep object added to carry out the step (PR #104). The former is more frequently used in code that uses Workflow.

• Add the 17-region aggregation (R17) node code list (PR #109).

• Add the 20-region aggregation (R20) node code list (PR #109).

v2023.5.31

• Adjust sdmx usage for version 2.10.0 (PR #101).

v2023.5.13

• Adjust generate_product() for pandas 2.0.0 (PR #98).

2023.4.2

• Add MESSAGEix-Nexus (model.water) (PR #88, PR #91).

• New utility function replace_par_data() (PR #90).

• disutility.get_spec() preserves all Annotations attached to the Code object used as a template for usage technologies (PR #90).

• Add CO2_Emission_Global_Total to the “A” relation codelist (PR #90).

• Adapter and MappingAdapter can be imported from message_ix_models.util (PR #90).

• Bump sdmx requirement from v2.2.0 to v2.8.0 (PR #90).

2023.2.8

• Codelists for the relation MESSAGEix set (PR #85):

  • Add three relation codelists.

  • The “bare” reference energy system now includes relations from the codelist indicated by model.Config.relations; default “A”.

• Commodities (commodity.yaml) (PR #85):

  • Add “biomass”, “non-comm”, “rc_spec”, and “rc_therm”.

  • Add “report” annotations for some items. These include string fragments to be used in variable names when reporting data in the IAMC data structure.

• generate_product() (and generate_set_elements()) can handle a regular expression to select a subset of codes for the Cartesian product (PR #85).

• New utility method Context.write_debug_archive() writes a ZIP archive containing files listed by Config.debug_paths (PR #85).

• WorkflowStep can store and apply keyword options for the optional clone() step at the start of the step execution (PR #85).

• Bugfix: WorkflowStep.__call__() ensures that Config.scenario_info on the Context instance passed to its callback matches the target scenario (PR #85).

2022.11.7

• Add the Zambia (ZMB) node code list (PR #83).

• Add the utility same_time(), to copy the set time in parameters (PR #83).

• New Config and model.Config dataclasses for clearer description/handling of recognized settings stored on Context (PR #82). ConfigHelper for convenience/utility functionality in message_ix_models-based code.

• New functions generate_product(), generate_set_elements(), get_region_codes() in model.structure (PR #82).

• Revise and improve the Workflow API (PR #82).

• Adjust for pandas 1.5.0 (PR #81).

2022.8.17

• Add nodes_ex_world() and use this in disutility.data_conversion() instead of expected a “World” node ID to be the first element in ScenarioInfo.N (PR #78).

• Add example files and documentation for IIASA Scenario Explorer metadata (PR #78).

• Expand ~ (i.e. $HOME) in the "message local data" configuration setting (PR #78).

2022.7.25

• Add get_advance_data(), and related tools for data from the ADVANCE project, including the node codelist for the data (PR #76).

• Add unit annotations to Commodities (commodity.yaml) (PR #76).

• New utility methods ScenarioInfo.io_units() to derive units for input and output parameters from units_for() commodity stocks and technology activities (PR #76).

• Transfer add_tax_emission() from message_data, improve, and add tests (PR #76).

• Unit annotations on commodity and technology codes are copied to child codes using process_units_anno() (PR #76).

• make_matched_dfs() accepts pint.Quantity to set both magnitude and units in generated data (PR #76).

• strip_par_data() also removes the set element for which data is being stripped (PR #76).

• The common CLI options --verbose and --dry-run are stored on Context automatically (PR #76).

• New utility method Context.set_scenario() (PR #76).

• iam_units.registry is the default unit registry even when message_data is not installed (PR #76).

• Expand broadcast() to allow DataFrame with multiple dimensions as input (PR #74).

2022.5.6

• Bump minimum required version of message_ix to v3.4.0 from v3.2.0 (PR #71).

• Add a documentation page on Distributed computing (PR #59).

• Add testing.not_ci() for marking tests not to be run on continuous integration services; improve session_context() (PR #62).

• apply_spec() also adds elements of the “node” set using ixmp.Platform.add_region() (PR #62).

• Add new logo the documentation (PR #68).

• Add Workflow; see Multi-scenario workflows (workflow) (PR #60).

2022.3.30

• Add adapt_R11_R12, a function for adapting data from the 11-region aggregation (R11) to the 12-region aggregation (R12) node lists (PR #56).

• Work around iiasa/ixmp#425 in disutility.data_conversion() (docs, PR #55).

2022.3.3

• Change the node name in R12.yaml from R12_CPA to R12_RCPA (PR #49).

• Register “message local data” ixmp configuration file setting and use to set the .Context.local_path when provided. See (4) Other, system-specific (“local”) directories (PR #47)

2022.1.26

• New Spec class for easier handling of specifications of model (or model variant) structure (PR #39)

• New utility function util.local_data_path() (PR #39).

• repr() of Context no longer prints a (potentially very long) list of all keys and settings (PR #39).

• as_codes() accepts a dict with Code values (PR #39).

Earlier releases

2021.11.24

• Add --years and --nodes to common_params() (PR #35).

• New utility function structure.codelists() (PR #35).

2021.7.27

• Improve caching using genno v1.8.0 (PR #29).

2021.7.22

• Migrate utilities cached(), check_support(), convert_units(), maybe_query(), series_of_pint_quantity() (PR #27)

• Add testing.NIE.

• Add the --jvmargs option to pytest (see pytest_addoption()).

• Remove .Context.get_config_file(), .get_path(), .load_config(), and .units, all deprecated since 2021-02-28.

2021.7.6

• Add identify_nodes(), a function for identifying a Node code lists based on a Scenario (PR #24).

• Add adapt_R11_R14, a function for adapting data from the 11-region aggregation (R11) to the 14-region aggregation (R14) node lists (PR #24).

• Add export_test_data() and mix-models export-test-data command (PR #16). See Prepare data for testing.

• Allow use of pytest’s persistent cache across test sessions (PR #23). See Reproducibility.

• Add the 12-region aggregation (R12) node code list (PR #14).

2021.4.7

• Add model.disutility, code for setting up structure and data for generalized consumer disutility (PR #13)

2021.3.24

• Add Years or time periods (year/*.yaml), YAML data files, ScenarioInfo.year_from_codes() and associated tests (GH #11, PR #12)

2021.3.22

• Migrate model.bare, model.build, model.cli, and associated documentation (PR #9)

• Migrate utilities: ScenarioInfo, add_par_data(), eval_anno(), iter_parameters(), and strip_par_data().

2021.3.3

• Migrate util.click, .util.logging; expand documentation (PR #8:).

• Context.clone_to_dest() method replaces clone_to_dest() function.

• Build PDF documentation on ReadTheDocs.

• Allow CLI commands from both message_ix_models and message_data via mix-models.

• Migrate mix-models techs CLI command.

2021.2.28

• Migrate Context class and testing module from message_data (PR #5:).

• Add load_private_data(), package_data_path(), private_data_path().

• Document: Data, metadata, and configuration and Command-line interface.

• Update node codelists to ensure they contain both current and former ISO 3166 codes for countries that have changed status (PR #6:). For instance, ANT dissolved into BES, CUW, and SXM in 2010; all four are included in R11_LAM so this list can be used to handle data from either before or after 2010.

2021.2.26

• Add get_codes() and related code lists (PR #2:).

• Add MessageDataFinder and document migrate (PR #3:).

2021.2.23

Initial release.