Is there any Python tool that can set configurations in dict/json format while can be modified by commandline arguments?

Is there any Python tool that can set configurations in dict/json format while can be modified by command line arguments?

For us who often run research experiments, do you often need to set a lot of command line arguments at the beginning of a python file and call them in the following code as args.*:

For example, the following example paragraph:

parser = argparse.ArgumentParser(description='index')
parser.add_argument('--index', default=0, type=int, help='party index')
parser.add_argument('--party_num', default=100, type=int)
args = parser.parse_args()

print(args.index)

With one more additional parameter, we need to write one line of parser.add_argument(...), when handwriting configuration of each parameter, it will be very tedious such as the name needs to add --, and modify the default value, type and description of the time is very troublesome, finally will lead to very long code and inconvenient to maintain.

Even if you use a more advanced library click, you still need to write option constantly, and you need to write the same amount of parameters at the input parameter field of the function to match all the option, writing code is really tedious, such as the following Click code:

import click

from caesar_encryption import encrypt

@click.command()
@click.argument('text', nargs=-1)
@click.option('--decrypt/--encrypt', '-d/-e')
@click.option('--key', '-k', default=1)
def caesar(text, decrypt, key):
    text_string = ' '.join(text)
    if decrypt:
        key = -key
    cyphertext = encrypt(text_string, key)
    click.echo(cyphertext)

if __name__ == '__main__':
    caesar()

I don't want to specify option over and over again when writing code, and write a lot of corresponding parameters in the function name parameter field, which is very troublesome.

So would it make the code look more structured and clearer if the parameter configuration could be written in the format of Python objects/JSON?

Also, is it possible to directly read and write command line arguments as if they were Python objects, or deep copy them, as if no command line arguments had been configured?

Most importantly, can we make the process of writing code faster, instead of adding a little bit of configuration line by line?

🔴 No definitive solution yet

📌 Solution 1

Easy-to-use Commandline Configuration Tool

https://github.com/NaiboWang/CommandlineConfig

A library for users to write (experiment in research) configurations in Python Dict or JSON format, while can read parameters from the command line to modify values.

Catalogue

- [Easy-to-use Commandline Configuration Tool](#easy-to-use-commandline-configuration-tool)
  - [Catalogue](#catalogue)
  - [Usage](#usage)
    - [Please submit issue](#please-submit-issue)
    - [Installation](#installation)
    - [Configuration Way](#configuration-way)
    - [Configuration parameters read and write method](#configuration-parameters-read-and-write-method)
      - [Write method](#write-method)
      - [Reading method](#reading-method)
      - [Pass configuration to functions](#pass-configuration-to-functions)
      - [Copy configuration](#copy-configuration)
      - [Store configuration parameters to local file or database](#store-configuration-parameters-to-local-file-or-database)
  - [Advanced options](#advanced-options)
    - [Restrict parameter input values to fixed enum types](#restrict-parameter-input-values-to-fixed-enum-types)
    - [Print parameter help descriptions](#print-parameter-help-descriptions)
      - [Set parameter descriptions](#set-parameter-descriptions)
      - [Print parameter help](#print-parameter-help)
  - [Things need attention](#things-need-attention)
    - [Conflict with Argparse](#conflict-with-argparse)
    - [Input value forced conversion](#input-value-forced-conversion)
    - [The list parameter needs to be assigned with a backslash before the string element quotes](#the-list-parameter-needs-to-be-assigned-with-a-backslash-before-the-string-element-quotes)
    - [Parameter naming convention](#parameter-naming-convention)
    - [Unlimited layer of nested objects](#unlimited-layer-of-nested-objects)
    - [Parameter integrity check, all parameters to be modified must be predefined](#parameter-integrity-check-all-parameters-to-be-modified-must-be-predefined)
    - [Special configurations in zsh environment](#special-configurations-in-zsh-environment)
  - [Full conversion example](#full-conversion-example)
  - [Example Running Script](#example-running-script)
  - [Shattered thoughts](#shattered-thoughts)
  - [TODO](#todo)

Usage

Please submit issue

If you encounter any problems during using with this tool, please raise an issue in the github page of this project, I will solve the bugs and problems encountered at the first time.

Meanwhile, welcome to submit issues to propose what functions you want to add to this tool and I will implement them when possible.

Installation

There are two ways to install this library:

    1. Install via pip:

        pip install commandline_config
      

    If already installed, you can upgrade it by the following command:

    ```shell
      pip install commandline_config --upgrade
    ```
    
    1. Import the commandline_config.py file directly from the /commandline_config folder of the github project into your own project directory, you need to install the dependency package prettytable:
    pip install prettytable
    

Configuration Way

    1. Import library:
    from commandline_config import Config
    
    1. Set the parameter name and initial value in JSON/Python Dict format, and add the parameter description by # comment. Currently supports nesting a dict inside another dict, and can nest unlimited layers.

           preset_config = {
             "index": 1, # Index of party
             "dataset": "mnist",
             'lr': 0.01, # learning rate 
             'normalization': True,
             "multi_information":[1,0.5,'test',"TEST"], # list
             "dbinfo":{
               "username":"nus",
               "password":123456,
               "retry_interval_time":5.5,
               "multi":{
                 "test":0.01, # Can nest to the 3th layer
               },
               "save_password":False,
               "certificate_info":["1",2,[3.5]] 
             }
           }
      

    That is, the initial configuration of the program is generated. Each key defined in preset_config dict is the parameter name and each value is the initial value of the parameter, and at the same time, the initial value type of the parameter is automatically detected according to the type of the set value.

    The above configuration contains six parameters: index, dataset, batch, normalization, multi_information and dbinfo, where the type of the parameter index is automatically detected as int, the default value is 1 and the description is "Index of party".

    Similarly, The type and default value of the second to fifth parameter are string: "mnist"; float:0.01; bool:True; list:[1,0.5,'test', "TEST"].

    The sixth parameter is a nested dictionary of type dict, which also contains 6 parameters, with the same type and default values as the first 6 parameters, and will not be repeated here.

    1. Create a configuration class object by preset_config dict in any function you want.
    if __name__ == '__main__':
        config = Config(preset_config)
        # Or give the configuration a name:
        config_with_name = Config(preset_config, name="Federated Learning Experiments")
    
        # Or you can store the preset_config in local file configuration.json and pass the filename to the Config class.
        config_from_file = Config("configuration.json")
    

    This means that the configuration object is successfully generated.

    1. Configuration of parameters can be printed directly via print function:
        print(config_with_name)
      ```
    
    The output results are:
    
    
    
    
      Configurations of Federated Learning Experiments:
    +-------------------+-------+--------------------------+
    |        Key        |  Type | Value                    |
    +-------------------+-------+--------------------------+
    |       index       |  int  | 1                        |
    |      dataset      |  str  | mnist                    |
    |         lr        | float | 0.01                     |
    |   normalization   |  bool | True                     |
    | multi_information |  list | [1, 0.5, 'test', 'TEST'] |
    |       dbinfo      |  dict | See sub table below      |
    +-------------------+-------+--------------------------+
    
    Configurations of dict dbinfo:
    +---------------------+-------+-----------------+
    |         Key         |  Type | Value           |
    +---------------------+-------+-----------------+
    |       username      |  str  | nus             |
    |       password      |  int  | 123456          |
    | retry_interval_time | float | 5.5             |
    |    save_password    |  bool | False           |
    |   certificate_info  |  list | ['1', 2, [3.5]] |
    +---------------------+-------+-----------------+
    
    Configurations of dict multi:
    +------+-------+-------+
    | Key  |  Type | Value |
    +------+-------+-------+
    | test | float | 15.0  |
    +------+-------+-------+
    ```
    

    Here the information of all parameters will be printed in table format. If you want to change the printing style, you can modify it by config_with_name.set_print_style(style=''). The values that can be taken for style are: both, table, json which means print both table and json at the same time, print only table, and json dictionary only.

    E.g.:

        # Only print json 
        config_with_name.set_print_style('json')
        print(config_with_name)
        print("----------")
        # Print table and json at the same time
        config_with_name.set_print_style('table')
        print(config_with_name)
      ```
    
    The output results are:
    
    
    
    
    Configurations of Federated Learning Experiments:
    {'index': 1, 'dataset': 'mnist', 'lr': 0.01, 'normalization': True, 'multi_information': [1, 0.5, 'test', 'TEST'], 'dbinfo': 'See below'}
    
    Configurations of dict dbinfo:
    {'username': 'nus', 'password': 123456, 'retry_interval_time': 5.5, 'save_password': False, 'certificate_info': ['1', 2, [3.5]]}
    
    
    ----------
    
    
    Configurations of Federated Learning Experiments:
    +-------------------+-------+--------------------------+
    |        Key        |  Type | Value                    |
    +-------------------+-------+--------------------------+
    |       index       |  int  | 1                        |
    |      dataset      |  str  | mnist                    |
    |         lr        | float | 0.01                     |
    |   normalization   |  bool | True                     |
    | multi_information |  list | [1, 0.5, 'test', 'TEST'] |
    |       dbinfo      |  dict | See sub table below      |
    +-------------------+-------+--------------------------+
    {'index': 1, 'dataset': 'mnist', 'lr': 0.01, 'normalization': True, 'multi_information': [1, 0.5, 'test', 'TEST'], 'dbinfo': 'See below'}
    
    Configurations of dict dbinfo:
    +---------------------+-------+-----------------+
    |         Key         |  Type | Value           |
    +---------------------+-------+-----------------+
    |       username      |  str  | nus             |
    |       password      |  int  | 123456          |
    | retry_interval_time | float | 5.5             |
    |    save_password    |  bool | False           |
    |   certificate_info  |  list | ['1', 2, [3.5]] |
    +---------------------+-------+-----------------+
    {'username': 'nus', 'password': 123456, 'retry_interval_time': 5.5, 'save_password': False, 'certificate_info': ['1', 2, [3.5]]}
    
    Configurations of dict multi:
    +------+-------+-------+
    | Key  |  Type | Value |
    +------+-------+-------+
    | test | float | 15.0  |
    +------+-------+-------+
    Configurations of dict multi:
    {'test': 15.0}
    ```
    

Configuration parameters read and write method

Write method

Configuration parameter values can be written in three ways.

    1. To receive command line arguments, simply pass --index 1 on the command line to modify the value of index to 1. Also, the considerations for passing values to different types of arguments are:
    • When passing bool type, you can use 0 or False for False, 1 or True or no value after the parameter for True: --normalization 1 or --normalization True or --normalization all can set the value of parameter normalization in the configuration to True.

    • When passing list type, empty array and multi-dimensional arrays can be passed.

    • To modify the value in the nested dict, please use --nested-parameter-name.sub-parameter-name.sub-parameter-name.….sub-parameter-name value to modify the value in the nested object, such as --dbinfo.password 987654 to change the value of the password parameter in the dbinfo subobject to 987654; --dbinfo.multi.test 1 to change the value of the test parameter in the multi dict which is in dbinfo subobject to ```. Currently this tool can supports unlimited layers/levels of nesting.

    • Note that the argument index must be in the preset_config object defined above:

          python test.py --dbinfo.password 987654 --dbinfo.multi.test 1 --index 0 --dataset emnist --normalization 0 --multi_information [\'sdf\',1,\"3.3\",,True,[1,[]]] 
      
    1. Use config.index = 2 directly in the code to change the value of the parameter index to 2. Again, list type parameters can be assigned as empty or multidimensional arrays. For nested objects, you can use config.dbinfo.save_password=True to modify the value of the save_password parameter in sub dict dbinfo to True.
    1. Way 1 and 2 will trigger type checking, that is, if the type of the assigned value and the type of the default value in the predefined dict preset_config does not match, the program will report an error, therefore, if you do not want to force type checking, you can use config["index"] = "sdf" to force the value of the parameter index to the string sdf (not recommended, it will cause unexpected impact).

Reading method

Read the value of the parameter dataset directly by means of config.dataset or config["dataset"].

 ```python
    print(config.dataset, config["index"])
    ```

The value of an argument a will be read by this order: the last value modified by config.a = * > the value of --a 2 specified by the command line > the initial value specified by "a":1 defined by preset_config.

For the list type, if a multidimensional array is passed, the information can be read via standard slice of python:

  ```python
    config.dbinfo.certificate_info = [1,[],[[2]]]
    print(config.dbinfo.certificate_info[2][0][0])
    ```

For parameters in nested objects, there are four ways to read the values of the parameters, all of which can be read successfully:

```
print(config.dbinfo.username)
print(config["dbinfo"].password)
print(config.dbinfo["retry_interval_time"])
print(config["dbinfo"]["save_password"])
```

Pass configuration to functions

Simply pass the above config object as a parameter to the function and call it:

 ```python
    def print_dataset_name(c):
      print(c.dataset, c["dataset"], c.dbinfo.certificate_info)
    
    print_dataset_name(c=config)
    ```

Copy configuration

A deep copy of the configuration object can be made by the deepcopy method:

 ```python
    from copy import deepcopy
    copy_config = deepcopy(config)
    # Modify new configuration's parameter value, will not affect the orignal configuration
    copy_config.index=15 
    ```

Store configuration parameters to local file or database

The entire parameter configuration can be stored to a local file or uploaded to a remote server such as mongodb, simply by config.save() storing the configuration as a config name (or config if there is no name).json file in the same directory, or you can specify the file name and path as follows:

 ```python
    config.save("config/test_config.json")
    ```

Then we successfully save the configuration to the local configuration.json file inside the config folder. The file content is as follows:

```json
{
  "index": 5,
  "dataset": "sdf",
  "lr": 15.5,
  "normalization": true,
  "msg_config": { "test": "ttt" },
  "multi_information": [2, "sd", "sdfdsf"],
  "dbinfo": {
    "username": "test",
    "password": 1,
    "retry_interval_time": 22.0,
    "multi":{
      "test": 0.01
    },
    "save_password": false,
    "certificate_info": [1, [], [[2]]]
  }
}
```

To store it into the database such as mongodb, you need to get the json sequence first corresponding to the parameters with the info = config.get_config() command, and serialize it with the json library.

The whole configuration parameters can be stored in a local file in JSON mode or uploaded to a remote server such as mongodb. You need to get the corresponding JSON sequence of parameters through the info = config.get_config() command and then insert it with corresponding insert command.

For example, to store the config_with_name configuration to mongodb:

 ```python
    import pymongo
    myclient = pymongo.MongoClient('mongodb://username:example.com:27017/', connect=False)
    mydb = myclient['exps']
    table = mydb["table"]
    # Get the configurations
    configuration = config.get_config()
    # Insert configuration dict into mongodb table
    table.insert_one(configuration)
    
    # Or make configuration as part of a bigger dict
    all_info = {
      "exp_time":"20220925",
      "configuration":configuration
    }
    table.insert_one(all_info)
    ```

Advanced options

Restrict parameter input values to fixed enum types

Set advanced options, such as enumerating Enum types, by passing the options parameter of the Config argument to the Config class.

 ```python
    option={}
    config = Config(preset_config, options=option)
    ```

If you want to limit the value of a parameter to a certain range, you can do so by configuring:

 ```python
    advanced_options = {
        'lr': {
            "enum": [0.001, 15.5, 0.01, 0.1] # restrict the lr value to one of 0.001, 15.5, 0.01, 0.1
        },
        'index': {
            "enum": [1, 2, 3] # Restrict the index value to 1, 2 and 3
        },
        "dbinfo": {
            "username": {
                "enum": ["XDU", "ZJU", "NUS"] # restrict the dbinfo.username field to XDU, ZJU and NUS
            },
            "multi":{
                "test":{
                    "enum": [1,0.1, 0.01, 15] # 3 layers nested
                }
            }
        },
    }
    
    config = Config(preset_config, options=advanced_options)
    ```

If enum is set, the following three ways to set a parameter to a value other than the qualified/speficied value will all report an error.

    1. The initial value of index is set to a value other than 1,2,3 in preset_config:

      preset_config = {
        "index":4,
      }
      
    1. The command line passes unqualified/unspecified values for the lr argument
      python example.py --lr 0.02
      ```
    
    
    1. The code changes the value of dbinfo.username to a value other than XDU, ZJU and NUS.

      config.dbinfo.username = "UEST"
      

    The output are:

    AttributeError: Can not set value 4 because the key 'index' has set enum list and you the value 4 is not in the enum list [1, 2, 3]!
    
    AttributeError: Can not set value 0.02 because the key 'lr' has set enum list and you the value 0.02 is not in the enum list [0.001, 15.5, 0.01, 0.1]!
    
    AttributeError: Can not set value nus because the key 'username' has set enum list and you the value nus is not in the enum list ['XDU', 'ZJU', 'NUS']!
    

mand line.