Polish up python support

bin/qmk

@@ -31,7 +31,10 @@ if script_name == 'qmk':
 elif not script_name.startswith('qmk-'):
     cli.log.error('Invalid symlink, must start with "qmk-": %s', script_name)
 else:
-    subcommand = script_name.replace('-', '.')
+    subcommand = script_name.replace('-', '.').replace('_', '.').split('.')
+    subcommand.insert(1, 'cli')
+    subcommand = '.'.join(subcommand)
+
     try:
         import_module(subcommand)
     except ModuleNotFoundError:

bin/qmk-hello

@@ -0,0 +1 @@
+qmk

docs/_summary.md

@@ -8,6 +8,7 @@
 
 * [QMK Basics](README.md)
   * [QMK Introduction](getting_started_introduction.md)
+  * [QMK CLI](cli.md)
   * [Contributing to QMK](contributing.md)
   * [How to Use Github](getting_started_github.md)
   * [Getting Help](getting_started_getting_help.md)
@@ -43,6 +44,7 @@
   * [Useful Functions](ref_functions.md)
   * [Configurator Support](reference_configurator_support.md)
   * [info.json Format](reference_info_json.md)
+  * [Python Development](python_development.md)
 
 * [Features](features.md)
   * [Basic Keycodes](keycodes_basic.md)

docs/cli.md

@@ -0,0 +1,31 @@
+# QMK CLI
+
+This page describes how to setup and use the QMK CLI.
+
+# Overview
+
+The QMK CLI makes building and working with QMK keyboards easier. We have provided a number of commands to help you work with QMK:
+
+* `qmk compile-json`
+
+# Setup
+
+Simply add the `qmk_firmware/bin` directory to your `PATH`. You can run the `qmk` commands from any directory.
+
+```
+export PATH=$PATH:$HOME/qmk_firmware/bin
+```
+
+You may want to add this to your `.profile`, `.bash_profile`, `.zsh_profile`, or other shell startup scripts.
+
+# Commands
+
+## `qmk compile-json`
+
+This command allows you to compile JSON files you have downloaded from <https://config.qmk.fm>.
+
+**Usage**:
+
+```
+qmk compile-json -f mine.json
+```

docs/coding_conventions_python.md

@@ -13,15 +13,15 @@ Most of our style follows PEP8 with some local modifications to make things less
 * In general we don't wrap lines, they can be as long as needed. If you do choose to wrap lines please do not wrap any wider than 76 columns.
 * Some of our practices conflict with the wider python community to make our codebase more approachable to non-pythonistas.
 
-# Flake8
+# YAPF
 
-You can use flake8 to check your code for style. We will provide a flake8 config file in the future.
+You can use [yapf](https://github.com/google/yapf) to style your code. We provide a config in [setup.cfg](setup.cfg).
 
 # Imports
 
 We don't have a hard and fast rule for when to use `import ...` vs `from ... import ...`. Understandability and maintainability is our ultimate goal.
 
-Generally we prefer to import specific function and class names from a module to keep code shorter and easier to understand. Sometimes this results in a name that is ambiguous, and in such cases we prefer to import the module instead. You should avoid using the "as" keyword when importing.
+Generally we prefer to import specific function and class names from a module to keep code shorter and easier to understand. Sometimes this results in a name that is ambiguous, and in such cases we prefer to import the module instead. You should avoid using the "as" keyword when importing, unless you are importing a compatability module.
 
 Imports should be one line per module. We group import statements together using the standard python rules- system, 3rd party, local.
 
@@ -154,33 +154,15 @@ Make your try/except blocks as short as possible. If you need a lot of try state
 
 # Tuples
 
-When defining one-item tuples always include a trailing comma so that it is obvious you are using a tuple. Do not rely on implicit one-item tuple unpacking.
+When defining one-item tuples always include a trailing comma so that it is obvious you are using a tuple. Do not rely on implicit one-item tuple unpacking. Better still use a list which is unambiguous.
 
 This is particularly important when using the printf-style format strings that are commonly used.
 
 # Lists and Dictionaries
 
-You should generally prefer to keep the definition on a single line. Break out to multiple lines sooner rather than later to aid readability.
+We have configured YAPF to differentiate between sequence styles with a trailing comma. When a trailing comma is omitted YAPF will format the sequence as a single line. When a trailing comma is included YAPF will format the sequence with one item per line.
 
-When you break it out to multiple lines do so in a way that makes it easier to edit later. Sometimes this means one item per line. Sometimes this means grouping like items together on the same line, so you can easily move them around. Use your best judgement.
-
-# Trailing Commas
-
-It is OK to include trailing commas in sequences.
-
-# Whitespace
-
-Follow standard typographic rules for the use of spaces around punctuation.
-
-No whitespace inside parentheses, brackets or braces.
-
-No whitespace before a comma, semicolon, or colon. Do use whitespace after a comma, semicolon, or colon, except at the end of the line.
-
-No whitespace before the open paren/bracket that starts an argument list, indexing or slicing.
-
-Surround binary operators with a single space on either side for assignment (=), comparisons (==, <, >, !=, <>, <=, >=, in, not in, is, is not), and Booleans (and, or, not). Use your better judgment for the insertion of spaces around arithmetic operators (+, -, *, /, //, %, **, @).
-
-Never use spaces around = when passing keyword arguments or defining a default parameter value
+You should generally prefer to keep short definition on a single line. Break out to multiple lines sooner rather than later to aid readability and maintainability.
 
 # Parentheses
 
@@ -197,6 +179,8 @@ print('Hello, %s!' % (name,))
 
 This style is used by the logging module, which we make use of extensively, and we have adopted it in other places for consistency. It is also more familiar to C programmers, who are a big part of our casual audience.
 
+Our included CLI module has support for using these without using the percent (%) operator. Look at `cli.echo()` and the various `cli.log` functions (EG, `cli.log.info()`) for more details.
+
 # Comprehensions & Generator Expressions
 
 We encourage the liberal use of comprehensions and generators, but do not let them get too complex. If you need complexity fall back to a for loop that is easier to understand.
@@ -215,7 +199,7 @@ Conditional expressions are if statements that are in line with code. For exampl
 x = 1 if cond else 2
 ```
 
-It's generally not a good idea to use these as function arguments, list and dictionary items, etc. It's too easy to overlook.
+It's generally not a good idea to use these as function arguments, sequence items, etc. It's too easy to overlook.
 
 # Default Argument Values
 
@@ -327,11 +311,3 @@ FIXME(username): Revisit this code when the frob feature is done.
 # Unit Tests
 
 These are good. We should have some one day.
-
-# Parting Words
-
-BE CONSISTENT.
-
-If you're editing code, take a few minutes to look at the code around you and determine its style. If they use spaces around all their arithmetic operators, you should too. If their comments have little boxes of hash marks around them, make your comments have little boxes of hash marks around them too.
-
-The point of having style guidelines is to have a common vocabulary of coding so people can concentrate on what you're saying rather than on how you're saying it. We present global style rules here so people know the vocabulary, but local style is also important. If code you add to a file looks drastically different from the existing code around it, it throws readers out of their rhythm when they go to read it. Avoid this.

docs/python_development.md

@@ -0,0 +1,45 @@
+# Python Development in QMK
+
+This document gives an overview of how QMK has structured its python code. You should read this before working on any of the python code.
+
+## Script directories
+
+There are two places scripts live in QMK: `qmk_firmware/bin` and `qmk_firmware/util`. You should use `bin` for any python scripts that utilize the `qmk` wrapper. Scripts that are standalone and not run very often live in `util`.
+
+We discourage putting anything into `bin` that does not utilize the `qmk` wrapper. If you think you have a good reason for doing so please talk to us about your use case.
+
+## Python Modules
+
+Most of the QMK python modules can be found in `qmk_firmware/lib/python`. This is the path that we append to `sys.path`.
+
+We have a module hierarchy under that path:
+
+* `qmk_firmware/lib/python`
+    * `milc.py` - The CLI library we use. Will be pulled out into its own module in the future.
+    * `qmk` - Code associated with QMK
+        * `cli` - Modules that will be imported for CLI commands.
+        * `errors.py` - Errors that can be raised within QMK apps
+        * `keymap.py` - Functions for working with keymaps
+
+## CLI Scripts
+
+We have a CLI wrapper that you should utilize for any user facing scripts. We think it's pretty easy to use and it gives you a lot of nice things for free.
+
+To use the wrapper simply place a module into `qmk_firmware/lib/python/qmk/cli`, and create a symlink to `bin/qmk` named after your module. Dashes in command names will be converted into dots so you can use hierarchy to manage commands.
+
+When `qmk` is run it checks to see how it was invoked. If it was invoked as `qmk` the module name is take from `sys.argv[1]`. If it was invoked as `qmk-<module-name>` then everything after the first dash is taken as the module name. Dashes and underscores are converted to dots, and then `qmk.cli` is prepended before the module is imported.
+
+The module uses `@cli.entrypoint()` and `@cli.argument()` decorators to define an entrypoint, which is where execution starts.
+
+## Example CLI Script
+
+We have provided a QMK Hello World script you can use as an example. To run it simply run `qmk hello` or `qmk-hello`. The source code is listed below.
+
+```
+from milc import cli
+
+@cli.argument('-n', '--name', default='World', help='Name to greet.')
+@cli.entrypoint('QMK Python Hello World.')
+def main(cli):
+    cli.echo('Hello, %s!', cli.config.general.name)
+```

lib/python/qmk/cli/hello.py

@@ -0,0 +1,11 @@
+"""QMK Python Hello World
+
+This is an example QMK CLI script.
+"""
+from milc import cli
+
+
+@cli.argument('-n', '--name', default='World', help='Name to greet.')
+@cli.entrypoint('QMK Python Hello World.')
+def main(cli):
+    cli.echo('Hello, %s!', cli.config.general.name)

lib/python/qmk/errors.py

@@ -1,8 +1,4 @@
-class Error(Exception):
-    pass
-
-
-class NoSuchKeyboardError(Error):
+class NoSuchKeyboardError(Exception):
     """Raised when we can't find a keyboard/keymap directory.
     """
 

util/freebsd_install.sh

@@ -1,4 +1,5 @@
 #!/bin/sh
+util_dir=$(dirname "$0")
 pkg update
 pkg install -y \
 	git \

util/linux_install.sh

@@ -8,6 +8,8 @@ SLACKWARE_WARNING="You will need the following packages from slackbuilds.org:\n\
 
 SOLUS_INFO="Your tools are now installed. To start using them, open new terminal or source these scripts:\n\t/usr/share/defaults/etc/profile.d/50-arm-toolchain-path.sh\n\t/usr/share/defaults/etc/profile.d/50-avr-toolchain-path.sh"
 
+util_dir=$(dirname "$0")
+
 if grep ID /etc/os-release | grep -qE "fedora"; then
 	sudo dnf install \
 		arm-none-eabi-binutils-cs \

util/macos_install.sh

@@ -1,5 +1,7 @@
 #!/bin/bash
 
+util_dir=$(dirname "$0")
+
 if ! brew --version 2>&1 > /dev/null; then
 	echo "Error! Homebrew not installed or broken!"
 	echo -n "Would you like to install homebrew now? [y/n] "

util/msys2_install.sh

@@ -5,6 +5,7 @@ download_dir=~/qmk_utils
 avrtools=avr8-gnu-toolchain
 armtools=gcc-arm-none-eabi
 installflip=false
+util_dir=$(dirname "$0")
 
 echo "Installing dependencies needed for the installation (quazip)"
 pacman --needed -S base-devel mingw-w64-x86_64-toolchain msys/git msys/p7zip msys/python3 msys/unzip

util/wsl_install.sh

@@ -1,6 +1,7 @@
 #!/bin/bash
 
-dir=$(cd -P -- "$(dirname -- "$0")" && pwd -P)
+util_dir=$(dirname "$0")
+dir=$(cd -P -- "$util_dir" && pwd -P)
 pushd "$dir";
 
 if [[ $dir != /mnt/* ]];