Python Code Quality
Code quality is essential for software engineering. In Python community, Python designer generated several PEP(Python Enhancement Proposals). Python Code Quality Authority provided several tools for Python.
Here are the details about what I've learned for this topic.
Python Designer's view
Python language designer discussed all Python proposals in PEP.
These are key PEPs related to Python code quality.
PEP 8 – Style Guide for Python Code
PEP 257 - Docstring Conventions
PEP 482 - Literature Overview for Type Hints
PEP 483 - The Theory of Type Hints
PEP 484 - Type Hints : Python 3.5, created 29-Sep-2014
PEP 8 Code Style
PEP 8 – Style Guide for Python Code covers the following topics:
Naming Conventions is the biggest part.
PEP 484 introduced Type Hints and PEP 526 introduced variable annotations.
Tools Design after PEP8
pycodestyle/Pep8 checks the Code style and autopep8 can do the formatter.
PEP 257 Docstring Style
Following PEP 257 - Docstring Conventions, we have three different style variations:
-
Google : recommended by Khan Academy , VS Code extension can generate docstring.
-
NumPy : NumPy, SciPy, and Pandas use it.
-
Sphinx : used by Django, NumPy, SciPy, Scikit-Learn, Matplotlib, most basic docstring format.
Reference: 3 Different Docstring Formats for Python
Documenting Python code with Sphinx
[^Google Vs NumPy’s Docstrings:]: The output for both docstrings looks similar, the main difference between the two styles is that Google uses indentation to separate sections, whereas NumPy uses underlines. NumPy style tends to require more vertical space, whereas Google-style tends to use more horizontal space. Google-style tends to be easier to read for short and simple docstrings, whereas NumPy-style tends to be easier to read for long and in-depth docstrings.
pydocstyle linter was designed to PEP257.
Linter and Formatter
The articles mentioned the following:
- Type checkers verify that your program follows their own type annotations (aka type hints). (Mypy, Pyright, Pyre, Pytype)
- Error linters point out syntax errors or other code that will result in unhandled exceptions and crashes. (Pylint, Pyflakes, Flake8)
- Style linters point out issues that don't cause bugs but make the code less readable or are not in line with style guides such as Python's PEP 8 document. (Pylint, Flake8)
- Packaging linters point out issues related to packaging your code for distribution on PyPI with properly formatted descriptions, versions, and meta data fields. (Pyroma)
- Security linters point out possible security vulnerabilities in your code. (Bandit, Dodgy)
- Code formatters change the style of your code (mostly revolving around proper whitespace) without affecting the behavior of the program. (Black)
- Dead code linters remove commented-out code from your program, since this practice should be skipped in favor of proper version control. (Vulture, eradicate)
- Docstring linters/formatters point out (and may correctly format) style issues in docstrings that aren't in line with Python's PEP 257 document. (pydocstringformatter, docformatter)
- Complexity analyzers point out code that is so complex that they can affect readability. (mccabe, Radon)
Ruff
Ruff can be used to replace Flake8 (plus dozens of plugins), Black, isort, pydocstyle, pyupgrade, autoflake, and more, all while executing tens or hundreds of times faster than any individual tool. Pylint supports 409 rules compared to Ruff's 224 rules.
Pylint
pylint is a static code analyser to detect issues in code. It uses AST analysis. However, it's fairly slow for the big project. Ruff is working on the project to implement all rules in Pylint:
Implement Pylint, we can use the following rules to enable
[tool.pylint]
disable = ["all"]
enable = [
"abstract-class-instantiated",
"abstract-method",
"access-member-before-definition",
"anomalous-unicode-escape-in-string",
"arguments-differ",
"arguments-out-of-order",
"arguments-renamed",
"assigning-non-slot",
"assignment-from-no-return",
"assignment-from-none",
"attribute-defined-outside-init",
"bad-builtin",
"bad-except-order",
"bad-exception-cause",
"bad-file-encoding",
"bad-indentation",
"bad-mcs-classmethod-argument",
"bad-mcs-method-argument",
"bad-reversed-sequence",
"bad-staticmethod-argument",
"bad-super-call",
"bad-thread-instantiation",
"catching-non-exception",
"chained-comparison",
"class-variable-slots-conflict",
"compare-to-zero",
"comparison-with-callable",
"condition-evals-to-constant",
"confusing-consecutive-elif",
"confusing-with-statement",
"consider-swap-variables",
"consider-using-assignment-expr",
"consider-using-augmented-assign",
"consider-using-dict-items",
"consider-using-enumerate",
"consider-using-f-string",
"consider-using-from-import",
"consider-using-join",
"consider-using-max-builtin",
"consider-using-min-builtin",
"consider-using-namedtuple-or-dataclass",
"consider-using-tuple",
"consider-using-with",
"cyclic-import",
"deprecated-argument",
"deprecated-class",
"deprecated-decorator",
"deprecated-method",
"deprecated-module",
"deprecated-typing-alias",
"dict-init-mutate",
"dict-iter-missing-items",
"differing-param-doc",
"differing-type-doc",
"disallowed-name",
"duplicate-code",
"empty-comment",
"global-at-module-level",
"global-variable-undefined",
"import-error",
"import-private-name",
"inconsistent-mro",
"inherit-non-class",
"invalid-bool-returned",
"invalid-bytes-returned",
"invalid-character-carriage-return",
"invalid-characters-in-docstring",
"invalid-class-object",
"invalid-enum-extension",
"invalid-envvar-value",
"invalid-format-index",
"invalid-format-returned",
"invalid-getnewargs-ex-returned",
"invalid-getnewargs-returned",
"invalid-hash-returned",
"invalid-index-returned",
"invalid-length-hint-returned",
"invalid-length-returned",
"invalid-metaclass",
"invalid-overridden-method",
"invalid-repr-returned",
"invalid-sequence-index",
"invalid-slice-index",
"invalid-slice-step",
"invalid-slots",
"invalid-slots-object",
"invalid-star-assignment-target",
"invalid-str-returned",
"invalid-unary-operand-type",
"invalid-unicode-codec",
"isinstance-second-argument-not-valid-type",
"logging-format-truncate",
"logging-unsupported-format",
"method-cache-max-size-none",
"method-hidden",
"misplaced-format-function",
"missing-any-param-doc",
"missing-format-attribute",
"missing-kwoa",
"missing-param-doc",
"missing-parentheses-for-call-in-test",
"missing-raises-doc",
"missing-return-doc",
"missing-return-type-doc",
"missing-timeout",
"missing-type-doc",
"missing-yield-doc",
"missing-yield-type-doc",
"mixed-line-endings",
"modified-iterating-dict",
"modified-iterating-list",
"modified-iterating-set",
"multiple-constructor-doc",
"nan-comparison",
"no-member",
"no-name-in-module",
"no-value-for-parameter",
"non-iterator-returned",
"non-parent-init-called",
"non-str-assignment-to-dunder-name",
"nonlocal-and-global",
"not-a-mapping",
"not-an-iterable",
"not-async-context-manager",
"not-callable",
"not-context-manager",
"overlapping-except",
"overridden-final-method",
"pointless-string-statement",
"possibly-unused-variable",
"potential-index-error",
"preferred-module",
"raising-bad-type",
"raising-format-tuple",
"raising-non-exception",
"redeclared-assigned-name",
"redefined-outer-name",
"redefined-slots-in-subclass",
"redefined-variable-type",
"redundant-keyword-arg",
"redundant-returns-doc",
"redundant-u-string-prefix",
"redundant-unittest-assert",
"redundant-yields-doc",
"self-cls-assignment",
"shallow-copy-environ",
"signature-differs",
"simplifiable-condition",
"simplifiable-if-expression",
"simplifiable-if-statement",
"simplify-boolean-expression",
"singledispatch-method",
"singledispatchmethod-function",
"star-needs-assignment-target",
"stop-iteration-return",
"subclassed-final-class",
"super-init-not-called",
"super-without-brackets",
"superfluous-parens",
"too-few-public-methods",
"too-many-ancestors",
"too-many-function-args",
"too-many-instance-attributes",
"too-many-lines",
"too-many-nested-blocks",
"too-many-try-statements",
"trailing-newlines",
"trailing-whitespace",
"unbalanced-dict-unpacking",
"unbalanced-tuple-unpacking",
"undefined-loop-variable",
"unexpected-keyword-arg",
"unexpected-line-ending-format",
"unhashable-member",
"unnecessary-dunder-call",
"unnecessary-ellipsis",
"unpacking-non-sequence",
"unreachable",
"unsubscriptable-object",
"unsupported-assignment-operation",
"unsupported-binary-operation",
"unsupported-delete-operation",
"unsupported-membership-test",
"unused-private-member",
"unused-wildcard-import",
"use-implicit-booleaness-not-comparison",
"use-implicit-booleaness-not-len",
"use-maxsplit-arg",
"used-before-assignment",
"useless-param-doc",
"useless-parent-delegation",
"useless-type-doc",
"using-constant-test",
"using-final-decorator-in-unsupported-version",
"while-used",
"wrong-exception-operation",
"wrong-spelling-in-comment",
"wrong-spelling-in-docstring",
]
Type Checking
PEP 484 Type Hints stub files was introduced in 2014.
PEP 526 - Syntax for Variable Annotations
PEP 560 - Core support for typing module and generic types
PEP 589 - TypedDict: Type Hints for Dictionaries with a Fixed Set of Keys
PEP 604 - Allow writing union types as X | Y
PEP 692 Using TypedDict for more precise **kwargs typing
4 Python type checkers to keep your code clean did a survey about four different type checking tools.
Mypy, by Dropbox in 2012, the first static type checking system for Python. Introduced "# type: ignore"
Pyright by Microsoft, part of Pylance and VSCode extension. Can work with pyproject.toml file.
Pyre by Facebook and Instagram, two tools in one: a type checker (Pyre) and a static code analysis tool (Pysa). It has "infer" command line option to guess about the types used.Pyre will also work with .pyi-format stub files.Pysa performs “taint analysis” on code to identify potential security issues
Pytype, by Google differs from the likes of Mypy in using inference instead of just type descriptors.
As a VSCode user, I've found Pyright works great.
Tools Performance
How can we improve the performance? Can we use Rust Binding Python just like Ruff to boost Pyright performance?