https://python-packaging.readthedocs.io/en/latest/command-line-scripts.html

use asyncio instead of bash and tee?

examples that show what you can do:

• single-file notebook server!
• demo with axolotl
• simple webserver
• use pandas to convert CSV to excel
• use language model
• use for PDF extraction

Try to convert this to a single comment block:

# create conda env
conda create -n multistep python=3.10 # higher versions might break vllm
conda activate multistep

# Linux install cuda and cuda toolkit and pytorch. Good as of 2024-05-14 Tue 15:45
conda install cuda -c nvidia/label/cuda-12.1.0
conda install pytorch torchvision torchaudio pytorch-cuda=12.1 -c pytorch -c nvidia

# conda-install conda-packaged dependencies
conda install openai networkx numpy pandas regex

# pip-install pip-only dependencies
python3 -m pip install claudette
python3 -m pip install vllm
python3 -m pip install datasets
python3 -m pip install nltk

• could name directory with first script name used
  • con. hides later scripts
• could add symlink in the dir back to the script?
  • but what if different scripts have the same name? add junk to uniqiify?

design goal:

• instead of putting this script in the path, it should also be sufficiently literally to append it to any python script

issues:

• overriding existing defintiions of main
• collisions on globals namespace

page defining location: macOS Library Directory Details

options:

• $HOME/Library/Caches/python-script/
• $HOME/Library/Developer/python-script/

• https://rust-script.org/
• https://github.com/mxcl/swift-sh/tree/master/Examples
• scriptisto (more general, but more opaque)

Options:

1. embed environment.yml
   • works with -n if environment.yml does not specify the env name!
   • works with -prefix if environment.yml does not specify the env name!

1. embed whatever format conda install --file expects
   • that format is also what conda install expectsspecification?apparently yes.
   • that format is the install specification

conda install docs note that with –file conda intsll will “Read package versions from the given file.”

packaging

https://packaging.python.org/en/latest/guides/writing-pyproject-toml/#writing-pyproject-toml

single-sourcing version

macOS Sonoma ships with Python 3.9.6 and with setuptools 58.04

setuptools can be configured with pyproject.toml only as setuptools 61.0.0

So:

• must not rely on pyproject.toml
• can use a setup.cfg

twine upload to PyPi

. ├── pyproject.toml ├── python_script │ ├── __init__.py │ ├── __main__.py │ └── python_script.py

properties:

• pro: $ python3 -m python_script
• pro: $ python-script
• con: then python_script.py can’t just go into the path!

Conflict:

• copy-file installability is good, and requires files be named like CL tools
• pip-installability is good
• but pip requires modules to have bad names for a CL tool

Options:

1. duplicate the file, python-script AND python_script??
2. use exec to load a file without the py extension?

One might think there was an option 3, to just use a Python-compliant name also for the CLI, by calling it python_script.py or pythonscript.py. But this seems not to work. The python3 -m builds a shim. When that shim runs, it does not find the package named python_script. Not clear why. In other words, there is a defect in the python3 -m run mechanism such that you cannot name one of your project entry point scripts “foo.py” if the import name of your package is “foo”, since then

. ├── pyproject.toml ├── python_script │ ├── __init__.py │ ├── __main__.py │ └── python_script.py

names:

• pythonscript
• python-script
• pythonscript.py
• python_script.py
• pyrun
• pyrunscript
• python-script.py
• pyscript:
  • no used
• pydepscript
• depscript
  • no. no mention of python

The project name “pythonscript” was rejected.

presumably because of the existing project PythonScript1, a dead stub of a project. collision policy

will try pythonrunscript

An official spec for embedded deps: https://packaging.python.org/en/latest/specifications/inline-script-metadata/#inline-script-metadata

PEP: https://peps.python.org/pep-0723/

Noted re: inline script metadata:

one example for a tool called “some-toml”

ANother example for the tool called “script”:

Possible use of this syntax: add support for doc comments with these tools:

• pythonrunscript-requirements.txt
• pythonrunscript-conda_install_specs.txt

current parser works by:

• defining 3 begin patterns, 1 end pattern, and 1 data pattern
• defining 4 parser patterns (3 in-block, 1 out-of-block)
• elif-based pattern matching on (line_type, match obj)

notable: https://pip.wtf/

• PRO. lightweight, comprehensible. Very good. <10 LOC.
• CON. semi-nonstandard syntax for specifying deps
• CON. Only pip, no conda
• CON. installs and uses dedicated env, unconditionally, and even if current env suffices
  • inefficient.
  • interferes with other workflows, like developing the script in an IDE-managed env

Evolution?:

• only install deps if needed and confirmed?
  • q: does pip itself already providing UI prompting?
  • q: how to check deps?
    • also with pip

• [ ] Q: how to convert all version specs valid in script TOML, to specs valid in conda python specs and requirementx.txt dep specs?
• exact version format required in PEP723 TOML?

  for deps, Uses https://packaging.python.org/en/latest/specifications/dependency-specifiers/#dependency-specifiers

  for python, uses https://packaging.python.org/en/latest/specifications/version-specifiers/#version-specifiers

• exact version format required in requirements.txt?

  requirements.txt ALLOWs use of requirements specifiers: https://pip.pypa.io/en/stable/reference/requirements-file-format/

  https://pip.pypa.io/en/stable/reference/requirement-specifiers/#requirement-specifiers

  “Generally speaking, a requirement specifier is composed of a project name followed by optional version specifiers.”

• exact version format required in a conda spec?

• [X] update README.
• [X] update CLI docs
• [X] update dry run mode
• [ ] verify that TOML syntax for python version and pip versions matches conda install spec and reqs.txt syntax
• [X] enhance fork into environment with components installed through pip and conda

run_ffmpeg:

• run as executable, triggering pythonrunscript.py:
  • BROKEN
  • incorrectly tries to interpret the file not as python?
  • does not interpreter just as if called by the python3 in the env
    • but: calling conda run -p /Users/alexis/Library/Caches/pythonrunscript/60b45f611c8f43cc4df61e1bd5157fd8/condaenv python3 run_ffmpeg.py works
• run from python3:
  • works: runs as desired, as if ffmpeg -version was called directly
• run from pythonrunscript.py:
  • works: runs as desired, as if ffmpeg -version was called directly

run_conda.py:

• run from python3:
• run as executable:

• what behavior is desired?
• does workaround improve behavior?

NO WRAPPER:

test of ffmpeg:

• ffmpeg:
  • exits, with code 1
• ./run_ffmpeg:
  • exits with code 1, and a message emitted by conda saying:
  • BAD: discrepancy. conda inserts a message
• pythonrunscript.py run_ffmpeg.py:
  • exits with code 1
  • BAD: and a message emitted by conda saying. (same)

test of ffmpeg -version:

• ffmpeg -version:
  • exits, with code 0
• ./run_ffmpeg -version:
  • exits, with code 0
  • BAD: conda intercepts the command line argument.
• pythonrunscript.py run_ffmpeg.py -version:
  • BAD: conda intercepts the command line argument.

test of python3:

• ./python3:
  • opens interactive prompt
• ./run_python3.py:
  • VERY VERY BAD: exits immediately
• pythonrunscript ./run_python3.py:
  • without wrapper: opens interactive prompt
  • with wrapper:
• python3 ./run_python3.py:
  • opens interactive prompt

test of python3 -V:

• ./python3 -V:
  • works
• ./run_python3.py -V:
  • works
• pythonrunscript ./run_python3.py -V:
  • works

WITH WRAPPER:

• Finding: wrapper SUCCEEDS in fixing the problem that with -version.
• wrapper does not prevent interpreter for quitting rather than opening a terminal
• but wrapper did not cause that problem

summary of non-transparencies:

• conda run inserts a message when exit code == 1
• conda intercepts command line arguments like “-version”
  • fixed with wrapper
• conda fails to run an interactive prommpt
  • fixed with command line arg to conda –no-capture-output

ERROR conda.cli.main_run:execute(125): `conda run /Users/alexis/Library/Caches/pythonrunscript/60b45f611c8f43cc4df61e1bd5157fd8/condaenv/bin/python3 ./run_ffmpeg.py` failed. (See above for error)

• scripts can now call conda-installed executable
• worked around a known conda bug
• one non-transparency, is that if pythonrunscript is used to run a script in a conda environment, which exits with a nonzero exit code, then conda will print some additional logging at exit time.

next steps:

• [X] re-run examples as tests
• [ ] add meatier examples?
• [X] update versioning metadata
• [x] add versioning to CLT
• [X] copy script to binary
• [ ] deploy to pypi

poetry build poetry publish