Merge pull request #132473 from teto/luarocks-pyupdater

Luarocks pyupdater
This commit is contained in:
Matthieu Coudron 2021-08-09 22:49:10 +02:00 committed by GitHub
commit 9578494089
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
7 changed files with 656 additions and 472 deletions

View file

@ -139,11 +139,9 @@ the whitelist maintainers/scripts/luarocks-packages.csv and updated by running m
[luarocks2nix](https://github.com/nix-community/luarocks) is a tool capable of generating nix derivations from both rockspec and src.rock (and favors the src.rock). [luarocks2nix](https://github.com/nix-community/luarocks) is a tool capable of generating nix derivations from both rockspec and src.rock (and favors the src.rock).
The automation only goes so far though and some packages need to be customized. The automation only goes so far though and some packages need to be customized.
These customizations go in `pkgs/development/lua-modules/overrides.nix`. These customizations go in `pkgs/development/lua-modules/overrides.nix`.
For instance if the rockspec defines `external_dependencies`, these need to be manually added in its rockspec file then it won't work. For instance if the rockspec defines `external_dependencies`, these need to be manually added to the overrides.nix.
You can try converting luarocks packages to nix packages with the command `nix-shell -p luarocks-nix` and then `luarocks nix PKG_NAME`. You can try converting luarocks packages to nix packages with the command `nix-shell -p luarocks-nix` and then `luarocks nix PKG_NAME`.
Nix rely on luarocks to install lua packages, basically it runs:
`luarocks make --deps-mode=none --tree $out`
#### Packaging a library manually {#packaging-a-library-manually} #### Packaging a library manually {#packaging-a-library-manually}
@ -161,8 +159,8 @@ are not packaged for luarocks. You can see a few examples at `pkgs/top-level/lua
### Lua interpreters {#lua-interpreters} ### Lua interpreters {#lua-interpreters}
Versions 5.1, 5.2 and 5.3 of the lua interpreter are available as Versions 5.1, 5.2, 5.3 and 5.4 of the lua interpreter are available as
respectively `lua5_1`, `lua5_2` and `lua5_3`. Luajit is available too. respectively `lua5_1`, `lua5_2`, `lua5_3` and `lua5_4`. Luajit is available too.
The Nix expressions for the interpreters can be found in `pkgs/development/interpreters/lua-5`. The Nix expressions for the interpreters can be found in `pkgs/development/interpreters/lua-5`.
#### Attributes on lua interpreters packages {#attributes-on-lua-interpreters-packages} #### Attributes on lua interpreters packages {#attributes-on-lua-interpreters-packages}

View file

@ -309,7 +309,7 @@ Sample output2:
## Adding new plugins to nixpkgs {#adding-new-plugins-to-nixpkgs} ## Adding new plugins to nixpkgs {#adding-new-plugins-to-nixpkgs}
Nix expressions for Vim plugins are stored in [pkgs/misc/vim-plugins](/pkgs/misc/vim-plugins). For the vast majority of plugins, Nix expressions are automatically generated by running [`./update.py`](/pkgs/misc/vim-plugins/update.py). This creates a [generated.nix](/pkgs/misc/vim-plugins/generated.nix) file based on the plugins listed in [vim-plugin-names](/pkgs/misc/vim-plugins/vim-plugin-names). Plugins are listed in alphabetical order in `vim-plugin-names` using the format `[github username]/[repository]`. For example https://github.com/scrooloose/nerdtree becomes `scrooloose/nerdtree`. Nix expressions for Vim plugins are stored in [pkgs/misc/vim-plugins](/pkgs/misc/vim-plugins). For the vast majority of plugins, Nix expressions are automatically generated by running [`./update.py`](/pkgs/misc/vim-plugins/update.py). This creates a [generated.nix](/pkgs/misc/vim-plugins/generated.nix) file based on the plugins listed in [vim-plugin-names](/pkgs/misc/vim-plugins/vim-plugin-names). Plugins are listed in alphabetical order in `vim-plugin-names` using the format `[github username]/[repository]@[gitref]`. For example https://github.com/scrooloose/nerdtree becomes `scrooloose/nerdtree`.
Some plugins require overrides in order to function properly. Overrides are placed in [overrides.nix](/pkgs/misc/vim-plugins/overrides.nix). Overrides are most often required when a plugin requires some dependencies, or extra steps are required during the build process. For example `deoplete-fish` requires both `deoplete-nvim` and `vim-fish`, and so the following override was added: Some plugins require overrides in order to function properly. Overrides are placed in [overrides.nix](/pkgs/misc/vim-plugins/overrides.nix). Overrides are most often required when a plugin requires some dependencies, or extra steps are required during the build process. For example `deoplete-fish` requires both `deoplete-nvim` and `vim-fish`, and so the following override was added:

View file

@ -1,89 +1,89 @@
# nix name,luarocks name,server,version,luaversion,maintainers name,server,version,luaversion,maintainers
alt-getopt,,,,,arobyn alt-getopt,,,,arobyn
ansicolors,,,,, ansicolors,,,,
argparse,,,,, bit32,,5.3.0-1,lua5_1,lblasc
basexx,,,,, argparse,,,,
binaryheap,,,,,vcunat basexx,,,,
bit32,,,,lua5_1,lblasc binaryheap,,,,vcunat
busted,,,,, busted,,,,
cassowary,,,,,marsam alerque cassowary,,,,marsam alerque
cjson,lua-cjson,,,, compat53,,0.7-1,,vcunat
compat53,,,,,vcunat cosmo,,,,marsam
cosmo,,,,,marsam coxpcall,,1.17.0-1,,
coxpcall,,,1.17.0-1,, cqueues,,,,vcunat
cqueues,,,,,vcunat cyrussasl,,,,
cyrussasl,,,,, digestif,,0.2-1,lua5_3,
digestif,,,,lua5_3, dkjson,,,,
dkjson,,,,, fifo,,,,
fifo,,,,, gitsigns.nvim,,,lua5_1,
gitsigns.nvim,,,,lua5_1, http,,0.3-0,,vcunat
http,,,,,vcunat inspect,,,,
inspect,,,,, ldbus,http://luarocks.org/dev,,,
ldbus,,http://luarocks.org/dev,,, ldoc,,,,
ldoc,,,,, lgi,,,,
lgi,,,,, linenoise,,,,
linenoise,,,,, ljsyscall,,,lua5_1,lblasc
ljsyscall,,,,lua5_1,lblasc lpeg,,,,vyp
lpeg,,,,,vyp lpeg_patterns,,,,
lpeg_patterns,,,,, lpeglabel,,,,
lpeglabel,,,,, lpty,,,,
lpty,,,,, lrexlib-gnu,,,,
lrexlib-gnu,,,,, lrexlib-pcre,,,,vyp
lrexlib-pcre,,,,,vyp lrexlib-posix,,,,
lrexlib-posix,,,,, ltermbox,,,,
ltermbox,,,,, lua-cjson,,,,
lua-cmsgpack,,,,, lua-cmsgpack,,,,
lua-iconv,,,,, lua-iconv,,,,
lua-lsp,,http://luarocks.org/dev,,, lua-lsp,http://luarocks.org/dev,,,
lua-messagepack,,,,, lua-messagepack,,,,
lua-resty-http,,,,, lua-resty-http,,,,
lua-resty-jwt,,,,, lua-resty-jwt,,,,
lua-resty-openidc,,,,, lua-resty-openidc,,,,
lua-resty-openssl,,,,, lua-resty-openssl,,,,
lua-resty-session,,,,, lua-resty-session,,,,
lua-term,,,,, lua-term,,,,
lua-toml,,,,, lua-toml,,,,
lua-zlib,,,,,koral lua-zlib,,,,koral
lua_cliargs,,,,, lua_cliargs,,,,
luabitop,,,,, luabitop,,,,
luacheck,,,,, luacheck,,,,
luacov,,,,, luacov,,,,
luadbi,,,,, luadbi,,,,
luadbi-mysql,,,,, luadbi-mysql,,,,
luadbi-postgresql,,,,, luadbi-postgresql,,,,
luadbi-sqlite3,,,,, luadbi-sqlite3,,,,
luadoc,,,,, luadoc,,,,
luaepnf,,,,, luaepnf,,,,
luaevent,,,,, luaevent,,,,
luaexpat,,,1.3.0-1,,arobyn flosse luaexpat,,1.3.0-1,,arobyn flosse
luaffi,,http://luarocks.org/dev,,, luaffi,http://luarocks.org/dev,,,
luafilesystem,,,1.7.0-2,,flosse luafilesystem,,1.7.0-2,,flosse
lualogging,,,,, lualogging,,,,
luaossl,,,,lua5_1, luaossl,,,lua5_1,
luaposix,,,,,vyp lblasc luaposix,,34.1.1-1,,vyp lblasc
luarepl,,,,, luarepl,,,,
luasec,,,,,flosse luasec,,,,flosse
luasocket,,,,, luasocket,,,,
luasql-sqlite3,,,,,vyp luasql-sqlite3,,,,vyp
luassert,,,,, luassert,,,,
luasystem,,,,, luasystem,,,,
luautf8,,,,,pstn luautf8,,,,pstn
luazip,,,,, luazip,,,,
lua-yajl,,,,,pstn lua-yajl,,,,pstn
luuid,,,,, luuid,,,,
luv,,,,, luv,,1.30.0-0,,
lyaml,,,,,lblasc lyaml,,,,lblasc
markdown,,,,, markdown,,,,
mediator_lua,,,,, mediator_lua,,,,
mpack,,,,, mpack,,,,
moonscript,,,,,arobyn moonscript,,,,arobyn
nvim-client,,,,, nvim-client,,,,
penlight,,,,, penlight,,,,
plenary.nvim,,,,lua5_1, plenary.nvim,,,lua5_1,
rapidjson,,,,, rapidjson,,,,
readline,,,,, readline,,,,
say,,,,, say,,,,
std-_debug,std._debug,,,, std._debug,,,,
std_normalize,std.normalize,,,, std.normalize,,,,
stdlib,,,,,vyp stdlib,,,,vyp
vstruct,,,,, vstruct,,,,

1 # nix name name luarocks name server version luaversion maintainers
2 alt-getopt alt-getopt arobyn
3 ansicolors ansicolors
4 argparse bit32 5.3.0-1 lua5_1 lblasc
5 basexx argparse
6 binaryheap basexx vcunat
7 bit32 binaryheap lua5_1 lblasc vcunat
8 busted busted
9 cassowary cassowary marsam alerque
10 cjson compat53 lua-cjson 0.7-1 vcunat
11 compat53 cosmo vcunat marsam
12 cosmo coxpcall 1.17.0-1 marsam
13 coxpcall cqueues 1.17.0-1 vcunat
14 cqueues cyrussasl vcunat
15 cyrussasl digestif 0.2-1 lua5_3
16 digestif dkjson lua5_3
17 dkjson fifo
18 fifo gitsigns.nvim lua5_1
19 gitsigns.nvim http 0.3-0 lua5_1 vcunat
20 http inspect vcunat
21 inspect ldbus http://luarocks.org/dev
22 ldbus ldoc http://luarocks.org/dev
23 ldoc lgi
24 lgi linenoise
25 linenoise ljsyscall lua5_1 lblasc
26 ljsyscall lpeg lua5_1 lblasc vyp
27 lpeg lpeg_patterns vyp
28 lpeg_patterns lpeglabel
29 lpeglabel lpty
30 lpty lrexlib-gnu
31 lrexlib-gnu lrexlib-pcre vyp
32 lrexlib-pcre lrexlib-posix vyp
33 lrexlib-posix ltermbox
34 ltermbox lua-cjson
35 lua-cmsgpack lua-cmsgpack
36 lua-iconv lua-iconv
37 lua-lsp lua-lsp http://luarocks.org/dev
38 lua-messagepack lua-messagepack
39 lua-resty-http lua-resty-http
40 lua-resty-jwt lua-resty-jwt
41 lua-resty-openidc lua-resty-openidc
42 lua-resty-openssl lua-resty-openssl
43 lua-resty-session lua-resty-session
44 lua-term lua-term
45 lua-toml lua-toml
46 lua-zlib lua-zlib koral
47 lua_cliargs lua_cliargs
48 luabitop luabitop
49 luacheck luacheck
50 luacov luacov
51 luadbi luadbi
52 luadbi-mysql luadbi-mysql
53 luadbi-postgresql luadbi-postgresql
54 luadbi-sqlite3 luadbi-sqlite3
55 luadoc luadoc
56 luaepnf luaepnf
57 luaevent luaevent
58 luaexpat luaexpat 1.3.0-1 arobyn flosse
59 luaffi luaffi http://luarocks.org/dev
60 luafilesystem luafilesystem 1.7.0-2 flosse
61 lualogging lualogging
62 luaossl luaossl lua5_1
63 luaposix luaposix 34.1.1-1 vyp lblasc
64 luarepl luarepl
65 luasec luasec flosse
66 luasocket luasocket
67 luasql-sqlite3 luasql-sqlite3 vyp
68 luassert luassert
69 luasystem luasystem
70 luautf8 luautf8 pstn
71 luazip luazip
72 lua-yajl lua-yajl pstn
73 luuid luuid
74 luv luv 1.30.0-0
75 lyaml lyaml lblasc
76 markdown markdown
77 mediator_lua mediator_lua
78 mpack mpack
79 moonscript moonscript arobyn
80 nvim-client nvim-client
81 penlight penlight
82 plenary.nvim plenary.nvim lua5_1
83 rapidjson rapidjson
84 readline readline
85 say say
86 std-_debug std._debug std._debug
87 std_normalize std.normalize std.normalize
88 stdlib stdlib vyp
89 vstruct vstruct

View file

@ -28,6 +28,7 @@ from pathlib import Path
from typing import Dict, List, Optional, Tuple, Union, Any, Callable from typing import Dict, List, Optional, Tuple, Union, Any, Callable
from urllib.parse import urljoin, urlparse from urllib.parse import urljoin, urlparse
from tempfile import NamedTemporaryFile from tempfile import NamedTemporaryFile
from dataclasses import dataclass
import git import git
@ -82,6 +83,13 @@ def make_request(url: str) -> urllib.request.Request:
headers["Authorization"] = f"token {token}" headers["Authorization"] = f"token {token}"
return urllib.request.Request(url, headers=headers) return urllib.request.Request(url, headers=headers)
@dataclass
class PluginDesc:
owner: str
repo: str
branch: str
alias: str
class Repo: class Repo:
def __init__( def __init__(
@ -201,15 +209,39 @@ class Editor:
deprecated: Optional[Path] = None, deprecated: Optional[Path] = None,
cache_file: Optional[str] = None, cache_file: Optional[str] = None,
): ):
log.debug("get_plugins:", get_plugins)
self.name = name self.name = name
self.root = root self.root = root
self.get_plugins = get_plugins self.get_plugins = get_plugins
self.generate_nix = generate_nix self._generate_nix = generate_nix
self.default_in = default_in or root.joinpath(f"{name}-plugin-names") self.default_in = default_in or root.joinpath(f"{name}-plugin-names")
self.default_out = default_out or root.joinpath("generated.nix") self.default_out = default_out or root.joinpath("generated.nix")
self.deprecated = deprecated or root.joinpath("deprecated.json") self.deprecated = deprecated or root.joinpath("deprecated.json")
self.cache_file = cache_file or f"{name}-plugin-cache.json" self.cache_file = cache_file or f"{name}-plugin-cache.json"
def get_current_plugins(self):
"""To fill the cache"""
return get_current_plugins(self)
def load_plugin_spec(self, plugin_file) -> List[PluginDesc]:
return load_plugin_spec(plugin_file)
def generate_nix(self, plugins, outfile):
'''Returns nothing for now, writes directly to outfile'''
self._generate_nix(plugins, outfile)
def get_update(self, input_file: str, outfile: str, proc: int):
return get_update(input_file, outfile, proc, editor=self)
@property
def attr_path(self):
return self.name + "Plugins"
def rewrite_input(self, *args, **kwargs):
return rewrite_input(*args, **kwargs)
class CleanEnvironment(object): class CleanEnvironment(object):
def __enter__(self) -> None: def __enter__(self) -> None:
@ -228,7 +260,9 @@ class CleanEnvironment(object):
def get_current_plugins(editor: Editor) -> List[Plugin]: def get_current_plugins(editor: Editor) -> List[Plugin]:
with CleanEnvironment(): with CleanEnvironment():
out = subprocess.check_output(["nix", "eval", "--json", editor.get_plugins]) cmd = ["nix", "eval", "--json", editor.get_plugins]
log.debug("Running command %s", cmd)
out = subprocess.check_output(cmd)
data = json.loads(out) data = json.loads(out)
plugins = [] plugins = []
for name, attr in data.items(): for name, attr in data.items():
@ -244,12 +278,13 @@ def prefetch_plugin(
alias: Optional[str], alias: Optional[str],
cache: "Optional[Cache]" = None, cache: "Optional[Cache]" = None,
) -> Tuple[Plugin, Dict[str, str]]: ) -> Tuple[Plugin, Dict[str, str]]:
log.info("Prefetching plugin %s", repo_name) log.info(f"Fetching last commit for plugin {user}/{repo_name}@{branch}")
repo = Repo(user, repo_name, branch, alias) repo = Repo(user, repo_name, branch, alias)
commit, date = repo.latest_commit() commit, date = repo.latest_commit()
has_submodules = repo.has_submodules() has_submodules = repo.has_submodules()
cached_plugin = cache[commit] if cache else None cached_plugin = cache[commit] if cache else None
if cached_plugin is not None: if cached_plugin is not None:
log.debug("Cache hit !")
cached_plugin.name = alias or repo_name cached_plugin.name = alias or repo_name
cached_plugin.date = date cached_plugin.date = date
return cached_plugin, repo.redirect return cached_plugin, repo.redirect
@ -306,8 +341,7 @@ def check_results(
sys.exit(1) sys.exit(1)
def parse_plugin_line(line: str) -> PluginDesc:
def parse_plugin_line(line: str) -> Tuple[str, str, str, Optional[str]]:
branch = "master" branch = "master"
alias = None alias = None
name, repo = line.split("/") name, repo = line.split("/")
@ -317,15 +351,15 @@ def parse_plugin_line(line: str) -> Tuple[str, str, str, Optional[str]]:
if "@" in repo: if "@" in repo:
repo, branch = repo.split("@") repo, branch = repo.split("@")
return (name.strip(), repo.strip(), branch.strip(), alias) return PluginDesc(name.strip(), repo.strip(), branch.strip(), alias)
def load_plugin_spec(plugin_file: str) -> List[Tuple[str, str, str, Optional[str]]]: def load_plugin_spec(plugin_file: str) -> List[PluginDesc]:
plugins = [] plugins = []
with open(plugin_file) as f: with open(plugin_file) as f:
for line in f: for line in f:
plugin = parse_plugin_line(line) plugin = parse_plugin_line(line)
if not plugin[0]: if not plugin.owner:
msg = f"Invalid repository {line}, must be in the format owner/repo[ as alias]" msg = f"Invalid repository {line}, must be in the format owner/repo[ as alias]"
print(msg, file=sys.stderr) print(msg, file=sys.stderr)
sys.exit(1) sys.exit(1)
@ -387,12 +421,11 @@ class Cache:
def prefetch( def prefetch(
args: Tuple[str, str, str, Optional[str]], cache: Cache args: PluginDesc, cache: Cache
) -> Tuple[str, str, Union[Exception, Plugin], dict]: ) -> Tuple[str, str, Union[Exception, Plugin], dict]:
assert len(args) == 4 owner, repo = args.owner, args.repo
owner, repo, branch, alias = args
try: try:
plugin, redirect = prefetch_plugin(owner, repo, branch, alias, cache) plugin, redirect = prefetch_plugin(owner, repo, args.branch, args.alias, cache)
cache[plugin.commit] = plugin cache[plugin.commit] = plugin
return (owner, repo, plugin, redirect) return (owner, repo, plugin, redirect)
except Exception as e: except Exception as e:
@ -433,7 +466,7 @@ def rewrite_input(
with open(input_file, "w") as f: with open(input_file, "w") as f:
f.writelines(lines) f.writelines(lines)
# TODO move to Editor ?
def parse_args(editor: Editor): def parse_args(editor: Editor):
parser = argparse.ArgumentParser( parser = argparse.ArgumentParser(
description=( description=(
@ -446,7 +479,7 @@ def parse_args(editor: Editor):
dest="add_plugins", dest="add_plugins",
default=[], default=[],
action="append", action="append",
help=f"Plugin to add to {editor.name}Plugins from Github in the form owner/repo", help=f"Plugin to add to {editor.attr_path} from Github in the form owner/repo",
) )
parser.add_argument( parser.add_argument(
"--input-names", "--input-names",
@ -493,11 +526,11 @@ def commit(repo: git.Repo, message: str, files: List[Path]) -> None:
def get_update(input_file: str, outfile: str, proc: int, editor: Editor): def get_update(input_file: str, outfile: str, proc: int, editor: Editor):
cache: Cache = Cache(get_current_plugins(editor), editor.cache_file) cache: Cache = Cache(editor.get_current_plugins(), editor.cache_file)
_prefetch = functools.partial(prefetch, cache=cache) _prefetch = functools.partial(prefetch, cache=cache)
def update() -> dict: def update() -> dict:
plugin_names = load_plugin_spec(input_file) plugin_names = editor.load_plugin_spec(input_file)
try: try:
pool = Pool(processes=proc) pool = Pool(processes=proc)
@ -522,33 +555,33 @@ def update_plugins(editor: Editor):
log.info("Start updating plugins") log.info("Start updating plugins")
nixpkgs_repo = git.Repo(editor.root, search_parent_directories=True) nixpkgs_repo = git.Repo(editor.root, search_parent_directories=True)
update = get_update(args.input_file, args.outfile, args.proc, editor) update = editor.get_update(args.input_file, args.outfile, args.proc)
redirects = update() redirects = update()
rewrite_input(args.input_file, editor.deprecated, redirects) editor.rewrite_input(args.input_file, editor.deprecated, redirects)
autocommit = not args.no_commit autocommit = not args.no_commit
if autocommit: if autocommit:
commit(nixpkgs_repo, f"{editor.name}Plugins: update", [args.outfile]) commit(nixpkgs_repo, f"{editor.attr_path}: update", [args.outfile])
if redirects: if redirects:
update() update()
if autocommit: if autocommit:
commit( commit(
nixpkgs_repo, nixpkgs_repo,
f"{editor.name}Plugins: resolve github repository redirects", f"{editor.attr_path}: resolve github repository redirects",
[args.outfile, args.input_file, editor.deprecated], [args.outfile, args.input_file, editor.deprecated],
) )
for plugin_line in args.add_plugins: for plugin_line in args.add_plugins:
rewrite_input(args.input_file, editor.deprecated, append=(plugin_line + "\n",)) editor.rewrite_input(args.input_file, editor.deprecated, append=(plugin_line + "\n",))
update() update()
plugin = fetch_plugin_from_pluginline(plugin_line) plugin = fetch_plugin_from_pluginline(plugin_line)
if autocommit: if autocommit:
commit( commit(
nixpkgs_repo, nixpkgs_repo,
"{editor}Plugins.{name}: init at {version}".format( "{editor.attr_path}.{name}: init at {version}".format(
editor=editor.name, name=plugin.normalized_name, version=plugin.version editor=editor.name, name=plugin.normalized_name, version=plugin.version
), ),
[args.outfile, args.input_file], [args.outfile, args.input_file],

View file

@ -1,140 +1,179 @@
#!/usr/bin/env nix-shell #!/usr/bin/env nix-shell
#!nix-shell update-luarocks-shell.nix -i bash #!nix-shell -p nix-prefetch-git luarocks-nix python3 python3Packages.GitPython nix -i python3
# You'll likely want to use # format:
# `` # $ nix run nixpkgs.python3Packages.black -c black update.py
# nixpkgs $ maintainers/scripts/update-luarocks-packages pkgs/development/lua-modules/generated-packages.nix # type-check:
# `` # $ nix run nixpkgs.python3Packages.mypy -c mypy update.py
# to update all libraries in that folder. # linted:
# to debug, redirect stderr to stdout with 2>&1 # $ nix run nixpkgs.python3Packages.flake8 -c flake8 --ignore E501,E265,E402 update.py
# stop the script upon C-C import inspect
set -eu -o pipefail import os
import tempfile
import shutil
from dataclasses import dataclass
import subprocess
import csv
import logging
CSV_FILE="maintainers/scripts/luarocks-packages.csv" from typing import List
from pathlib import Path
LOG_LEVELS = {
logging.getLevelName(level): level for level in [
logging.DEBUG, logging.INFO, logging.WARN, logging.ERROR ]
}
log = logging.getLogger()
log.addHandler(logging.StreamHandler())
ROOT = Path(os.path.dirname(os.path.abspath(inspect.getfile(inspect.currentframe())))).parent.parent
from pluginupdate import Editor, parse_args, update_plugins, PluginDesc, CleanEnvironment
PKG_LIST="maintainers/scripts/luarocks-packages.csv"
TMP_FILE="$(mktemp)" TMP_FILE="$(mktemp)"
# Set in the update-luarocks-shell.nix GENERATED_NIXFILE="pkgs/development/lua-modules/generated-packages.nix"
NIXPKGS_PATH="$LUAROCKS_NIXPKGS_PATH" LUAROCKS_CONFIG="$NIXPKGS_PATH/maintainers/scripts/luarocks-config.lua"
export LUAROCKS_CONFIG="$NIXPKGS_PATH/maintainers/scripts/luarocks-config.lua"
# 10 is a pretty arbitrary number of simultaneous jobs, but it is generally HEADER = """
# impolite to hit a webserver with *too* many simultaneous connections :) /* {GENERATED_NIXFILE} is an auto-generated file -- DO NOT EDIT!
PARALLEL_JOBS=1
exit_trap() {
local lc="$BASH_COMMAND" rc=$?
test $rc -eq 0 || echo -e "*** error $rc: $lc.\nGenerated temporary file in $TMP_FILE" >&2
}
print_help() {
echo "Usage: $0 <GENERATED_FILE>"
echo "(most likely pkgs/development/lua-modules/generated-packages.nix)"
echo ""
echo " -c <CSV_FILE> to set the list of luarocks package to generate"
exit 1
}
if [ $# -lt 1 ]; then
print_help
exit 1
fi
trap exit_trap EXIT
while getopts ":hc:" opt; do
case $opt in
h)
print_help
;;
c)
echo "Loading package list from $OPTARG !" >&2
CSV_FILE="$OPTARG"
;;
\?)
echo "Invalid option: -$OPTARG" >&2
;;
esac
shift $((OPTIND - 1))
done
GENERATED_NIXFILE="$1"
HEADER="
/* ${GENERATED_NIXFILE} is an auto-generated file -- DO NOT EDIT!
Regenerate it with: Regenerate it with:
nixpkgs$ ${0} ${GENERATED_NIXFILE} nixpkgs$ ./maintainers/scripts/update-luarocks-packages
These packages are manually refined in lua-overrides.nix You can customize the generated packages in pkgs/development/lua-modules/overrides.nix
*/ */
{ self, stdenv, lib, fetchurl, fetchgit, pkgs, ... } @ args: """.format(GENERATED_NIXFILE=GENERATED_NIXFILE)
FOOTER="""
}
/* GENERATED - do not edit this file */
"""
@dataclass
class LuaPlugin:
name: str
version: str
server: str
luaversion: str
maintainers: str
@property
def normalized_name(self) -> str:
return self.name.replace(".", "-")
# rename Editor to LangUpdate/ EcosystemUpdater
class LuaEditor(Editor):
def get_current_plugins(self):
return []
def load_plugin_spec(self, input_file) -> List[PluginDesc]:
luaPackages = []
csvfilename=input_file
log.info("Loading package descriptions from %s", csvfilename)
with open(csvfilename, newline='') as csvfile:
reader = csv.DictReader(csvfile,)
for row in reader:
# name,server,version,luaversion,maintainers
plugin = LuaPlugin(**row)
luaPackages.append(plugin)
return luaPackages
@property
def attr_path(self):
return "luaPackages"
def get_update(self, input_file: str, outfile: str, _: int):
def update() -> dict:
plugin_specs = self.load_plugin_spec(input_file)
self.generate_nix(plugin_specs, outfile)
redirects = []
return redirects
return update
def rewrite_input(self, *args, **kwargs):
# not implemented yet
pass
def generate_nix(
plugins: List[LuaPlugin],
outfilename: str
):
sorted_plugins = sorted(plugins, key=lambda v: v.name.lower())
# plug = {}
# selon le manifest luarocks.org/manifest
def _generate_pkg_nix(plug):
cmd = [ "luarocks", "nix", plug.name]
if plug.server:
cmd.append(f"--only-server={plug.server}")
if plug.maintainers:
cmd.append(f"--maintainers={plug.maintainers}")
if plug.version:
cmd.append(plug.version)
if plug.luaversion:
with CleanEnvironment():
local_pkgs = str(ROOT.resolve())
cmd2 = ["nix-build", "--no-out-link", local_pkgs, "-A", f"{plug.luaversion}"]
log.debug("running %s", cmd2)
lua_drv_path=subprocess.check_output(cmd2, text=True).strip()
cmd.append(f"--lua-dir={lua_drv_path}/bin")
log.debug("running %s", cmd)
output = subprocess.check_output(cmd, text=True)
return output
with tempfile.NamedTemporaryFile("w+") as f:
f.write(HEADER)
f.write("""
{ self, stdenv, lib, fetchurl, fetchgit, ... } @ args:
self: super: self: super:
with self; with self;
{ {
" """)
FOOTER=" for plugin in sorted_plugins:
}
/* GENERATED */
"
function convert_pkg() { nix_expr = _generate_pkg_nix(plugin)
nix_pkg_name="$1" f.write(f"{plugin.normalized_name} = {nix_expr}"
lua_pkg_name="$2" )
server="$3" f.write(FOOTER)
pkg_version="$4" f.flush()
lua_version="$5"
maintainers="$6"
if [ "${nix_pkg_name:0:1}" == "#" ]; then # if everything went fine, move the generated file to its destination
echo "Skipping comment ${*}" >&2 # using copy since move doesn't work across disks
return shutil.copy(f.name, outfilename)
fi
# Normalize package name print(f"updated {outfilename}")
nix_pkg_name_normalized=$(sed 's/\./-/' <(echo "$nix_pkg_name"))
if [ -z "$lua_pkg_name" ]; then def load_plugin_spec():
echo "Using nix_name as lua_pkg_name for '$nix_pkg_name'" >&2 pass
lua_pkg_name="$nix_pkg_name"
fi
echo "Building expression for $lua_pkg_name (version $pkg_version) from server [$server]" >&2
luarocks_args=(nix)
if [[ -n $server ]]; then
luarocks_args+=("--only-server=$server")
fi
if [[ -n $maintainers ]]; then
luarocks_args+=("--maintainers=$maintainers")
fi
if [[ -n $lua_version ]]; then
lua_drv_path=$(nix-build --no-out-link "$NIXPKGS_PATH" -A "$lua_version")
luarocks_args+=("--lua-dir=$lua_drv_path/bin")
fi
luarocks_args+=("$lua_pkg_name")
if [[ -n $pkg_version ]]; then
luarocks_args+=("$pkg_version")
fi
echo "Running 'luarocks ${luarocks_args[*]}'" >&2
if drv="$nix_pkg_name_normalized = $(luarocks "${luarocks_args[@]}")"; then
echo "$drv"
else
echo "Failed to convert $nix_pkg_name" >&2
return 1
fi
}
# params needed when called via callPackage def main():
echo "$HEADER" | tee "$TMP_FILE"
# Ensure parallel can run our bash function editor = LuaEditor("lua", ROOT, '', generate_nix,
export -f convert_pkg default_in = ROOT.joinpath(PKG_LIST),
export SHELL=bash default_out = ROOT.joinpath(GENERATED_NIXFILE)
# Read each line in the csv file and run convert_pkg for each, in parallel )
parallel --group --keep-order --halt now,fail=1 --jobs "$PARALLEL_JOBS" --colsep ',' convert_pkg {} <"$CSV_FILE" | tee -a "$TMP_FILE"
# close the set args = parse_args(editor)
echo "$FOOTER" | tee -a "$TMP_FILE" log.setLevel(LOG_LEVELS[args.debug])
cp "$TMP_FILE" "$GENERATED_NIXFILE" update_plugins(editor)
if __name__ == "__main__":
main()
# vim: set ts=4 sw=4 ft=sh:

File diff suppressed because it is too large Load diff

View file

@ -344,7 +344,11 @@ with super;
})).overrideAttrs (old: { })).overrideAttrs (old: {
# Without this, source root is wrongly set to ./readline-2.6/doc # Without this, source root is wrongly set to ./readline-2.6/doc
setSourceRoot = '' setSourceRoot = ''
sourceRoot=./readline-2.6 sourceRoot=./readline-3.0
''; '';
}); });
# aliases
cjson = super.lua-cjson;
} }