update-luarocks-packages: use pluginupdate.py

Trying to reuse the update scripts used by kakoune/vim to provide the
user with an unified convergence. Some stuff doesn't work yet (parallel
download, caching) but I (anyone else welcome to try too) will improve
it in other PRs.
This commit is contained in:
Matthieu Coudron 2021-07-30 02:25:15 +02:00
parent 7b554c9477
commit 560a4e4e08
6 changed files with 672 additions and 524 deletions

View file

@ -309,7 +309,7 @@ Sample output2:
## Adding new plugins to nixpkgs {#adding-new-plugins-to-nixpkgs} ## Adding new plugins to nixpkgs {#adding-new-plugins-to-nixpkgs}
Nix expressions for Vim plugins are stored in [pkgs/misc/vim-plugins](/pkgs/misc/vim-plugins). For the vast majority of plugins, Nix expressions are automatically generated by running [`./update.py`](/pkgs/misc/vim-plugins/update.py). This creates a [generated.nix](/pkgs/misc/vim-plugins/generated.nix) file based on the plugins listed in [vim-plugin-names](/pkgs/misc/vim-plugins/vim-plugin-names). Plugins are listed in alphabetical order in `vim-plugin-names` using the format `[github username]/[repository]`. For example https://github.com/scrooloose/nerdtree becomes `scrooloose/nerdtree`. Nix expressions for Vim plugins are stored in [pkgs/misc/vim-plugins](/pkgs/misc/vim-plugins). For the vast majority of plugins, Nix expressions are automatically generated by running [`./update.py`](/pkgs/misc/vim-plugins/update.py). This creates a [generated.nix](/pkgs/misc/vim-plugins/generated.nix) file based on the plugins listed in [vim-plugin-names](/pkgs/misc/vim-plugins/vim-plugin-names). Plugins are listed in alphabetical order in `vim-plugin-names` using the format `[github username]/[repository]@[gitref]`. For example https://github.com/scrooloose/nerdtree becomes `scrooloose/nerdtree`.
Some plugins require overrides in order to function properly. Overrides are placed in [overrides.nix](/pkgs/misc/vim-plugins/overrides.nix). Overrides are most often required when a plugin requires some dependencies, or extra steps are required during the build process. For example `deoplete-fish` requires both `deoplete-nvim` and `vim-fish`, and so the following override was added: Some plugins require overrides in order to function properly. Overrides are placed in [overrides.nix](/pkgs/misc/vim-plugins/overrides.nix). Overrides are most often required when a plugin requires some dependencies, or extra steps are required during the build process. For example `deoplete-fish` requires both `deoplete-nvim` and `vim-fish`, and so the following override was added:

View file

@ -1,89 +1,87 @@
# nix name,luarocks name,server,version,luaversion,maintainers name,server,version,luaversion,maintainers
alt-getopt,,,,,arobyn bit32,,,lua5_1,lblasc
ansicolors,,,,, argparse,,,,
argparse,,,,, basexx,,,,
basexx,,,,, binaryheap,,,,vcunat
binaryheap,,,,,vcunat busted,,,,
bit32,,,,lua5_1,lblasc cassowary,,,,marsam alerque
busted,,,,, compat53,,,,vcunat
cassowary,,,,,marsam alerque cosmo,,,,marsam
cjson,lua-cjson,,,, coxpcall,,1.17.0-1,,
compat53,,,,,vcunat cqueues,,,,vcunat
cosmo,,,,,marsam cyrussasl,,,,
coxpcall,,,1.17.0-1,, digestif,,,lua5_3,
cqueues,,,,,vcunat dkjson,,,,
cyrussasl,,,,, fifo,,,,
digestif,,,,lua5_3, gitsigns.nvim,,,lua5_1,
dkjson,,,,, http,,,,vcunat
fifo,,,,, inspect,,,,
gitsigns.nvim,,,,lua5_1, ldbus,http://luarocks.org/dev,,,
http,,,,,vcunat ldoc,,,,
inspect,,,,, lgi,,,,
ldbus,,http://luarocks.org/dev,,, linenoise,,,,
ldoc,,,,, ljsyscall,,,lua5_1,lblasc
lgi,,,,, lpeg,,,,vyp
linenoise,,,,, lpeg_patterns,,,,
ljsyscall,,,,lua5_1,lblasc lpeglabel,,,,
lpeg,,,,,vyp lpty,,,,
lpeg_patterns,,,,, lrexlib-gnu,,,,
lpeglabel,,,,, lrexlib-pcre,,,,vyp
lpty,,,,, lrexlib-posix,,,,
lrexlib-gnu,,,,, ltermbox,,,,
lrexlib-pcre,,,,,vyp lua-cjson,,,,
lrexlib-posix,,,,, lua-cmsgpack,,,,
ltermbox,,,,, lua-iconv,,,,
lua-cmsgpack,,,,, lua-lsp,http://luarocks.org/dev,,,
lua-iconv,,,,, lua-messagepack,,,,
lua-lsp,,http://luarocks.org/dev,,, lua-resty-http,,,,
lua-messagepack,,,,, lua-resty-jwt,,,,
lua-resty-http,,,,, lua-resty-openidc,,,,
lua-resty-jwt,,,,, lua-resty-openssl,,,,
lua-resty-openidc,,,,, lua-resty-session,,,,
lua-resty-openssl,,,,, lua-term,,,,
lua-resty-session,,,,, lua-toml,,,,
lua-term,,,,, lua-zlib,,,,koral
lua-toml,,,,, lua_cliargs,,,,
lua-zlib,,,,,koral luabitop,,,,
lua_cliargs,,,,, luacheck,,,,
luabitop,,,,, luacov,,,,
luacheck,,,,, luadbi,,,,
luacov,,,,, luadbi-mysql,,,,
luadbi,,,,, luadbi-postgresql,,,,
luadbi-mysql,,,,, luadbi-sqlite3,,,,
luadbi-postgresql,,,,, luadoc,,,,
luadbi-sqlite3,,,,, luaepnf,,,,
luadoc,,,,, luaevent,,,,
luaepnf,,,,, luaexpat,,1.3.0-1,,arobyn flosse
luaevent,,,,, luaffi,http://luarocks.org/dev,,,
luaexpat,,,1.3.0-1,,arobyn flosse luafilesystem,,1.7.0-2,,flosse
luaffi,,http://luarocks.org/dev,,, lualogging,,,,
luafilesystem,,,1.7.0-2,,flosse luaossl,,,lua5_1,
lualogging,,,,, luaposix,,,,vyp lblasc
luaossl,,,,lua5_1, luarepl,,,,
luaposix,,,,,vyp lblasc luasec,,,,flosse
luarepl,,,,, luasocket,,,,
luasec,,,,,flosse luasql-sqlite3,,,,vyp
luasocket,,,,, luassert,,,,
luasql-sqlite3,,,,,vyp luasystem,,,,
luassert,,,,, luautf8,,,,pstn
luasystem,,,,, luazip,,,,
luautf8,,,,,pstn lua-yajl,,,,pstn
luazip,,,,, luuid,,,,
lua-yajl,,,,,pstn luv,,,,
luuid,,,,, lyaml,,,,lblasc
luv,,,,, markdown,,,,
lyaml,,,,,lblasc mediator_lua,,,,
markdown,,,,, mpack,,,,
mediator_lua,,,,, moonscript,,,,arobyn
mpack,,,,, nvim-client,,,,
moonscript,,,,,arobyn penlight,,,,
nvim-client,,,,, plenary.nvim,,,lua5_1,
penlight,,,,, rapidjson,,,,
plenary.nvim,,,,lua5_1, readline,,,,
rapidjson,,,,, say,,,,
readline,,,,, std._debug,,,,
say,,,,, std.normalize,,,,
std-_debug,std._debug,,,, stdlib,,,,vyp
std_normalize,std.normalize,,,, vstruct,,,,
stdlib,,,,,vyp
vstruct,,,,,

1 # nix name name luarocks name server version luaversion maintainers
2 alt-getopt bit32 lua5_1 arobyn lblasc
3 ansicolors argparse
4 argparse basexx
5 basexx binaryheap vcunat
6 binaryheap busted vcunat
7 bit32 cassowary lua5_1 lblasc marsam alerque
8 busted compat53 vcunat
9 cassowary cosmo marsam alerque marsam
10 cjson coxpcall lua-cjson 1.17.0-1
11 compat53 cqueues vcunat
12 cosmo cyrussasl marsam
13 coxpcall digestif 1.17.0-1 lua5_3
14 cqueues dkjson vcunat
15 cyrussasl fifo
16 digestif gitsigns.nvim lua5_3 lua5_1
17 dkjson http vcunat
18 fifo inspect
19 gitsigns.nvim ldbus http://luarocks.org/dev lua5_1
20 http ldoc vcunat
21 inspect lgi
22 ldbus linenoise http://luarocks.org/dev
23 ldoc ljsyscall lua5_1 lblasc
24 lgi lpeg vyp
25 linenoise lpeg_patterns
26 ljsyscall lpeglabel lua5_1 lblasc
27 lpeg lpty vyp
28 lpeg_patterns lrexlib-gnu
29 lpeglabel lrexlib-pcre vyp
30 lpty lrexlib-posix
31 lrexlib-gnu ltermbox
32 lrexlib-pcre lua-cjson vyp
33 lrexlib-posix lua-cmsgpack
34 ltermbox lua-iconv
35 lua-cmsgpack lua-lsp http://luarocks.org/dev
36 lua-iconv lua-messagepack
37 lua-lsp lua-resty-http http://luarocks.org/dev
38 lua-messagepack lua-resty-jwt
39 lua-resty-http lua-resty-openidc
40 lua-resty-jwt lua-resty-openssl
41 lua-resty-openidc lua-resty-session
42 lua-resty-openssl lua-term
43 lua-resty-session lua-toml
44 lua-term lua-zlib koral
45 lua-toml lua_cliargs
46 lua-zlib luabitop koral
47 lua_cliargs luacheck
48 luabitop luacov
49 luacheck luadbi
50 luacov luadbi-mysql
51 luadbi luadbi-postgresql
52 luadbi-mysql luadbi-sqlite3
53 luadbi-postgresql luadoc
54 luadbi-sqlite3 luaepnf
55 luadoc luaevent
56 luaepnf luaexpat 1.3.0-1 arobyn flosse
57 luaevent luaffi http://luarocks.org/dev
58 luaexpat luafilesystem 1.3.0-1 1.7.0-2 arobyn flosse flosse
59 luaffi lualogging http://luarocks.org/dev
60 luafilesystem luaossl 1.7.0-2 lua5_1 flosse
61 lualogging luaposix vyp lblasc
62 luaossl luarepl lua5_1
63 luaposix luasec vyp lblasc flosse
64 luarepl luasocket
65 luasec luasql-sqlite3 flosse vyp
66 luasocket luassert
67 luasql-sqlite3 luasystem vyp
68 luassert luautf8 pstn
69 luasystem luazip
70 luautf8 lua-yajl pstn
71 luazip luuid
72 lua-yajl luv pstn
73 luuid lyaml lblasc
74 luv markdown
75 lyaml mediator_lua lblasc
76 markdown mpack
77 mediator_lua moonscript arobyn
78 mpack nvim-client
79 moonscript penlight arobyn
80 nvim-client plenary.nvim lua5_1
81 penlight rapidjson
82 plenary.nvim readline lua5_1
83 rapidjson say
84 readline std._debug
85 say std.normalize
86 std-_debug stdlib std._debug vyp
87 std_normalize vstruct std.normalize
stdlib vyp
vstruct

View file

@ -28,6 +28,7 @@ from pathlib import Path
from typing import Dict, List, Optional, Tuple, Union, Any, Callable from typing import Dict, List, Optional, Tuple, Union, Any, Callable
from urllib.parse import urljoin, urlparse from urllib.parse import urljoin, urlparse
from tempfile import NamedTemporaryFile from tempfile import NamedTemporaryFile
from dataclasses import dataclass
import git import git
@ -82,6 +83,13 @@ def make_request(url: str) -> urllib.request.Request:
headers["Authorization"] = f"token {token}" headers["Authorization"] = f"token {token}"
return urllib.request.Request(url, headers=headers) return urllib.request.Request(url, headers=headers)
@dataclass
class PluginDesc:
owner: str
repo: str
branch: str
alias: str
class Repo: class Repo:
def __init__( def __init__(
@ -195,21 +203,46 @@ class Editor:
name: str, name: str,
root: Path, root: Path,
get_plugins: str, get_plugins: str,
# TODO modifier pour accepter un plugin
generate_nix: Callable[[List[Tuple[str, str, Plugin]], str], None], generate_nix: Callable[[List[Tuple[str, str, Plugin]], str], None],
default_in: Optional[Path] = None, default_in: Optional[Path] = None,
default_out: Optional[Path] = None, default_out: Optional[Path] = None,
deprecated: Optional[Path] = None, deprecated: Optional[Path] = None,
cache_file: Optional[str] = None, cache_file: Optional[str] = None,
): ):
log.debug("get_plugins:", get_plugins)
self.name = name self.name = name
self.root = root self.root = root
self.get_plugins = get_plugins self.get_plugins = get_plugins
self.generate_nix = generate_nix self._generate_nix = generate_nix
self.default_in = default_in or root.joinpath(f"{name}-plugin-names") self.default_in = default_in or root.joinpath(f"{name}-plugin-names")
self.default_out = default_out or root.joinpath("generated.nix") self.default_out = default_out or root.joinpath("generated.nix")
self.deprecated = deprecated or root.joinpath("deprecated.json") self.deprecated = deprecated or root.joinpath("deprecated.json")
self.cache_file = cache_file or f"{name}-plugin-cache.json" self.cache_file = cache_file or f"{name}-plugin-cache.json"
def get_current_plugins(self):
"""To fill the cache"""
return get_current_plugins(self)
def load_plugin_spec(self, plugin_file) -> List[PluginDesc]:
return load_plugin_spec(plugin_file)
def generate_nix(self, plugins, outfile):
'''Returns nothing for now, writes directly to outfile'''
self._generate_nix(plugins, outfile)
def get_update(self, input_file: str, outfile: str, proc: int):
return get_update(input_file, outfile, proc, editor=self)
@property
def attr_path(self):
return self.name + "Plugins"
def rewrite_input(self, *args, **kwargs):
return rewrite_input(*args, **kwargs)
class CleanEnvironment(object): class CleanEnvironment(object):
def __enter__(self) -> None: def __enter__(self) -> None:
@ -228,7 +261,9 @@ class CleanEnvironment(object):
def get_current_plugins(editor: Editor) -> List[Plugin]: def get_current_plugins(editor: Editor) -> List[Plugin]:
with CleanEnvironment(): with CleanEnvironment():
out = subprocess.check_output(["nix", "eval", "--json", editor.get_plugins]) cmd = ["nix", "eval", "--json", editor.get_plugins]
log.debug("Running command %s", cmd)
out = subprocess.check_output(cmd)
data = json.loads(out) data = json.loads(out)
plugins = [] plugins = []
for name, attr in data.items(): for name, attr in data.items():
@ -244,12 +279,13 @@ def prefetch_plugin(
alias: Optional[str], alias: Optional[str],
cache: "Optional[Cache]" = None, cache: "Optional[Cache]" = None,
) -> Tuple[Plugin, Dict[str, str]]: ) -> Tuple[Plugin, Dict[str, str]]:
log.info("Prefetching plugin %s", repo_name) log.info(f"Fetching last commit for plugin {user}/{repo_name}@{branch}")
repo = Repo(user, repo_name, branch, alias) repo = Repo(user, repo_name, branch, alias)
commit, date = repo.latest_commit() commit, date = repo.latest_commit()
has_submodules = repo.has_submodules() has_submodules = repo.has_submodules()
cached_plugin = cache[commit] if cache else None cached_plugin = cache[commit] if cache else None
if cached_plugin is not None: if cached_plugin is not None:
log.debug("Cache hit !")
cached_plugin.name = alias or repo_name cached_plugin.name = alias or repo_name
cached_plugin.date = date cached_plugin.date = date
return cached_plugin, repo.redirect return cached_plugin, repo.redirect
@ -306,8 +342,7 @@ def check_results(
sys.exit(1) sys.exit(1)
def parse_plugin_line(line: str) -> PluginDesc:
def parse_plugin_line(line: str) -> Tuple[str, str, str, Optional[str]]:
branch = "master" branch = "master"
alias = None alias = None
name, repo = line.split("/") name, repo = line.split("/")
@ -317,15 +352,15 @@ def parse_plugin_line(line: str) -> Tuple[str, str, str, Optional[str]]:
if "@" in repo: if "@" in repo:
repo, branch = repo.split("@") repo, branch = repo.split("@")
return (name.strip(), repo.strip(), branch.strip(), alias) return PluginDesc(name.strip(), repo.strip(), branch.strip(), alias)
def load_plugin_spec(plugin_file: str) -> List[Tuple[str, str, str, Optional[str]]]: def load_plugin_spec(plugin_file: str) -> List[PluginDesc]:
plugins = [] plugins = []
with open(plugin_file) as f: with open(plugin_file) as f:
for line in f: for line in f:
plugin = parse_plugin_line(line) plugin = parse_plugin_line(line)
if not plugin[0]: if not plugin.owner:
msg = f"Invalid repository {line}, must be in the format owner/repo[ as alias]" msg = f"Invalid repository {line}, must be in the format owner/repo[ as alias]"
print(msg, file=sys.stderr) print(msg, file=sys.stderr)
sys.exit(1) sys.exit(1)
@ -387,12 +422,11 @@ class Cache:
def prefetch( def prefetch(
args: Tuple[str, str, str, Optional[str]], cache: Cache args: PluginDesc, cache: Cache
) -> Tuple[str, str, Union[Exception, Plugin], dict]: ) -> Tuple[str, str, Union[Exception, Plugin], dict]:
assert len(args) == 4 owner, repo = args.owner, args.repo
owner, repo, branch, alias = args
try: try:
plugin, redirect = prefetch_plugin(owner, repo, branch, alias, cache) plugin, redirect = prefetch_plugin(owner, repo, args.branch, args.alias, cache)
cache[plugin.commit] = plugin cache[plugin.commit] = plugin
return (owner, repo, plugin, redirect) return (owner, repo, plugin, redirect)
except Exception as e: except Exception as e:
@ -433,7 +467,7 @@ def rewrite_input(
with open(input_file, "w") as f: with open(input_file, "w") as f:
f.writelines(lines) f.writelines(lines)
# TODO move to Editor ?
def parse_args(editor: Editor): def parse_args(editor: Editor):
parser = argparse.ArgumentParser( parser = argparse.ArgumentParser(
description=( description=(
@ -446,7 +480,7 @@ def parse_args(editor: Editor):
dest="add_plugins", dest="add_plugins",
default=[], default=[],
action="append", action="append",
help=f"Plugin to add to {editor.name}Plugins from Github in the form owner/repo", help=f"Plugin to add to {editor.attr_path} from Github in the form owner/repo",
) )
parser.add_argument( parser.add_argument(
"--input-names", "--input-names",
@ -493,11 +527,11 @@ def commit(repo: git.Repo, message: str, files: List[Path]) -> None:
def get_update(input_file: str, outfile: str, proc: int, editor: Editor): def get_update(input_file: str, outfile: str, proc: int, editor: Editor):
cache: Cache = Cache(get_current_plugins(editor), editor.cache_file) cache: Cache = Cache(editor.get_current_plugins(), editor.cache_file)
_prefetch = functools.partial(prefetch, cache=cache) _prefetch = functools.partial(prefetch, cache=cache)
def update() -> dict: def update() -> dict:
plugin_names = load_plugin_spec(input_file) plugin_names = editor.load_plugin_spec(input_file)
try: try:
pool = Pool(processes=proc) pool = Pool(processes=proc)
@ -522,33 +556,33 @@ def update_plugins(editor: Editor):
log.info("Start updating plugins") log.info("Start updating plugins")
nixpkgs_repo = git.Repo(editor.root, search_parent_directories=True) nixpkgs_repo = git.Repo(editor.root, search_parent_directories=True)
update = get_update(args.input_file, args.outfile, args.proc, editor) update = editor.get_update(args.input_file, args.outfile, args.proc)
redirects = update() redirects = update()
rewrite_input(args.input_file, editor.deprecated, redirects) editor.rewrite_input(args.input_file, editor.deprecated, redirects)
autocommit = not args.no_commit autocommit = not args.no_commit
if autocommit: if autocommit:
commit(nixpkgs_repo, f"{editor.name}Plugins: update", [args.outfile]) commit(nixpkgs_repo, f"{editor.attr_path}: update", [args.outfile])
if redirects: if redirects:
update() update()
if autocommit: if autocommit:
commit( commit(
nixpkgs_repo, nixpkgs_repo,
f"{editor.name}Plugins: resolve github repository redirects", f"{editor.attr_path}: resolve github repository redirects",
[args.outfile, args.input_file, editor.deprecated], [args.outfile, args.input_file, editor.deprecated],
) )
for plugin_line in args.add_plugins: for plugin_line in args.add_plugins:
rewrite_input(args.input_file, editor.deprecated, append=(plugin_line + "\n",)) editor.rewrite_input(args.input_file, editor.deprecated, append=(plugin_line + "\n",))
update() update()
plugin = fetch_plugin_from_pluginline(plugin_line) plugin = fetch_plugin_from_pluginline(plugin_line)
if autocommit: if autocommit:
commit( commit(
nixpkgs_repo, nixpkgs_repo,
"{editor}Plugins.{name}: init at {version}".format( "{editor.attr_path}.{name}: init at {version}".format(
editor=editor.name, name=plugin.normalized_name, version=plugin.version editor=editor.name, name=plugin.normalized_name, version=plugin.version
), ),
[args.outfile, args.input_file], [args.outfile, args.input_file],

View file

@ -1,140 +1,179 @@
#!/usr/bin/env nix-shell #!/usr/bin/env nix-shell
#!nix-shell update-luarocks-shell.nix -i bash #!nix-shell -p nix-prefetch-git luarocks-nix python3 python3Packages.GitPython nix -i python3
# You'll likely want to use # format:
# `` # $ nix run nixpkgs.python3Packages.black -c black update.py
# nixpkgs $ maintainers/scripts/update-luarocks-packages pkgs/development/lua-modules/generated-packages.nix # type-check:
# `` # $ nix run nixpkgs.python3Packages.mypy -c mypy update.py
# to update all libraries in that folder. # linted:
# to debug, redirect stderr to stdout with 2>&1 # $ nix run nixpkgs.python3Packages.flake8 -c flake8 --ignore E501,E265,E402 update.py
# stop the script upon C-C import inspect
set -eu -o pipefail import os
import tempfile
import shutil
from dataclasses import dataclass
import subprocess
import csv
import logging
CSV_FILE="maintainers/scripts/luarocks-packages.csv" from typing import List
from pathlib import Path
LOG_LEVELS = {
logging.getLevelName(level): level for level in [
logging.DEBUG, logging.INFO, logging.WARN, logging.ERROR ]
}
log = logging.getLogger()
log.addHandler(logging.StreamHandler())
ROOT = Path(os.path.dirname(os.path.abspath(inspect.getfile(inspect.currentframe())))).parent.parent
from pluginupdate import Editor, parse_args, update_plugins, PluginDesc, CleanEnvironment
PKG_LIST="maintainers/scripts/luarocks-packages.csv"
TMP_FILE="$(mktemp)" TMP_FILE="$(mktemp)"
# Set in the update-luarocks-shell.nix GENERATED_NIXFILE="pkgs/development/lua-modules/generated-packages.nix"
NIXPKGS_PATH="$LUAROCKS_NIXPKGS_PATH" LUAROCKS_CONFIG="$NIXPKGS_PATH/maintainers/scripts/luarocks-config.lua"
export LUAROCKS_CONFIG="$NIXPKGS_PATH/maintainers/scripts/luarocks-config.lua"
# 10 is a pretty arbitrary number of simultaneous jobs, but it is generally HEADER = """
# impolite to hit a webserver with *too* many simultaneous connections :) /* {GENERATED_NIXFILE} is an auto-generated file -- DO NOT EDIT!
PARALLEL_JOBS=1
exit_trap() {
local lc="$BASH_COMMAND" rc=$?
test $rc -eq 0 || echo -e "*** error $rc: $lc.\nGenerated temporary file in $TMP_FILE" >&2
}
print_help() {
echo "Usage: $0 <GENERATED_FILE>"
echo "(most likely pkgs/development/lua-modules/generated-packages.nix)"
echo ""
echo " -c <CSV_FILE> to set the list of luarocks package to generate"
exit 1
}
if [ $# -lt 1 ]; then
print_help
exit 1
fi
trap exit_trap EXIT
while getopts ":hc:" opt; do
case $opt in
h)
print_help
;;
c)
echo "Loading package list from $OPTARG !" >&2
CSV_FILE="$OPTARG"
;;
\?)
echo "Invalid option: -$OPTARG" >&2
;;
esac
shift $((OPTIND - 1))
done
GENERATED_NIXFILE="$1"
HEADER="
/* ${GENERATED_NIXFILE} is an auto-generated file -- DO NOT EDIT!
Regenerate it with: Regenerate it with:
nixpkgs$ ${0} ${GENERATED_NIXFILE} nixpkgs$ ./maintainers/scripts/update-luarocks-packages
These packages are manually refined in lua-overrides.nix You can customize the generated packages in pkgs/development/lua-modules/overrides.nix
*/ */
{ self, stdenv, lib, fetchurl, fetchgit, pkgs, ... } @ args: """.format(GENERATED_NIXFILE=GENERATED_NIXFILE)
FOOTER="""
}
/* GENERATED - do not edit this file */
"""
@dataclass
class LuaPlugin:
name: str
version: str
server: str
luaversion: str
maintainers: str
@property
def normalized_name(self) -> str:
return self.name.replace(".", "-")
# rename Editor to LangUpdate/ EcosystemUpdater
class LuaEditor(Editor):
def get_current_plugins(self):
return []
def load_plugin_spec(self, input_file) -> List[PluginDesc]:
luaPackages = []
csvfilename=input_file
log.info("Loading package descriptions from %s", csvfilename)
with open(csvfilename, newline='') as csvfile:
reader = csv.DictReader(csvfile,)
for row in reader:
# name,server,version,luaversion,maintainers
plugin = LuaPlugin(**row)
luaPackages.append(plugin)
return luaPackages
@property
def attr_path(self):
return "luaPackages"
def get_update(self, input_file: str, outfile: str, _: int):
def update() -> dict:
plugin_specs = self.load_plugin_spec(input_file)
self.generate_nix(plugin_specs, outfile)
redirects = []
return redirects
return update
def rewrite_input(self, *args, **kwargs):
# not implemented yet
pass
def generate_nix(
plugins: List[LuaPlugin],
outfilename: str
):
sorted_plugins = sorted(plugins, key=lambda v: v.name.lower())
# plug = {}
# selon le manifest luarocks.org/manifest
def _generate_pkg_nix(plug):
cmd = [ "luarocks", "nix", plug.name]
if plug.server:
cmd.append(f"--only-server={plug.server}")
if plug.maintainers:
cmd.append(f"--maintainers={plug.maintainers}")
if plug.version:
cmd.append(plug.version)
if plug.luaversion:
with CleanEnvironment():
local_pkgs = str(ROOT.resolve())
cmd2 = ["nix-build", "--no-out-link", local_pkgs, "-A", f"{plug.luaversion}"]
log.debug("running %s", cmd2)
lua_drv_path=subprocess.check_output(cmd2, text=True).strip()
cmd.append(f"--lua-dir={lua_drv_path}/bin")
log.debug("running %s", cmd)
output = subprocess.check_output(cmd, text=True)
return output
with tempfile.NamedTemporaryFile("w+") as f:
f.write(HEADER)
f.write("""
{ self, stdenv, lib, fetchurl, fetchgit, ... } @ args:
self: super: self: super:
with self; with self;
{ {
" """)
FOOTER=" for plugin in sorted_plugins:
}
/* GENERATED */
"
function convert_pkg() { nix_expr = _generate_pkg_nix(plugin)
nix_pkg_name="$1" f.write(f"{plugin.normalized_name} = {nix_expr}"
lua_pkg_name="$2" )
server="$3" f.write(FOOTER)
pkg_version="$4" f.flush()
lua_version="$5"
maintainers="$6"
if [ "${nix_pkg_name:0:1}" == "#" ]; then # if everything went fine, move the generated file to its destination
echo "Skipping comment ${*}" >&2 # using copy since move doesn't work across disks
return shutil.copy(f.name, outfilename)
fi
# Normalize package name print(f"updated {outfilename}")
nix_pkg_name_normalized=$(sed 's/\./-/' <(echo "$nix_pkg_name"))
if [ -z "$lua_pkg_name" ]; then def load_plugin_spec():
echo "Using nix_name as lua_pkg_name for '$nix_pkg_name'" >&2 pass
lua_pkg_name="$nix_pkg_name"
fi
echo "Building expression for $lua_pkg_name (version $pkg_version) from server [$server]" >&2
luarocks_args=(nix)
if [[ -n $server ]]; then
luarocks_args+=("--only-server=$server")
fi
if [[ -n $maintainers ]]; then
luarocks_args+=("--maintainers=$maintainers")
fi
if [[ -n $lua_version ]]; then
lua_drv_path=$(nix-build --no-out-link "$NIXPKGS_PATH" -A "$lua_version")
luarocks_args+=("--lua-dir=$lua_drv_path/bin")
fi
luarocks_args+=("$lua_pkg_name")
if [[ -n $pkg_version ]]; then
luarocks_args+=("$pkg_version")
fi
echo "Running 'luarocks ${luarocks_args[*]}'" >&2
if drv="$nix_pkg_name_normalized = $(luarocks "${luarocks_args[@]}")"; then
echo "$drv"
else
echo "Failed to convert $nix_pkg_name" >&2
return 1
fi
}
# params needed when called via callPackage def main():
echo "$HEADER" | tee "$TMP_FILE"
# Ensure parallel can run our bash function editor = LuaEditor("lua", ROOT, '', generate_nix,
export -f convert_pkg default_in = ROOT.joinpath(PKG_LIST),
export SHELL=bash default_out = ROOT.joinpath(GENERATED_NIXFILE)
# Read each line in the csv file and run convert_pkg for each, in parallel )
parallel --group --keep-order --halt now,fail=1 --jobs "$PARALLEL_JOBS" --colsep ',' convert_pkg {} <"$CSV_FILE" | tee -a "$TMP_FILE"
# close the set args = parse_args(editor)
echo "$FOOTER" | tee -a "$TMP_FILE" log.setLevel(LOG_LEVELS[args.debug])
cp "$TMP_FILE" "$GENERATED_NIXFILE" update_plugins(editor)
if __name__ == "__main__":
main()
# vim: set ts=4 sw=4 ft=sh:

File diff suppressed because it is too large Load diff

View file

@ -347,4 +347,8 @@ with super;
sourceRoot=./readline-2.6 sourceRoot=./readline-2.6
''; '';
}); });
# aliases
cjson = super.lua-cjson;
} }