~icefox/garnet#9: 
Thoughts on stdlib

Basically, what layers do we want to make it degrade gracefully?

First off, I think that instead of rust's std/alloc/core split, with std re-exporting all the things below it, the divisions should be explicit and multi-layered. If part of the lib isn't available on a platform, then it just doesn't exist. Conversely, if a program doesn't need part of the lib, then it just doesn't use it, and it's easy to determine at compile time that it's unnecessary.

Thoughts so far:

  • core: Fundamental stuff that doesn't need anything but computation. Analogous to Rust's core. Everything requires core
  • alloc: Stuff that requires a memory allocator but nothing more than that.
  • thread: Threading and associated primitives.
  • sys: OS provided functionality. File/network I/O, timekeeping, process management, etc.

...I was thinking of breaking it down further but don't remember how now. So, a platform like web browser would have core and alloc, but not thread and sys.

Status
REPORTED
Submitter
~icefox
Assigned to
No-one
Submitted
4 years ago
Updated
2 days ago
Labels
T-THOUGHTS

~icefox 4 years ago*

Another potential structure:

  • bare: Pure computation
  • mem: Memory allocation and data structures
  • io: File/stream I/O
  • proc: Threads, processes etc.
  • sys: Other system-level things like timekeeping, secure RNG, etc.
  • arch: Architecture-specific things such as SIMD intrinsics
  • os: OS-specific things like particular API bindings.

~icefox 3 years ago

Things to make sure get added: Bounded ints. Ratios. Bignums???

~icefox 2 years ago

Something also to consider: Have a stdlib and a semi-official contrib of high-quality third-party projects. Rust has talked about this several time but no single contrib ever unified and gained momentum.

~icefox 1 year, 7 months ago*

More thoughts on contrib (maybe call it batteries): have a specific set of design criteria and goals for it, admit that they may need to change over time, and use semver to let people use different versions of it.

Goals:

  • Bug-free, easily auditable, heavily tested and fuzzed
  • Minimalist -- this may be the wrong decision but it seems like an okay place to start
  • Cross-platform -- as much as is feasible anyway.
  • Semver-compliant -- preferably in an automated way.
  • Sensible defaults -- without having a million different options

Anti-goals:

  • Platform-specific -- an example would be a nicer API over POSIX specific filesystem handling
  • Ultra performance -- Things should be as fast as they can be while adhering to the goals
  • 100% complete -- this is a library for people to start with, not to be the end of everything.
  • Huge -- this should not become infeasible to maintain.
  • Hip -- This is not a place for experimental tools, it is a place for basic utilities

This also needs policies to maintain these goals. Criteria for inclusion: Informal agreement of the maintainer to maintain it for a year.

Criteria for removal: Packages will be reviewed once per year. If a package is no longer maintained, can be replaced by a totally better one, or has been found to have unfixable bugs/misdesigns that introduce security problems, then it should be removed. (This list is probably incomplete.) If a package is replaced by a totally better one three years in a row then the ecosystem is probably not mature and that category of package should be removed for a while.

Examples of packages that should be included:

  • zstd: Basic compression stuff. Maybe zip/deflate/etc too... maybe not? Opinionated, remember
  • rng: PRNG with some good algorithm
  • secure-rng: a way to get secure randomness from the OS
  • hash: sha256 and 512
  • alternate hashtable algorithms for no more than 2 different common use cases
  • command line arg parsing: preferably in a way that makes it easy to conform to a single format.
  • http/etc clients and servers

Maybe included:

  • image/media/etc codecs???
  • Ugly but cross-platform GUI a la Tk?
  • Templating language?
  • Lua interpreter bindings? Would be interesting to make this available to everyone as a matter of course.

Not included:

  • Big GUI a la Qt
  • Rust's rand crate -- too many unneeded options
  • Rust's clap crate -- too many unneeded options
  • Rust's error_chain crate -- Has been superceded by new designs every year for years
  • Rust's tokei -- Big and do-everything for async

~icefox 1 year, 7 months ago

Oh now here's a hornet's-nest issue: What does the license for contrib have to be? I will tentatively say LGPL; if it's good enough for SDL then it's good enough for me, and SDL is a good example of LGPL code being widely used in both industry and open-source.

~icefox 1 year, 5 months ago

It was suggested that I take a look at Julia's I/O and net libs, might have some inspiration there.

~icefox 1 year, 1 month ago*

Other things to put in contrib:

  • Rust's log
  • Rust's lazy_static, once_cell or equivalents as necessary
  • JSON stuff and/or `serde
  • Terminal colors?
  • Simple SIMD vectors?
  • base64 codec

Inspiration: https://github.com/ar-nelson/schemepunk

Simon Heath 6 months ago ยท edit

C++ stdlib thoughts on allocator APIs:

https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2006/n2045.html actually has a lot of interesting things to talk about

https://www.open-std.org/jtc1/sc22/wg14/www/docs/n1085.htm seems an earlier iteration of similar ideas

https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2016/p0401r0.html discusses later thoughts

~plecra 8 days ago

https://www.libsdl.org/license.php afaict SDL has been zlib for over a decade btw. I'm personally pretty anti-(L)GPL in FOSS. It seems sensible to leave licensing options up to maintainers of contrib libraries though - requring a license at least as permissive as LGPL, but publishing as zlib is also ok.

I do think that contrib should have a requirement that your API itself must be licensed zlib/BSD/public domain/MIT - to ensure users are allowed to replace libraries with their own code and use their own license

~icefox 3 days ago

Good point on the SDL license change, thanks. glibc is LGPL, so to me that seems like a pretty good precedent to use instead. For contrib libs, mixing licenses with GPL gets complicated, and that's intentional. GPL's design makes it a floor, not a ceiling. Figuring out all the details and ins and outs of that is important, but not urgent.

I'm personally pretty anti-(L)GPL in FOSS.

I was too for a long time, then around 2020 or something I looked around at the world and was like "you know what? This is fucking dumb. Why are we letting people do this crap?" So this is my little way of trying to make the world a better place. Someone wants to use Garnet? Go ahead, there's no barriers. Someone wants to port garnetc and its libraries to their game console, proprietary hardware, or some other system designed to let them control and price-gouge the hell out of everyone possible? They can give something back to the world for once, or they can waste a lot of effort rewriting their own implementation.

~plecra 3 days ago*

Someone wants to port garnetc and its libraries to their game console, proprietary hardware, or some other system designed to let them control and price-gouge the hell out of everyone possible?

I wholeheartedly agree that this is great, and I prefer most of my FOSS work to fall under these kinds of terms. Here's the specific situation I'm concerned about:

  • I've written some code in garnet and built a useful foss tool that I now want to commercialize with a few B2B features
    • I've used some of the contrib libraries fairly pervasively, as they're blessed by the language and kept in good condition
  • I now want to rewrite (as in, completely rewrite them from scratch) a few of those contrib libraries so that I can use them in my closed source distribution.

However I can't because the API of the GPL'd libraries is under license! Even if I replace the implementation of the library, I still have a bunch of calls to the library API in the source code I've already written, and that's a bunch of GPL'd IP scattered across my whole project. The best I can do is now rewrite the whole project to use my new API, and the open source version suffers for it too.

To me, this makes any development on top of GPL'd libraries a massive hazard. I can never decide to use my own code in unshared projects without risking major rewrites (nor can anyone I might want to share my code with).

(edit: The default of MIT + Apache2 in rust's ecosystem is too permissive, I agree on that ๐Ÿ˜ Using a license that ensures projects can stay public is a good idea)

~icefox 2 days ago

Splitting license talk into its own issue. GOTO #66 <3

Register here or Log in to comment, or comment via email.