Fish – Update on the Rust port

1. It's amazing that they're doing this as a gradual C++ to Rust rewrite, while keeping it working end-to-end, if I understand correctly.

2. It's amazing how quickly this is going.

3. If you haven't tried fish yet, make sure to do so! It's a really ergonomic shell and overall very pleasant to use, with good defaults (you don't have to customize it, even though you can, for a great experience). I've switched from bash a couple years ago and haven't looked back since.

Bonus: you won't have to google how to write a for loop in bash ever again (which I, writing them rarely, and them being unintuitive enough, had to do every single time)!

Tip for some easy to remember Bash loop constructs:

Incremental index:

   for ((i = 0; i < 10; i++)); do echo $i; done

For iterating over an array:

   foo[0]=a
   foo[1]=b
   for i in ${foo[@]}; do echo $i; done

Please please please always quote variable expansion. Just do it everywhere, every time.

    for i in “${foo[@]}”; do

Or use Fish.

Or python…

python is unbelievably awkward for shell scripting though

I think with the cmd package it’s not actually that bad and quite ergonomic. Ymmv

Do you mean this cmd package?

https://docs.python.org/3/library/cmd.html

If so, that's entirely orthogonal to the problem. cmd is for writing the interactive "front end" to a command-line interpreter; it doesn't help at all in writing Python scripts to replace shell scripts (which typically run with no interactive interface at all).

The sort of problem I think fragmede is alluding to is that of writing Python code to emulate a shell script construct like:

    some-command 2>&1 | LANG=C sort | uniq -c | wc -l

e.g. constructing pipelines of external commands. It's certainly possible to do this in Python, but I'm not aware of any way that'd be anywhere near as concise as the shell syntax.

I really don't find using subprocess all that tedious personally. I usually have one helper function around `subproces.run` called `runx` that adds a little sugar.

But if you really want something more ergonomic, there's sh:

https://pypi.org/project/sh/

subprocess doesn't make it easy to construct pipelines -- it's possible, but involves a lot of subprocess.Popen(..., stdin=subprocess.PIPE, stdout=subprocess.PIPE) and cursing. The "sh" module doesn't support it at all; each command runs synchronously.

Incorrect. https://sh.readthedocs.io/en/latest/sections/piping.html

Just add `_piped=True` to the launch arguments and it'll work as expected where it won't fully buffer the output & wait for the command to complete.

YMMV but I found sh to be a step function better ergonomically, especially if you want to do anything remotely complex. I just wish that they would standardize it & clean it up to be a bit more pythonic (like command-line arguments as an array & then positional arguments with normal names instead of magic leading `_` positional arguments).

Shoot. I mixed up the module name. It's the sh module https://sh.readthedocs.io/en/latest/

    sort_env = os.environ.copy()
    sort_env["LANG"] = "C"
    sh.wc("-l", _in=sh.uniq("-c", _piped=True, _in=sh.sort(_env=sort_env, _piped=True, _in=sh("some-command", _piped=True, _err_to_out=True))))

It's not as natural in some ways for most people because you have to write it right to left instead of left to right as with the pipe syntax. If you split it over multiple lines it's better:

    some_command = sh("some-command", _piped=True, _err_to_out=True)
    sorted = sh.sort(_env=sort_env, _piped=True, _in=some_command)
    unique = sh.uniq("-c", _piped=True, _in=sorted)
    word_count = sh.wc("-l", _in=unique)

There's also all sorts of helpful stuff you can do like invoking a callback per line or chunk of output, with contexts for running a sequence of commands as sudo, etc etc.

And of course, you don't actually need to shell out to sort/uniq either:

   output_lines = sh("some-command", _err_to_out=True).splitlines()
   num_lines = len(list(set(output_lines)))

This is also cheaper because it avoids the sort which isn't strictly necessary for determining the number of unique lines (sorting is typically going to be an expensive way to do that for large files compared to a hash set because of the O(nlogn) string comparisons vs O(n) hashes).

It's really quite amazing and way less error prone too when maintaining anything more complicated. Of course, I've found it not as easy to develop muscle memory with it but that's my general experience with libraries.

Oh neat, I guess I missed the "_piped" arg when I looked. That does make it a lot better.

And of course, you don't actually need to shell out to sort/uniq either:

Yeah, it's a contrived example. Imagine something more important happening there. :)

yeah exactly. eg

    find . -type f | grep foo

is possible to do with os.walk() in python, but to get there is so unergonomic.

    for file_path in sh.find(".", "-type", "f", _iter=True):
        if "foo" in file_path:
            print(file_path)

Note that "find" is not some special function provided by the sh module - that module just uses convenience magic where it'll map whatever function you call under it as an actual command execution (i.e. construct a sh.Command using "find" as an argument & then execute it).

You can also pipe the output of find to grep if you want to run grep for some reason instead of writing that processing in python (maybe it's faster or just more convenient)

For writing on the command-line, yes. For anything over a screenful of code and/or using non-trivial math or non-scalar variables, I’ve found the opposite to be true. Python forces multi-line code more but the functionality is so much richer that it’s easy to end up replacing a hundred line shell script with half as many lines of easier-to-read code. Literally every time I’ve done that with a mature shell script I’ve also found at least one bug in the process, usually related to error handling or escaping, which the original author knew about but also knew would be painful to handle in shell.

^ this

To illustrate why, consider a file name with spaces.

    filename="my file.txt"
    cat $filename   # Incorrect: Tries to 'cat' two files: 'my' and 'file.txt'

    cat "$filename" # Correct: Treats 'my file.txt' as one file

Additionally,

    $ f=-x
    $ cat "$f"
    cat: invalid option -- 'x'

    $ cat -- "$f"
    cat: -x: No such file or directory
    # ^-- correct, but:

    $ f=-
    $ cat -- "$f"
    reading stdin...
    reading stdin...
    ^C

    $ f=./-
    $ cat -- "$f"
    cat: ./-: No such file or directory

...better to glob with ./* than *

Also for sanitizing user input.

  input=$1  # User input, potentially dangerous
  rm $input # Incorrect: Risky if input contains something like '*'

  rm "$input" # Correct: Safer, treats the user input as a single item

True, true. I don't worry about it for explicitly simple indexes because too many quotes are ugly. But in general it's completely right.

Just to illustrate for others, in fish that would be

  for i in (seq 10); echo $i; end

or usually written as

  for i in (seq 10)
    echo $i
  end

and that is already based on iteration, so for iterating over files, you'll similarly use

  for f in (ls); echo $f; end

and it works the same for arrays.

You don't even need the (ls) for file iteration:

  for i in *
    echo $i
  end

Fish also has globstar out of the box, so you can do

  for i in **/*
    echo $i
  end

[You don't even need the (ls) for file iteration:

  for i in *
    echo $i
  end

]

You don't even need the for for that (specific example):

echo *

at least in sh and bash.

echo * does not have the newline after each file

True, echo * will print all the filenames on one line, with a space between pairs, but for data processing purposes, the two snippets are equivalent, because as per shell behavior, newline, tab and space are all treated as space, unless quoted. Like the definition of whitespace in the K&R C book. After all, Unix is written in C.

You wouldn’t iterate over `echo *` anyway. This works just fine in bash/sh:

    for file in *; do
        echo "$file"
    done

Also, IIRC, and less commonly seen, you can do:

  for i 
  do 
    # your commands using $i here
  done

which will iterate over all the command line arguments to the script containing that for loop. It is a shortcut for:

  for i in $*
    # note: not for i in *
  # the rest is the same above

This is mentioned in either the shell man page or in the Kernighan & Pike book The Unix Programming Environment.

Your example in fish:

  for i in (seq 10); echo $i; end

directly translated to bash:

  for i in $(seq 10); do echo $i; done

No need for seq :)

    for i in {1..10}; do echo $i ; done

If, for whatever reason, you want leading zeroes - BASH will respect that. Do {01..10}

seq would work in bash too, but it's not part of the language, it's an external tool. C like syntax is more expressive I'd say for simple numerical increments.

“Just remember how to do it” isn’t really the problem.

It’s that the finer details of the syntax is sufficiently different from other things I do, and I don’t write shell scripts frequently enough to remember it.

exactly. i predict some kind of 'chat gpt shell' in the near future.

GitHub has already released this a couple months ago: https://docs.github.com/en/copilot/github-copilot-in-the-cli...

zsh makes this so much easier. This doesn't capture $i, but covers most use cases of "I want to run something more than once":

  repeat 10; echo "repeat me"

If you do need $i you can use a similar construct as bash, but more convenient:

  for ((i = 0; i < 10; i++)); echo $i

  for i in {0..10}; echo $i  # {n..m} loops also work in bash

  for ((i = 0; i < 10; i++)); { echo $i; echo $i }  # Multiple commands

Short loops are hugely helpful in interactive use because it's much less muckery for short one-liners (whether you should use them in scripts is probably a bit more controversial).

Also looping over arrays "just works" as you would expect without requiring special incantations:

  arr=(a b)
  for v in $arr; echo $v

---

Whether zsh or fish is better is a matter of taste, and arguably zsh has too many features, but IMHO bash is absolutely stuck as a "1989 ksh clone" (in more ways than one; it uses K&R C all over the place, still has asm hacks to make it run on 1980s versions of Xenix, and things like that).

Any reason to eschew short_loops in general that you're aware of? I ask because I'd probably use `for i ({0..9}) echo $i` in your for loop example. I've never managed to get my head around the necessity for a narrow short_repeat option when there is - for example - no short_select.

All of the zsh alternate forms feel far superior to me, in both interactive use and within scripts.

JNRowe notes the many off-by-one translations in other examples and tips hat to arp242

Oh yeah, I forgot about the {n..m} syntax; I almost always use repeat these days or the C-style loop if I need $i, as that's more "in the fingers", so to speak, from other languages, even though {n..m} is easier.

I don't know why you would want to avoid short loops, other than subjective stylistic reasons (which is completely valid of course). The patch that added short_repeat just asserts that "SHORT_LOOPS is bad"[1], but I don't now why the author thinks that.

Also: my previous comment was wrong; you don't need "setopt short_repeat"; you only need it if you explicitly turned off short loops with "setopt no_short_loops".

[1]: https://www.zsh.org/mla/workers/2019/msg01174.html

For iterating over an array:

Bash array syntax is error prone and hard to read, and the expansion of arrays still depend on the field separator and must fit in the environment.

Most of the time you should just rely on that field separator and do it the simple way:

  for i in "$f"; do echo $i; done

Much more obvious and more shell-like. That's how all bash functions process their parameters.

Set IFS only if you really need to. But in that case also consider something like xargs where you are not limited by environment, have null terminated fields, and offer parallelism.

Arrays are only useful when you need multidimensionality, at which point you should probably look at using data files and process with other tools such as sed. Or start looking at something like Perl or Python.

Your example proves that not using arrays is worse. That loop will only run once and just print every element on one line. The array equivalent works as expected , _isn't_ affected by IFS, and can handle spaces in individual elements

    f='a b c d e'
    for i in "$f"; do echo $i; done
    # prints a b c d e

    f="a b c 'd e'"
    for i in $f; do echo $i; done
    # prints
    # a
    # b
    # c
    # 'd
    # e'

    f=(a b c 'd e')
    for i in "${f[@]}"; do echo $i; done
    # prints
    # a
    # b
    # c
    # d e

Can't speak for others but one issue is the "; do"

I can generally cobble together python, and it'll be syntactically correct. I just need to check libs.

With shell I need to stop and make sure my semicolons in a for loop are correct.

If you're a heavy user it probably isn't a problem but all the little warts just make it difficult for an occasional user to keep a good enough model in their head.

You don't need the semicolon. Just use a newline instead; it's also easier to remember that way.

zsh, short loop:

    for i in {1..10}; <<<$i

powershell:

    foreach ($i in 1..10) { $i }

    1..10|%{$_}

For the first, I prefer range syntax:

    for i in {1..10}

1. It's amazing that they're doing this as a gradual C++ to Rust rewrite, while keeping it working end-to-end, if I understand correctly.

Seems to me they're not doing it gradually at all.

Another thing:

We plan on not doing any partial-rust release.

That means we would be doing e.g. fish 4.0 as fully rust and zero C++, and I think, contrary to what we usually do that warrants a beta. (Ordinarily we've stopped doing betas and release candidates because they simply don't get any testing).

We also still want to do a 3.7.0 in-between release that is still purely C++, so we have a better jumping off point for platforms that can't handle the rust version. It would be 3.6.1 with some neat changes backported.

https://github.com/fish-shell/fish-shell/discussions/10123#d...

The price of C++ compatibility is that it doesn't use Rust strings internally. It's all "wchar" and "WString", which are Microsoft C++ "wide character strings". This may be more of a Microsoft backwards compatibility issue than a C++ issue.

and yet they may be breaking Cygwin support?

I think the problem is more like, there are valid files that aren't representable as rust strings?

here are valid files that aren't representable as rust strings

Like? I don't use rust but I assume it's capable of utf 8 ?

The siblings are correct but also not precise in a way in which could be misleading.

Rust has one built-in string type: str, commonly written &str. This is a sequence of UTF-8 encoded bytes.

Rust also has several standard library string types:

* String is an owned version of &str. Also UTF-8.

* CString/CStr: like String and &str, but null terminated, with no specific encoding

* OsString/OsStr: an "os native" encoded string, in practice, this is the WTF-8 spec.

And of course, all sorts of user library-created string types.

The issue at hand isn't that there's some sort of special string type that Rust can't represent, it's that they have an existing codebase that uses a specific string type, so they're going to keep using that type for compatibility reasons, rather than one of the more usual Rust types. This means they won't have to transcode between the two types.

<3 WTF-8

If we're competing to be pedantic here, the problem is that Windows encodes paths as UTF-16 and Linux can support just about any random jumble of bytes in a path regardless of if those bytes are valid UTF-8 or not. Neither of these play nicely with Rust's "if it can be represented, it's valid" approach to code safety, so OsStr(ing) exists as a more permissive but less powerful analogue for such cases.

Rust is capable of UTF-8. It is not capable of not UTF-8. Any sequence of bytes that is not a valid UTF-8 string cannot be represented by the String type.

Rust strings enforce utf-8 encoding, yes. However, it seems Windows (which uses utf-16) allows ill-formed UTF-16, in particular it tolerates unpaired surrogates.

You can read more here http://simonsapin.github.io/wtf-8/

That's why OsString exists.

The Cygwin issue isn't strings (well, that could be another issue) but that Rust doesn't support Cygwin in the first place, at least according to the comments in the linked thread.

Well, for whatever reason, it doesn't meet their requirements... See the String section of https://github.com/fish-shell/fish-shell/blob/master/doc_int...

edit: here is a more specific rationale: https://github.com/fish-shell/fish-shell/pull/9512#discussio...

It has nothing to do with Windows. fish doesn't support Windows. Their use of wchar_t is the glibc wchar_t (wchar_t is not Microsoft-specific) which is a 32-bit type and stores UTF-32-encoded codepoints. The Rust type they're using is also the same ( https://github.com/fish-shell/fish-shell/blob/master/doc_int... ).

Seems to me they're not doing it gradually at all.

They are doing the release all-at-once, but developing gradually. From the original PR[1]:

This should be thought of as a "port" instead of a "rewrite" because we would not start from scratch; instead we would translate C++ to Rust, incrementally, module by module, in the span of one release. We'll use an FFI so the Rust and C++ bits can talk to each other until C++ is gone, and tests and CI keep passing at every commit.

[1] https://github.com/fish-shell/fish-shell/pull/9512

It's a bit of both. They are not doing any mixed C++/Rust releases, but you can check out the source and build it in the current mixed state and it will produce a binary that contains parts from both languages and works as a Fish shell.

I still rather write Bash scripts or just bust out Python or Perl, but I definitely prefer Fish for terminal usage.

one of the fastest shells out there will probably get even faster. ever since I switched from zsh to fish, my terminal productivity has gone up a lot.

I don't think C++ and Rust has that big of a performance difference. They are probably neck to neck with no clear winner.

In Rust it’s easier to leverage external crates/libraries that are more optimised than rolling your own. This can lead to performance increases (I don’t know if fish is doing this)

C programs without libraries tend to be really fast for the opposite reason. So this isn’t persuasive even though it seems like it should be true

This is only true for very simple single-threaded code. Once you’re doing anything the compiler can’t trivially optimize, you’re using libraries, getting left far behind by the Rust program, or both.

Since most industrial software is in C and C++ an example would be more convincing.

doing anything the compiler can’t trivially optimize

What does compiler optimization have to do with libraries?

single threaded

Plenty of C and C++ programs are multi threading using a system api like pthreads.

Since most industrial software is in C and C++ an example would be more convincing.

Agreed- do you have any to back up your original claim?

> doing anything the compiler can’t trivially optimize

What does compiler optimization have to do with libraries?

One of the most common reasons to use a library is because it has optimizations you want to use. For example, performance-sensitive programs link against OpenSSL even if they’re just using a couple of hash functions because the SHA-256 function you copied into your codebase won’t have their assembly backend or use of processor intrinsics. Repeat for string searches, pattern matching, all kinds of math, etc. where people have written libraries with pricessor-specific assembly, SIMD intrinsics, etc.

Plenty of C and C++ programs are multi threading using a system api like pthreads.

Yes - and the long history of bugs related to that is why many stayed single-threaded because it was more work than the author had time for to make the code thread-safe. Rust’s “fearless concurrency” talk isn’t just marketing but a reaction by people with a lot of experience in the area realizing just how much more they used concurrency when it was easy to do correctly rather than infamously hard.

examples

Linux, Maya, Photoshop, Chrome

OpenSSL

When C programmers do crypto they also use OpenSSL, but more for security than performance. Do you have an example of a rust library which is a better substitute for commonly hand written C?

fearless concurrency

Not related to the topic of discussion about performance gains of using libraries.

I have never experienced difficulties adding high (HPC level) performance libraries into my code. Can you elaborate?

It definitely makes a difference around threading; exposing thread safety in function API makes it easy to understand where/how you can add threading support, or what is preventing it.

The real difference is in compile-time abstractions. It's much easier to organize things in Rust, especially with a borrow checker. You can confidently remove overhead while being sure that it'll continue to be memory safe.

With some strong design decisions, and with modern C++ facilities, you can make sure that your code never does funny things in C++, too.

Yes, Rust's BC is a great technology, but C++ is not a runaway nuclear fire which contaminates and melts everything around it when it comes to sanity and security.

Yes, Rust allows you to run amok by not allowing it, but you can design a good foundation to your code which prevents tons of problems out of the gate in C++, and this is without using smart-pointers and other things available to you.

but you can design a good foundation to your code which prevents tons of problems out of the gate in C++,

And there is an almost perfect overlap between the people capable of pulling off a rewrite like this and the people who would write a good C++ foundation in the first place.

C++ is not a runaway nuclear fire which contaminates and melts everything around it when it comes to sanity and security.

Not sanity, maybe. But security? How much more would it take to declare a nuclear meltdown: https://alexgaynor.net/2020/may/27/science-on-memory-unsafet...

Yes, Rust's BC is a great technology, but C++ is not a runaway nuclear fire which contaminates and melts everything around it when it comes to sanity and security.

I never meant to imply such a thing. However, Rust is simply a much better-defined language. Moves do what you expect, smart pointers work like you'd expect, there are not 3 (or 5) different types of constructors you have to understand and implement properly if you want to build your own abstractions. You don't have to remember where to put std::move, you don't have to remember what happens to the variable after you move out of it, you don't have to use the awful variant accessor functions if you want a tagged union and so on.

Yes, you can do all this in C++. You can build safe abstractions. You can build layers on top of the STL to make it less awful. You can make use of modules and #embed and other nice new features that C++ is getting. But you need to be a lot more careful compared to Rust, because the compiler is not going to help you.

The advantages of Rust come when trying to do complex things with concurrency, and wanting to know that they're either going to be safe, or fail to compile.

Consider the now-classic "Fearless Concurrency" post about Stylo: https://blog.rust-lang.org/2017/11/14/Fearless-Concurrency-I...

Strictly, C++ pretty much has to be no slower than Rust. But it's certainly possible that a regular human might in practice write higher-performing parallel code when using Rust.

At the same time Rust forbids some practices pushing a coder to choose another path. To name a specific example, try to roll out your own dynamically sized type type or a self-referential one. Such small things could contribute to performance too.

Yes, there are definitely things one may do in C++ that aren't possible in Rust. In the absence of bugs, I think you could pretty much recreate any Rust program in C++ and have it be just as fast.

Of course, you can't assume the absence of bugs in real world software. The C++ version would lack any of the safety added by Rust's compile time checks.

Is shell performance a limitation for some workflows? I naively think of it as some glue which shunts the work elsewhere. Maybe in the efficiency of pipe? I suppose those lunatics who write full programs in bash?

I will always take faster code, but that seems like an odd bulletpoint to highlight given the many quality of life improvements fish has to offer.

Kind of, yeah. When I used zsh, oh-my-zsh made the shell startup a lot slower, even though I wasn’t going crazy with the customizations. Opening a new terminal tab would take a couple seconds to get a prompt. My current fish setup is imperceptibly fast to start.

Those little delays add friction. Not an intolerable amount, sure, but it’s there. While I switched to fish for other reasons, that alone would keep me from going back now.

I once spent a while adding timings between each item in my zsh config files to find what was slowest.

The nvm init was the worst of it, so now I have an alias for `$NVM_DIR/nvm.sh ] && \. "$NVM_DIR/nvm.sh" && \. $NVM_DIR/bash_completion` I just run manually before using nvm, rather than letting it slow me down half a second or whatever it was every time I open a new terminal.

It's worth poking around if things get slow enough to be a pain. There's a decent chance it's 1-2 things that you don't even care about all that much.

fish actually has a built-in profiler for its startup that can be enabled via an argument:

--profile-startup=PROFILE_FILE > Will write timing for fish startup to specified file.

zsh has the same; just add "zmodload zsh/zprof" at the top and "zprof" at the bottom of your zshrc.

You might check out rtx[1]

Its an asdf[2] rewrite, in rust, that can do most of the things nvm can

[1] https://github.com/jdx/rtx

[2] https://github.com/asdf-vm/asdf

Kind of, yeah. When I used zsh, oh-my-zsh made the shell startup a lot slower, even though I wasn’t going crazy with the customizations

A lot or most of that wasn't zsh being slow though, but oh-my-zsh calling out to external programs (like git to get powerline info and such) and them being slow.

True, but I can get the same functionality with fish without the slowdown. I don’t think zsh is bad. I understand what was making it slow. I’m just glad I can have all of those niceties plus fish’s speed.

See my other comment from earlier today[1] – launching 230 external programs (even very simple ones, /bin/[ in this case) takes about half a second. Since I would rate anything as >100ms to be a "noticeable delay" you have a "budget" of about 40 external processes, probably a bit less in real-world conditions (depending on which utilities you're using and what it's doing).

And that's 40 C processes. "python empty.py" takes about 30ms to 40ms due to the interpreter startup time; empty C file is about 2ms, "sh empty.sh" is about 4ms.

Please think that "launching external programs is fast and modern computers are fast, so why worry about this?", and that's true, but it's also not. Add up a few dozen and turns out it's actually quite slow.

[1]: https://news.ycombinator.com/item?id=38421199

If it takes 5 seconds for your prompt to display or become interactive, yes.

As far as I'm aware, most of the other quality of life improvements can be reproduced in zsh with plugins. This performance bulletpoint makes me want to consider switching back to fish from zsh.

This. Also completion performance.

I would not bet on this!

from the link :

  We have no real idea what the performance of the finished port is going to be. Early results are encouraging, but it is entirely possible this will be ~20% slower in some cases.

This part also seems relevant:

Fish also uses threads, for things like the autosuggestion and syntax highlighting, and we would like to have more "threads"/"concurrency"/"run-thing-in-backgroundness" (please insert the correct term here, I have never been good at these things). One large project has been to run multiple fish builtins and functions "at the same time", to enable things like backgrounding functions (ideally without using "subshells" because those are an annoying environment boundary that shows up in surprising places in other shells), and to simply be able to pipe two builtins into each other and have them actually process both ends of the pipe "simultaneously".

C++ offers few guarantees on what can be accessed from which thread. @ridiculousfish has been trying to crack this for years, and hasn't been confident enough in his solution. We want a tech stack that helps us here, and C++ doesn't.

The initial rewrite may be no faster, or possibly slower as stated, but long-term they'd be able to safely add more concurrency or parallelism that they weren't able to before.

I completely agree like someone cleverer than me said :

  It is easier to make correct program faster than making a fast program correct!

one of the fastest shells out there will probably get even faster […] my terminal productivity has gone up a lot.

Surely you’re not typing and managing jobs at a pace where C vs C++ vs Rust vs $lang matters… or I’m missing something about what fish is bringing to the table.

More on the motivation behind the rewrite.

https://github.com/fish-shell/fish-shell/pull/9512#issuecomm...

It is kind of telling that they thought porting to c++17 is "too much of a hassle" but rewriting everything in a new language isn't! Sorry, you're never ever going to beat "we did it to chase the new shiny" which is the story across the dev world.

That said, I don't use fish and wish them the best

They said that they thought it wasn't worth updating from C++11 to C++17. In their defense, C++17 adds only some incremental changes. There isn't much to port anyway.

Although I don't disagree with this:

Sorry, you're never ever going to beat "we did it to chase the new shiny" which is the story across the dev world.

I don't think that's the case here. Rust offers more advantages over C++11 than C++17 or C++20 do.

For people who don't want to find the relevant part of the original post, this is it:

We moved to C++11 in 2016 (some quick calculation shows that's 5 years after the standard was released), and we're still on C++11 because the pain of upgrading is larger than the promises of C++14/17.

But why would they need to "upgrade" to C++17/20 in the first place? If you don't need any new features, you can just stay at C++11. (C++11 vs. C++98 is a whole other category.)

So why exactly did they need to rewrite it in Rust?

So why exactly did they need to rewrite it in Rust?

The root comment links to a post literally about this, the post you're replying to lauded its content. Reading it sounds like a good idea. You can then still disagree, but the question you've asked is definitely answered there.

You're right. I got confused by the C++17 "upgrade" argument (which doesn't make any sense to me).

The following point made me laugh, though:

Being written in Rust will help fish continue to be perceived as modern and relevant.

Is this tongue-in-cheek or meant seriously?

Is this tongue-in-cheek or meant seriously?

Let’s assume it was originally written in FORTRAN. Would you think it was a joke or serious?

The writing is on the wall for C++. They’re a little ahead of the curve, but rewriting in a well-liked modern language will likely benefit the health of the project’s contributor pipeline for years to come. There aren’t a lot of people choosing to learn C++ (or FORTRAN) these days.

Fortran has a modern version that codes are written in today in the scientific world. Heck, I write fortran myself and did so last night now that I realise it.

I think you’re missing the point.

Do users still exist? Sure. Is it a healthy, thriving ecosystem with a promising future, increasing mindshare, and serious advantages over the alternatives? Not so much.

I believe it's meant seriously in that being written in Rust it's more likely to attract contributions from people doing it in their free time.

I have only read the discussion when the rewrite was first proposed, but I believe that was mentioned as an advantage.

being written in Rust it's more likely to attract contributions from people doing it in their free time.

Ah, because there are no Rust jobs? duck

I'll take the bait.

The increasing number of Rust jobs ask you for previous Rust experience, if only on open-source or hobby projects.

The fishshell.com website starts with:

Finally, a command line shell for the 90s

When in doubt, assume the Fish team is being tongue-in-cheek.

What you're saying

In their defense, C++17 adds only some incremental changes. There isn't much to port anyway.

contradicts what they're saying

and we're still on C++11 because the pain of upgrading is larger than the promises of C++14/17.

I wouldn't understand what "pain" they're suggesting since the "process" of upgrading the codebase to a newer C++ standard is merely done by replacing the -std=c++11 with -std=c++17 in your build flags. In 99% of the cases, you call it a day. In the remaining 1% of the cases, you deal with the warnings the compiler may have issued because of possibly deprecated functions/headers/keywords you may be using in your code. Since there are very few of them (https://isocpp.org/files/papers/p0636r0.html#removed), and are mostly quite a niche or very old legacy, in the worst case it's going to be a matter of a few days it will take you to fix them.

Since C++ (thankfully) pays attention to ABI stability (aka frowned upon "backward compatibility"), the code will still continue to run as before. Tada ... you're done.

P.S. C++17 is more than just an incremental release (https://isocpp.org/files/papers/p0636r0.html) and offers a lot of goodies both in the language spec (e.g. guaranteed copy elision, fold expressions, compile-time if, structured bindings, compile-time lambdas, improved TMP etc.) and library spec (string_view, variant, optional, any, read-write locks, polymorphic allocators, filesystem support etc.).

However, that does _not_ obligate you in any sense to all of the sudden rewrite your whole codebase (!) with new fancy C++17 features - the secret sauce in mature dev environments is that you do it incrementally and where you see fit.

Fish authors decided to take a drastically different route, and that's totally fine, it's their code after all, but in the absence of quantifying the "pain" or the lack of "promises" I call it a FUD.

What you're saying contradicts what they're saying

I don't think so. The fact that there are only incremental changes would mean that any sort of effort at all in updating may make the update considered to be not worth the effort.

P.S. C++17 is more than just an incremental release (https://isocpp.org/files/papers/p0636r0.html) and offers a lot of goodies both in the language spec (e.g. guaranteed copy elision, fold expressions, compile-time if, structured bindings, compile-time lambdas, improved TMP etc.) and library spec (string_view, variant, optional, any, read-write locks, polymorphic allocators, filesystem support etc.).

Some of those things I hadn't considered, but the majority of all those things I would consider to be incremental changes, even if they're good and useful changes.

I call it a FUD.

They've given their reasoning for why they decided to rewrite it in Rust instead of updating their C++ code to a more recent standard. They're not saying C++ is worthless, or that no one should use it; they're just giving their opinions on why they made the decisions they made. And you're right that without quantifying anything, it's obviously just their subjective opinions. It's not like the authors of the Fish shell are on a mission to discredit C++ by rewriting it in Rust, which is what "FUD" implies to me.

I don't think so.

It does because you claim that the changes are "incremental" and they claim that there's an (unsubstantiated) "pain of upgrading". These two formulations are in contrast one to another. The pain of upgrading would be something that is substantial or complex and long-lasting. Incremental OTOH is considered something more trivial and something that can be done in much less time. Switching to C++17 is an incremental change but not an incremental release.

Since they have not given any substantiated reason for "we're still on C++11 because the pain of upgrading is larger than the promises of C++14/17", I'm afraid that this is a FUD by its very own definition:

    FUD is generally a strategy to influence perception by disseminating negative and dubious or false information, and is a manifestation of the appeal to fear.

I hope it's clear now that giving false information with a negative appeal without substantiated content is not how it's done when it's done in good faith. I couldn't care less how they write software and in which language but bad arguments are bad arguments.

It is kind of telling that they thought porting to c++17 is "too much of a hassle" but rewriting everything in a new language isn't!

"Too much of a hassle" can mean "too much of a hassle in absolute terms" or "Too much of a hassle compared to what you gain". And porting to c++17 is probably high on the latter...

And this is quite literally the argument put forth by the maintainers. They could upgrade to a newer C++, but it’s a lot of hassle with few benefits.

On the other hand they chose to port to Rust which is likely an even bigger hassle but comes along with a bunch of benefits that make planned future development significantly easier.

Wow, thank you for posting this. I’ve been kinda curious to try Fish over the years, but seeing this message from one of the maintainers – how mature and well-reasoned, how both charmingly joking and firmly assertive it is – now really gets me thinking that I’ve been missing out on using an excellent project.

I used bash heavily for years but decided I’d spent enough time maintaining dotfiles & working around quirks around the middle of the last decade. I looked at a few alternatives but since fish had everything I wanted builtin, I decided to try it for a while. After the first day, I decided the experiment was over and switched everywhere. It’s great software both for how much it does and how little it forces you to think about how it works as opposed to what you’re trying to do.

As someone who uses zsh + spaceship, what benefits would I gain moving to fish as my main shell? I don’t write many bash scripts so the scripting side is less of a concern.

Eh, you can probably get very similar to fish with zsh + loads of plugins, but fish has lots of niceties out of the box (syntax highlighting, autosuggestion based on your directory). I've been using fish + starship with just a fzf plugin and it's got everything I need.

Check it out, it's effective with very little config

Speaking of starship, today I realized that I don't really need any of its features, I just like the way it looks. So I replicated it with pure fish by creating a simple function:

  function fish_prompt
      # Fedora Silverblue workaround
      set -l pwd (string replace /var/home /home $PWD)
  
      echo
      echo -s (set_color -o cyan)(prompt_pwd -D 3 $pwd) (set_color magenta)(fish_vcs_prompt)
      echo -ns (set_color green) "" (set_color normal) " "
  end

When I evaluated my transition from bash to something else over the years, I found fish to be more cost-effective to maintain. Many times, I only need to install the distro package to feel comfortable, instead of loading all my dotfiles and things like oh-my-zsh.

Out of the box, effortlessly, I have autocompletion for commands; puts tabs/indentation automatically whenever I start an ad-hoc multiline command inside a for/if statement; a color prompt; etc.

I use fisher for plugins and now have more functions in my dotfiles, but it has far fewer dependencies — like 3 or 4 plugins. When I tried zsh, I immediately added several plugins with oh-my-zsh to feel comfortable.

I'm evaluating zellij instead tmux for the same reason. Some extra batteries included for free, but staying simple enough to keep.

I switched from zsh to fish a couple of years ago. My setups are very similar in capability but my fish config is about 10% the size. I get nearly everything I want for free and now only use one plugin, fzf.fish

The first time I learned to write my own shell prompt by creating a file named “fish_prompt.fish” in the right directory, then writing a function in it named “fish_prompt” that echos a string to stdout, I was nearly ready to uninstall zsh. Using it ruined me on stuffing stuff into $PS1 and crossing my fingers.

I highly recommend everyone at least try it for a few days, if only to see what kinds of things you’d always taken granted are actually shell-specific. Worst case: you say “this isn’t for me” and go back to bash/zsh/whatever happy that you’re still using your favorite shell.

I like use fish + starship.rs. Which gives a pretty similar experience. Starship can actually share you prompt across different shells so I get the same experience in zsh and fish.

I find I stick to fish more because: 1. has more features that I like out of the box. zsh requires oh-my-zsh to get similar features and that's just another thing for me to install every time. 2. Help menu tab completion. Fish has many completions for common shell apps that will display descriptions from the help menu from just hitting tab. 3. Scripting is much better, while you might not use it much now it much now you don't have a hard corner to cut yourself on if you do need it at some point.

For me it's really about the autocomplete.

Speed!

fish does a lot out of the box that I needed plugins for in zsh. zsh's plugins are slow, so my shell startup was over three seconds. fish starts way faster, even with plugins.

My rationale for picking fish over zsh in 2017 was that it had everything I wanted included, as was quite noticeably faster. Since then I’ve spent almost no time maintaining shell configuration, which has been pleasant.

If you've already set up your zsh as you like it, I don't think there's a huge amount of benefit. The big benefit of fish is that it's pretty much the ideal shell, but straight out of the box, without configuration. Autocomplete just works, navigating the history is easy, it completes commands from your history, choosing a shell prompt can be as simple as opening a browser, etc. You can customise it pretty much as deeply as you need to - just like zsh - but the defaults are good enough that you've got a really nice shell out of the box.

Tbh, little, mostly convenience (best metaphor I can give is installing Sublime Text vs manually installing your vim and your own personal configs). By default it supports a bunch of interesting things, autocomplete, iteractive filesystem navigation, theming (I used oh-my-fish in the past, not sure how it is now), etc.

I like it, I would love to run it as my main shell, but I just do too much scripting for my own good to hop on this wagon.

So, what new features do we get from that? Except that it doesn't work on Windows any more?

Never worked on Windows. The programming language has nothing to do with that anyway.

fish.exe works on Windows just fine if you install it as part of MSYS2, which I prefer anyway to the kludge that is WSL.

Why do you think WSL (WSL2 I presume) is a kludge?

It's an OS inside an OS, you have to setup everything twice (Windows AND Linux) and file system access times always bites you one way or another (with WSL2 it's the IDE on Windows that suffers from accessing the Linux FS). Because it's in a container, localhost does not get resolved to the Windows host, so if you have stuff that's spread across both OS it's also annoying. Also has weird issues with corporate VPNs. And the other day switching to fish shell as default would just freeze the whole container with no way to debug it, WTF? Unlike a real Linux env, it's a black box, if it starts acting weird your best bet is just to trash and recreate it.

I mean, it works and gives a somewhat decent Linux CLI experience but personally I stick with MSYS2 whenever I can so I have a single env to deal with.

I didn't believe all the hate until I watched someone unfamiliar with Linux set up an empty Electron project in WSL.

After we hit the third "Linux"-bug that turned out to only happen in WSL, I suggested to maybe just install Linux in a normal VM. Clunky, yes. But at least it has a clear boundary. Unless you move something across it, everything will just work.

It does.

With Go, it would have worked out of the box on all the targets Go can support.

unless you have to deal with OS specific APIs around process management and tty input, which is the bulk of what makes a shell challenging to implement

As long as you called no platform specific APIs. The same is true of Rust and many (even most) languages. Go isn't magic here.

It does.

It works in Cygwin right now, it doesn't work on Win32 without a compatibility layer. Personally, I don't think of that as "working on Windows", but I concede that that's somewhat up to interpretation on the "on Windows" part. I'm confident that someone is going to get a MingW build working even if it's not official.

With Go, it would have worked out of the box on all the targets Go can support.

That's absolutely disingenious. If you choose to only use the Go standard library , sure. But the same is true for Rust and even for C++. If you don't use any platform-specific APIs most languages will work on any target. Fish, however, does use platform-specific APIs and nothing about Go changes that.

As someone who daily drives fish on ~4 different operating systems and loves it, I've been really excited about this rewrite and blown away by how fast it's been progressing given the size and complexity of the codebase. It will also somewhat lower the barrier to contributing for a lot of people, including myself.

Another thing I'm very excited about, which has received less attention, is the planned future fish rewrite to use UTF-8 instead of wchar_t (typically UTF-16). UCS2 and UTF-16 have been a plague on software and fortunately Rust makes working with UTF-8 a breeze.

wchar_t (typically UTF-16)

Complicating supporting multiple compilers in one code base, sizeof(wchar_t) is 2 bytes in MSVC (UTF-16) and 4 bytes in gcc (UTF-32).

MSVC isn't used on Unix, is it? It is probably not a concern for them, because they do not support Windows anyway.

It's 16 bits on mingw GCC on windows, as that also uses the Microsoft libc. I hesitate to say that it's UTF-16 as it's entirely up to the app to use the type properly.

True, AFAIK, but plenty of cross-platform software supports MSVC for compiling on Windows and clang and gcc for other platforms.

Oh! I missed this, what's the original reason for doing the move? Any insight on this?

Hm, why the downvote? I'm genuinely interested.

Presumably because the second to top comment links this issue, which answers your question: https://github.com/fish-shell/fish-shell/pull/9512#issuecomm...

Thanks!

Why, was the C++ codebase becoming unmanageable?

Becoming?

I was raised in C++ spaguetti code bases, I didn't see a loved and well groomed project until I became a man.

Comment by the creator: https://github.com/fish-shell/fish-shell/pull/9512#issuecomm...

tl;dr

  - string handling
  - threading
  - cmake issues

I've always been curious about Fish, but was turned off by it not being POSIX-compliant. Is there anybody who can share their experiences with issues and workarounds in this area?

Is there anybody who can share their experiences with issues and workarounds in this area?

Just use your Bash/zsh to run Bash/zsh scripts: `bash script.sh`.

turned off by it not being POSIX-compliant.

I was like this when I first started using Fish. I spent some time to learn some different syntax to get away from the POSIX-compliance. Overall, I think the saner syntax of Fish is worth the effort. Now I use the terminal much happier.

Besides, 90% of the time I don't need to follow POSIX-compliant. Many scripts or extensions nowadays support Fish.

For enterprise use, I still stick to Bash/zsh. They're the standard.

POSIX compliance is practically a non-issue for fish as it's not intended to be a replacement for bash/shell scripts. It's intended to be an interactive shell (eg. your daily driver) and to that end, it's quite good.

The downside of this is that you do need to "context switch" a little, especially if you've been using bash/zsh for most of your life. I find it easy to differentiate between when I am using the shell as a user interface and when I'm using bash/sh as an interpreter, but for others that might not be as intuitive.

I've also found that since I've stopped using POSIX compliant shells as my daily driver, I tend to "promote" my scripts to a more general purpose language much sooner than I used to when nearly everything could be done in bash.

Been using fish for a year. Although I don’t like how they break compatibility with standards so that some scripts don’t work (gvm, nvm, etc), it has all the features I would usually set up manually with zsh. After trying to configure zsh a few different machines, I gave up and went for default fish on all of them.

It’s already working so well, not sure what there is to gain by porting to Rust. Security? If something can maliciously tamper with fish, it’s probably already on your system or am I missing something?

They have a variety of reasons to move to rust, as outlined in their original rust discussion[1]. Mostly around finding other contributors, and adding an async/parallel mode they're comfortable with.

[1] https://github.com/fish-shell/fish-shell/pull/9512

I love fish. Switch from ZSH+Prezto to just plain Fish and I'm not missing anything and it's way faster. All the servers have bash and I still write bash scripts, but for day-to-day terminal stuff it's Fish all the way for me.

Fish is such an underrated shell. They make some very strong, opinionated choices (like ditching POSIX compatibility), and the end result is an extremely nice shell. I've been using it for a bit over a decade now, and still love it.

Fish shell scripting is also extremely pleasant to write, it feels like an ergonomic bash.

There are a few warts, as with any project, but none of them have ever actually ever blocked me from just getting what I wanted done.

I’m glad rust is replacing legacy languages like C and C++. It’s time to stop using memory unsafe language ffs!