Ripgrep is faster than grep, ag, Git grep, ucg, pt, sift (2016)

I wrote a bash version of this:

  function frg {
    result=`rg --ignore-case --color=always --line-number --no-heading "$@" |
      fzf --ansi \
          --color 'hl:-1:underline,hl+:-1:underline:reverse' \
          --delimiter ':' \
          --preview "bat --color=always {1} --theme='Solarized (light)' --highlight-line {2}" \
          --preview-window 'up,60%,border-bottom,+{2}+3/3,~3'`
    file="${result%%:*}"
    linenumber=`echo "${result}" | cut -d: -f2`
    if [ ! -z "$file" ]; then
            $EDITOR +"${linenumber}" "$file"
    fi
  }

Vim is almost broken for me without fzf+rg. Feels like I’m manually grinding coffee instead of using electricity.

Oh. Thanks for the tip. This might make me finally embrace powershell. I’ve been using WSL+zsh+fzf as a Windows CLI for continuity with day job Mac tools, but git CLI performance is only usable inside the WSL file system.

With fzf you can add lots of files to git while skipping some if you want:

    fza = "!git ls-files -m -o --exclude-standard | fzf -m --print0 | xargs -0 git add"

That alias as well as fzf+fd really speed up some parts of my workflow.

Oh and shameless plug for my guide on what to include in your zsh setup on macOS: https://gist.github.com/aclarknexient/0ffcb98aa262c585c49d4b...

Infact I would recommend a step further to integrate rip-grep-all (rga) with fzf that can do a fuzzy search not just on text files but on all types of files including pdfs, zip files. More details here [1]

[1] https://github.com/phiresky/ripgrep-all/wiki/fzf-Integration

Not bad. For those who want to try it, install all prereqs with: choco install fzf bat ripgrep

How do you scroll the preview window with keyboard ?

This is a gem; thank you.

This is pretty much my exact use of ripgrep, too. I use it as a starting point to zero in on files/projects in a several-hundred repo codebase, and then go from there....

I use it neovim with fzf search.

Thanks for this!

grep is a general purpose tool for searching for text in all types of files, baked into the standards for UNIX. Some programmers use it to search source code. Other people use it for other types of text searches that have nothing to do with source code, they rely on it in scripts, they don't use it as part of a text-based programmer UI, they rely on it to never crash, etc.

ripgrep is a specialist, opinionated tool, designed primarily to search through source code repositories.

There's not much you can add to general purpose text search to make it faster; you can make it use mmap() at the risk of it crashing on truncated files, you can reduce the expressiveness of regular expressions so they can be computed faster. You could throw out general support for all locales and charsets and hardcode support for only UTF-8 / UTF-16, but you shouldn't.

There are multiple alternatives you can already use as an alternative, like ripgrep. What are you proposing, switching out the command `grep` for another utility?

Sounds like that could introduce a ton of breakage, for little value. People who want a faster grep will use a different thing, while people who use grep can continue to use it. Sounds like an ideal situation already.

Honestly: I rely on my shell scripts working and any "grep replacement" has to work with all the old crusty shell scripts out there, likely including ones that use odd "quirks" and GNU options.

If you want to innovate in this space, why sign up for all that? Invent a better wheel, and if people like it, they'll migrate over time.

I remember using ag in the old days, and I use rg now. But there's things rg does by default that I don't like at times... so I go back to old fashioned grep.

rg is at the point where many programmers use it. I think it is on its way to becoming one of those "standard tools". It needs... another 5 years?

When POSIX has a rg standard... we'll know ripgrep "succeeded" and teargrep will soon come into existence ;)

I guess it is because after decades of use, grep has probably been fixed to handle lots of user cases that the new tools don´t handle because they haven´t found them yet.

For the same reason the 40yo chair I currently sit in is not being replaced with Razer UltraSeat XR3000-A. It's comfortable, fits the workplace around it, and there's no reason for getting a replacement and rebuilding everything. (Partially because a Razer-like chair already stands nearby taking care of my clothes, but that's where the analogy ends.)

Lack of interest maybe? I don't use grep, I use an UI that lets me click and jump to a file. Or the builtin search in my IDE.

Complete guess: it works just fine for 99.9999% of users, but greater than (1-0.999999)% chance that it would break compatibility or have a bug. Anyone who would need the performance gain would know about specialized alternatives.

There's like a whole host of things you could use to explain it. Inertia. Compatibility. Resistance to change. Innovator's dilemma. And so on. (I do not say any of these things pejoratively! All of those things apply to me too.)

With respect to compatibility, see my FAQ on the topic: https://github.com/BurntSushi/ripgrep/blob/master/FAQ.md#pos...

These benchmarking results are seven years old, so perhaps it has been.

My entirely anecdotal and unscientific impression is that rg and grep perform similarly on Linux (though rg has nicer defaults for searching through source code). The old version of grep that Apple preinstalls on the Mac was slower last time I checked though.

Someone designed unix based on the idea that some system functions are both core OS functions AND tools for human use. That leads to some bizarre outcomes decades later like "there must be a program called xyz that accepts these arguments and works exactly like this".

What? Git grep is all you ever need in my experience, and it's ~faster than~ (edit: as fast as) ripgrep when searching a git repo.

I don’t think there’s been a point to using `git grep` since ack started parsing gitignore. As far as I’m concerned the use case of `git grep` is to search into non-checked-out trees (by giving it a tree-ish). And it’s not super great at that, because it searches a static tree-ish, so pickaxe filters are generally more useful (though they’re slow).

Once again mercurial has/had more useful defaults, `hg grep` searches through the history by default, that’s it’s job.

So I was casually searching for "ugrep vs ripgrep" articles, when I stumbled upon a couple reddit posts where apparently the authors of ugrep and ripgrep seemed to have a multi-year feud on reddit, eg. https://www.reddit.com/r/programming/comments/120wqvr/ripgre...

So weird. I mean, it's just about some open source tool, right? :-/

I’m a die-hard ripgrep fan, but just recently found ugrep looking for one feature that ripgrep lacks: searching in zip archives (without decompressing them to disk).

Ugrep has that. In my case, I’m working with zipped corpora of millions of small text files, so I can skip unpacking the whole thing to the filesystem (certain filesystems have trouble at this scale).

I’m grateful for both tools. Thanks to the respective authors!

It's also a drop-in replacement for grep as it supports the same flags and regular expression syntax.

I'm scared that if I start using Google search syntax in my grepping that I'll mostly get results trying to sell me something :)

Thanks for mentioning this.

I think the killer feature is compatibility with existing grep command line switches. Not needing to learn a whole new set of options is quite nice.

Is the TUI better than just sending the results through fzf? For me the configurability and flexibility of fzf would be hard to compete with.

Even if the answer is instant, you have a 50% performance improvement in your search just from typing "rg" instead of "grep"!

From my perspective it's a no brainer. I don't HAVE a grep (because I don't have a Unix) so when I install a grep, any grep, reaching for rg is natural. It's modern and maintained. I have no scripts anywhere that might expect grep to be called "grep".

Of course if you already have a grep (e.g. you run Unix/Linux) then the story is different. Your system probably already has a grep. Replacing it takes effort and that effort needs to have some return.

A few years ago I worked on a Solaris box that would lock the whole machine up whenever I grepped through the log files. Like it wouldn't just be slow, the web server that was running on it would literally stop serving requests while it was grepping.

I never worked out how that could be happening.

Multi-GB log files. Even with LC_ALL=C, grep is painfully slow.

On extremely slow systems, such as Windows. There I can search in multiple repos only with rg.

I am amused by this comment, because it shows a dramatically different type of thinking. I have probably have thought "I wish this was faster" for nearly everything I do on a computer :)

Maybe I'm missing something but I only use it with AND conditions (usually in the form of 'foo 'bar and it only matches lines with foo AND bar both present)

That’s one of the reasons I made this actually https://github.com/boyter/cs

I wanted and boolean syntax mixed with fzf instant search. It’s not as fast as ripgrep of course but it’s not solving the same problem.

If you don't mind using PCRE, you can do it. For example:

    rg -P '(?=.*pat1)(?=.*pat2)(?=.*pat3)'

Author of ripgrep here.

Looking at your regex---just by inspection, I haven't tried it, so I could be wrong---but I think you can drop the --pcre2 flag. I also think you can drop the second and third \b assertion. You might need the first one though.

Addendum to my comment above:

Ripgrep can be used quite easily with the project.el package too that comes out of the box in Emacs. So it is not really necessary to install an external package to make use of ripgrep within Emacs. We first need to configure xref-search-program to ripgrep as shown below, otherwise it defaults to grep which can be quite slow on large directories:

  (setq xref-search-program 'ripgrep)

  rg -i --null -nH --no-heading --no-messages -g '!*/' -e foo

Deadgrep (uses ripgrep and evil-collection has a binding) takes me to my happy place -

https://github.com/Wilfred/deadgrep

this is a good option but i still use rg.el for the occasion that i want to search several projects at once or a subfolder within a project, where i would otherwise use ‘rgrep’

Not exactly but yes, it ultimately uses the `memchr` crate [1] which provides SIMD-optimized character and string search routines. But it uses `_mm256_cmpeq_epi8` instead of `_mm256_sad_epu8`.

[1] https://docs.rs/memchr/latest/memchr/

_mm256_:(_epu8

...which is awesome if you can request/install VS Code but not ripgrep.

You can find the rg binary in the VS installation (at least, I can on Windows at my place of employment).

I did not know this that's actually very interesting.

It's not new it has been in vscode for 7 years.

I've always wondered how in world search in VS Code is so fast given it's an Electron app - now I know.

Yeah, the -M command is wonderful (super handy for ignoring minified files that you don't want to see results from, etc), and also great is the -g command (eg `-g *.cs` and you'll just search in files that have the .cs extension).

Also the fact that it is a standalone portable executable can be super handy. Often when working on a new machine, I'll drop in the executable and an alias for grep that points to rg, so if muscle memory kicks in and I type grep it will still use rg.

Author of ripgrep here.

Yes, qgrep uses indexing, which will always give it a leg up over other tools that don't use indexing. But of course, now you need to setup and maintain an index. The UX isn't quite as simple as "just run a search."

But there isn't much of a mystery here. Someone might neglect to use qgrep for exactly the same reason that "grep is fast enough for me" might prevent someone from using ripgrep. And indeed, "grep is fast enough" is very much true in some non-trivial fraction of cases. There are many many searches in which you won't be able to perceive the speed difference between ripgrep and grep, if any exists. And, analogously, the difference between qgrep and ripgrep. The cases I'm thinking of tend to be small haystacks. If you have only a small thing to search, then perhaps even the speed of a "naive" grep is fast enough.

So if ripgrep, say, completes a search of the Linux kernel in under 100ms, is that annoying enough to push you towards a different kind of tool that uses indexing? Maybe, depends on what you're doing. But probably not for standard interactive usage.

This is my interpretation anyway of your wonderment of (in your words) "why people forget the qgrep option." YMMV.

I have flirted with the idea of adding indexing to ripgrep: https://github.com/BurntSushi/ripgrep/issues/1497

this above all true UNLESS you need multi-line matches with UTF8, where ripgrep is not so fast, because it needs to fall back to the other PCRE2 lib

That's not true. Multiline searches certainly do not require PCRE2. I don't know what you mean by "with UTF8," but the default regex engine has Unicode support.

PCRE2 is a fully optional dependency of ripgrep. You can build ripgrep without PCRE2 and it will still have multiline search support.

Why not just make new ones? Regardless of whether I use Conda, Pyenv or virtualenv, I always consider the environments to be disposable.

fastmod is another nice and user-friendly Rust tool for such cases.

In my experience they are all horribly i/o bound and the search takes as long as files load from disk and that's quite long, after that the difference can't possibly be meaningful. When files are in cache search time is dominated by time it takes me to navigate the file system and write the command, and again performance difference can't possibly be meaningful.

In 2016, I'd say speed was definitely a defining attribute. ag has very significant performance cliffs. You can see them in the blog post.

But as I mentioned in my comparison to qgrep elsewhere in the thread, everyone has different workloads. And for some workloads, perf differences might not matter. It really just depends. 900 KLOC isn't that big, and indeed, for simple queries pretty much any non-naive grep is going to chew through it very very quickly.

As for comparisons in other ways, at least for ag, it's on life support. I thought it was going to get removed from Debian, but it looks like someone rescued it: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=999962

The blog post also compares Unicode support, and contextualizes its performance. ag essentially has zero Unicode support. Unicode support isn't universally applicable of course---you may not care about it---but it satisfies your non-perf comparison criteria. :-)

did you try it on big dataset? there is a huge difference actually

I love grep, but ripgrep really improves on it.

Very often, I don't want to look for files that aren't tracked under Git VC, and I'm not looking for matches in binary files, so by default, ripgrep does that, which can cut time by 99%. I used to grep in small dirs, now I can I can ripgrep in my whole home, not that I do it, but I can. That + Sourcegraph on master branch, and it makes searching for any other thing than plain text feel sooo slow (Atlassian Confluence and Jira, Google docs, etc.).

Thank you so much Burntsushi and contributors!

Never will be: https://github.com/BurntSushi/ripgrep/blob/master/FAQ.md#pos...

If you need drop-in compatibility with grep, then use grep. :-)

Discussions:

"Ripgrep – A new command line search tool" https://news.ycombinator.com/item?id=12564442 (740 points | Sept 23, 2016 | 209 comments) - there are discussions related to speed too

"Ripgrep is faster (2016)" https://news.ycombinator.com/item?id=17941319 (98 points | Sept 8, 2018 | 40 comments)

Why not just use :grep in vim and navigate the vim quick fix list?