Regex character "$" doesn't mean "end-of-string"

Folks who've worked with regular expressions before might know about ^ meaning "start-of-string" and correspondingly see $ as "end-of-string".

Huh. I always think of them as "start-of-line" and "end-of-line". I mean, a lot of the time when I'm working with regexes, I'm working with text a line at a time so the effect is the same, but that doesn't change how I think of those operators.

Maybe because a fair amount of the work I do with regexes (and, probably, how I was introduced to them) is via `grep`, so I'm often thinking of the inputs as "lines" rather than "strings"?

Does anyone consider RegEx to be standardised? Moving to a new context is always a relearning exercise in my experience.

I can hear thousands of bad hiring manager's adding 'How do you match the end of a string in a regex?' to their list of 'Ha! You don't know the trick!' questions designed to catch out candidates.

The new-line character is an actual character "at the end" of the string though so it makes sense that $ would include the new-line character in multi-line matching.

Special misery case: Visual Studio supports regex search, where '$' matches \n.

The end of line character is usually the standard Windows \r\n.

Yes, that means if you want to really match the end of line you have to match "\r$". So broken.

this is mostly due to the different types of regex and less about it being platform dependent. $ was end of string in pcre which is the "old" perl compatible regex. python has its own which has quirks as mentioned, re2 is another option in go for example, and i think rust has its own version as well iirc.

Regexp was one of the first things I truly internalized years ago when I was discovering Perl (which still lives in a cozy place in my heart due to a lovely “Camel” book).

Today most important bit of information is knowledge that implementations differ and I made a habit of pulling reference sheet for a thing I work with.

E.g. Emacs Regexp annoyingly doesn’t have word in form of “\w” but uses “\s_-“ (or something no reference sheet on screen) as character class (but Emacs has the best documentation and discoverability - a hill I’m willing to die on)

Some utilities require parenthesis escaping and some not. Sometimes this behavior is configurable and sometimes it’s not.

I lived through whole confusion, annoyance, denial phase and now I just accept it. Concept is the same everywhere but flavor changes.

Note: The table of data was gathered from regex101.com, I didn't test using the actual runtimes.

Has anyone confirmed this behaviour directly against the runtimes/languages? Newlines at the end of a string are certainly something that could get lost in transit inside an online service involving multiple runtimes.

This has security implications! Example exploitable ruby code:

  unless person_id =~ /^\d+$/
    abort "Bad person ID"
  end
  sql = "select * from people where person_id = #{person_id}"

So if you're trying to match a string without a newline at the end, you can't only use $ in Python! My expectation was having multiline mode disabled wouldn't have had this newline-matching behavior, but that isn't the case.

A reproducible example would be nice. I don’t understand what it is he cannot do. `re.search('$', 'no new lines')` returns a match.

In 30 years of developing software I don’t think I ever used multi-line regexp even once.

Seems odd to leave Perl off the list, given it's regex related.

Here's the explanation for $ in the perlre docs:

  $   Match the end of the string                 
      (or before newline at the end of the      
      string; or before any newline if /m is     
      used)

Raku (formerly Perl 6) has picked ^ and $ for start-of-string and end-of-string, and has introduced ^^ and $$ for start-of-line and end-of-line. No multi line mode is available or necessary. (There's also \h for horizontal and \v for vertical whitespace)

That's one of the benefits of a complete rethink/rewrite, you can learn from the fact that the old behavior surprised people.

Why isn’t Perl anywhere on that chart when mentioning regex?

In Lua it's only the start/end of the string

A pattern is a sequence of pattern items. A caret '^' at the beginning of a pattern anchors the match at the beginning of the subject string. A '$' at the end of a pattern anchors the match at the end of the subject string. At other positions, '^' and '$' have no special meaning and represent themselves.

https://www.lua.org/manual/5.3/manual.html#6.4.1

Lua's pattern matching is much simpler than regexes though.

Unlike several other scripting languages, Lua does not use POSIX regular expressions (regexp) for pattern matching. The main reason for this is size: A typical implementation of POSIX regexp takes more than 4,000 lines of code. This is bigger than all Lua standard libraries together. In comparison, the implementation of pattern matching in Lua has less than 500 lines.

https://www.lua.org/pil/20.1.html

This seems like the perfect opportunity to introduce those unfamiliar to Robert Elder. He makes cool YouTube[0] and blog content[1] and has a series on regular expressions[2] and does some quite deep dives into the differing behaviour of the different tools that implement the various versions.

His latest on the topic is cool too: https://www.youtube.com/watch?v=ys7yUyyQA-Y

He's has quite a lot of content that HN folks might be interested in I think, like the reality and woes of consulting[3]

[0] https://www.youtube.com/@RobertElderSoftware

[1] https://blog.robertelder.org/

[2] https://blog.robertelder.org/regular-expressions/

[3] https://www.youtube.com/watch?v=cK87ktENPrI

I'm confused by this blog-post. In the table: what is the reg-ex pattern tested and against which input?

Seriously, just write one unit test for your regex.

Isn't a string with a newline character automatically multiline?

The new line is just empty but not the first line anymore.

By default, '$' only matches at the end of the string and immediately before the newline (if any) at the end of the string.

The rationale was probably "it should be easier to match input strings" and now it's harder for everyone.

The table in the article makes this look complicated, but it really isn't. All the cases in the article can be grouped into two families:

- The JS/Go/Rust family, which treats $ like \z and does not support \Z at all

- The Java, .NET, PHP, Python family, which treats $ like \Z and may or may not (Python) support \z.

\Z does away with \n before the end of the string, while \z treats \n as a regular character. For multiline $ the distinction doesn't matter, because \n is the end.

Really the only deviation from the rule is Python's \Z, which is indeed weird.

Tldr;

$ does not mean end of string in Python.

Structural regexes as found in the sam editor are an obscure but well engineered regex engine. I am far from an expert but my main takeaway from them is that most regex engines have an implied structure built around "lines" of text. While you can work around this, it is awkward. Structural regexes allow you to explicitly define the structure of a match, that is, you get to tell the engine what a "line" is.

http://man.cat-v.org/plan_9/1/sam

I think there's a big opportunity to re-write Regex as a SQL-type language. It's too bad I don't feel like trying.

Cmd-F perl

no matches

Regex would really benefit from a comprehensive industrial standard. It's such a powerful tool that you have to keep relearning whenever you switch contexts.

Totally get the desire, but also feels like last two paragraphs are solvable with

``` re.match(text).extract().rstrip(“\n”) ```

You can also use (?m) to enable multiline processing on PCRE-compatible regexp engines.

The fact that there are so many different peculiarities in different regex systems has always raised the hairs on the back of my neck. As in when a tool accepts a regex and I have to a trawl the manual to find out exactly what regex is acceptable to it.

  > So if you're trying to match a string without a newline at the end, you can't 
  only use $ in Python! My expectation was having multiline mode disabled 
  wouldn't have had this newline-matching behavior, but that isn't the case.

  > 3.206 Line - A sequence of zero or more non- <newline> characters plus a terminating <newline> character.

Of course it’s line. How could it be the end of the string when the matter at hand is defining the string?

Is this a bug?

As for the good old reference implementation (not "Parameter Efficient Reinforcement Learning"):

  my $string = "cat\n";
  /cat$/s  -> true
  /cat\Z/s -> true
  /cat\z/s -> false

Another poor soul trying to solve one problem using regex and now they have two… ;)

I would hold a code review hostage if any file does not end with an empty new line.

My reasoning would be if the file is transmitted and gets truncated nobody would know for sure if it does not end a new line. Brownie points if this is code end has a comment that the files ends there.

The article calls computer languages platforms but the are computer languages. Bash is not included. Weird. I believe the most common use of regular expressions is the use of grep or egrep with bash or some other shell but, who knows. Maybe I am hanging with the wrong crowd.

Interesting

This led to a few serious bugs in Ruby-based apps. Always use \A\z

https://homakov.blogspot.com/2012/05/saferweb-injects-in-var...

https://sakurity.com/blog/2015/02/28/openuri.html

https://sakurity.com/blog/2015/06/04/mongo_ruby_regexp.html

Fun fact: in Postgres, 'cat\n' matches 'cat$' when the so called "weird" newline matching is enabled :)

https://www.postgresql.org/docs/current/functions-matching.h...

If you really want to learn regex, you'll have a hard time piecing it all together via blog posts.

Brad Freidl's Mastering Regular Expressions is a good book to read if you want to stop being surprised/lost.

I'll admit I stopped at the dive into DFA/NFA engine details.

Something I found really surprising about Python's regexp implementation is that it doesn't support the typical character classes like [:alnum:] etc.

It must be some kind of philosophical objection because there's no way something with as much water under the bridge as Python simply hasn't got around to it.

People are confused about strings and lines. A string is a sequence of characters, a line can be two different things. If you consider the newline a line terminator, then a line is a sequence of non-newline characters - possibly zero - plus a newline. If there is no new-line at the end, then it is not a [complete] line. That is what POSIX uses. If you consider the newline a line separator, then a line is a sequence of non-newline characters - possibly zero. In either case, the content of the line ends before the newline, either because the newline terminates the line or because it separates the line from the next. [1]

The semantics of ^ and $ is based on lines - whether single-line or multi-line mode. For string based semantics - which you could also think of as entire file if you are dealing with files - use \A and \Z or their equivalents.

[1] Both interpretations have their merits. If you transmit text over a serial connection, it is useful to have a newline as line terminator so that you know when you received a complete line. If you put text into text files, it might arguably be easier to look at a newline as a line separator because then you can not have a invalid last line. On the other hand having line terminators in text files allows you to detect incompletely written lines.

$ is the regex’s “the buck stops here” symbol. Here at the end of the line. :)

Wait, in non-multiline mode, it only matches _one_ trailing newline? And not any other whitespace, including \r or \r\n? That is indeed surprising behavior. Why? Why not just make it end of string like the author expected?

    >>> import re
    >>> bool(re.search('abc$', 'abc'))
    True
    >>> bool(re.search('abc$', 'abc\n'))
    True
    >>> bool(re.search('abc$', 'abc\n\n'))
    False
    >>> bool(re.search('abc$', 'abc '))
    False
    >>> bool(re.search('abc$', 'abc\t'))
    False
    >>> bool(re.search('abc$', 'abc\r'))
    False
    >>> bool(re.search('abc$', 'abc\r\n'))
    False

POSIX regexes and Python regexes are different. In general, you need to reference the regex documentation for your implementation, since the syntax is not universal.

Per POSIX chapter 9[1]:

9.2 … "The use of regular expressions is generally associated with text processing. REs (BREs and EREs) operate on text strings; that is, zero or more characters followed by an end-of-string delimiter (typically NUL). Some utilities employing regular expressions limit the processing to lines; that is, zero or more characters followed by a <newline>."

and 9.3.8 … "A <dollar-sign> ( '$' ) shall be an anchor when used as the last character of an entire BRE. The implementation may treat a <dollar-sign> as an anchor when used as the last character of a subexpression. The <dollar-sign> shall anchor the expression (or optionally subexpression) to the end of the string being matched; the <dollar-sign> can be said to match the end-of-string following the last character."

combine to mean that $ may match the end of string OR the end of the line, and it's up to the utility (or mode) to define which. Most of the common utilities (grep, sed, awk, Python, etc) treat it as end of line by default, since they operate on lines by default.

THERE IS NO SINGLE UNIVERSAL REGULAR EXPRESSION SYNTAX. You cannot reliably read or write regular expressions without knowing which language & options are being used.

[1] https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1...

perlre Metacharacters documentation states: $ Match the end of the string (or before newline at the end of the string; or before any newline if /m is used)

(/m enables multiline mode)

There are many differences between implementations of regex. To name a few. Lookbehind, atomic groups, named capturing groups, recursion, timeouts and my favorite interop problem, unicode.