One pattern I really like is opening the networks tab of the browser, finding the request I'm interested in and "copying as curl" - the browser generates the equivalent command in curl.
Then I'd use something like https://curlconverter.com/ to convert it into request code of the language I'm using.
In a way curl is like the "intermediate representation" that we can use to translate into anything.
This kind of thing was one of the reasons Visual Basic macros for Microsoft Office was so successful. You can perform actions in Word, Excel, watch the macros they produce then customise themselves to your needs afterwards in code. It is a simple and powerful concept, so good to see it in curl.
I was going to say, there has to be a generic pattern for things like this. Also made me think of AREXX on the Amiga.
There is, but we need to throw away our outdated current programming model. Think Lisp or Smalltalk. There should not be a separation between program written in some language, operating system and shell [1].
You'd simply run:
CURL url: "https://example.com" method: 'post
(Syntax in this example from a Smalltalk-like environment I have been designing, should be understandable enough)
---
1: "An operating system is a collection of things that don't fit in a language. There shouldn't be one." — Dan Ingalls, 1981
Given that we already have the concepts of exported entry points and extern in C, this should already be possible to a certain extent. The only other thing which needs to happen is that ELFs should have the concept of exported data structures, so while "URL" might not be a defined structure in the OS, something like curl can provide it.
Too bad we're stuck with UNIX/POSIX model - you could even take this idea of exported data structures and have the terminal represent data in the preferred users format instead of having tools like jq.
We have dlopen and we can list exported symbols, but we have no information about a function's arguments, ABI and calling convention, so it's pretty much impossible to turn UNIX into a fully late-bound and interactive REPL. Same issue with syscalls.
The only way is starting from scratch, with a novel approach and programming model.
Why can’t you add additional data about the data types of parameters in a separate ELF section? It would be only used by programs that look for it like a specially designed shell.
The calling convention can be assumed to be the same as used by the OS/arch combination.
You could, but you then have to recompile the world with this new information stored as a ELF header or something, and good luck if the library is not written in C (so it has its own conventions, ABI, memory model and binary format)
I'm talking about the status quo today, not how you can improve in a perfect world where everybody adopts a better way of doing things.
Implementing an half-baked Smalltalk layer on top of UNIX will not turn UNIX into a Smalltalk environment.
Or you could add some more sections to object files. Just make clang and/or gcc export function signatures, structs, enums, and typedefs
I wonder if one of the existing interpreted languages (python/javascript/ruby/whatever) could maintain a patch for llmv/gcc which did exactly that and in the process make the most incredible seamless integration ever between itself and C (and also C++ now with its ABI stabilizing!)
PowerShell?
What you're looking for exists, it's powershell.
Invoke-WebRequest -Method 'Get' -Uri 'https://example.com'
That you can pipe properly by piping objects not strings, e.g. :
Invoke-WebRequest -Method 'Get' -Uri 'https://example.com' | Select-Object -ExpandProperty Headers
Key Value
--- -----
Age 438810
Vary Accept-Encoding
X-Cache HIT
Content-Length 1256
Cache-Control max-age=604800
Content-Type text/html; charset=UTF-8
Date Mon, 29 Jan 2024 15:08:42 GMT
Expires Mon, 05 Feb 2024 15:08:42 GMT
ETag "3147526947+ident"
Last-Modified Thu, 17 Oct 2019 07:18:26 GMT
Server ECS (nyb/1D15)Yes yes, power shell is powerful and really good. I wanted to hate it because it seems too verbose and I don't like the mix of capitals and hashes in names. but the APIs they make available from .Net are pretty phenomenal. Extreme verbosity aside, done day I'm have to seriously learn it
Can't hate it, but I can't love it either. I wish some other more "normal" languages (Ruby? Python?) had a better "shell" story and dotnet integration.
Can you copy and paste that snippet as is into a C# program? This is what I meant.
Powershell has a high level object model but it doesn't make Windows itself anymore programmable and interactive, as a whole than zsh does for Linux. It is no Lisp Machine.
You might find this work and talks on liballocs by Stephen Kell interesting. The pitch is how to enable smalltalk like reflection for unix as it exists today. https://www.humprog.org/~stephen/#works-in-progress
While it doesn't appear to have been updated in many years, Microsoft built a similarly useful tool[1] that lets you browse the structure of a given Office document and see C# code that generates various components of it.
[1] https://github.com/dotnet/Open-XML-SDK/releases/tag/v2.5
Or the excel side by side xaml/UI of the wpf editor in visual studio.
In an enthusiastic tone of an AI enthusiast: Thankfully, now that we have ChatGPT, this feature is obsolete and the curl executable doesn't have to contain half-baked quines in it anymore!
It's not a quine though, as it's not producing curl itself
In Italian, it's common to append 'ino' or 'ina' to something when you want to imply it's a smaller or cuter version of itself.
So 'quinino' might work here.
If you want something with more rhyme, poco-quine can also work, I guess, pq for short.
libpq is taken by PostgreSQL though ;)
One could argue it's a half-baked quine.
mm parbaked quine, just pop it in the compiler add a main driver and serve with some gengetopt for a delicious webcrawler!
I'd never change --libcurl and gengetopt[0] with some output from some artificial thingy which babbles semi-truths which doesn't understand what it's doing.
They are deterministic tools which does what you want in a battle tested way, and will let me sleep well at night, which is an underappreciated feature of mature programs.
First I've heard of gengetopt, thanks!
Hurray for deterministic tools! Yes, there is no reason to risk using an LLM when you have a code generator that you can trust.
Please also add support for different so we can write
—-libcurl example.py
Its quite effective to use ChatGPT to convert C code to Python ..
... or "curl calls to python webscraping". Although having to look for the right version of libraries with different number names and therefore different feature sets was tedious to do manually, AI might just guess fast and sometimes even right as to which import to use.
Tools already convert a curl command line to a Python requests script.
Try https://curlconverter.com/ - "Convert curl commands to Python, JavaScript and more"
You can probably get pretty far with the generated C code, then look up the same curl options in the Python bindings:
Libcurl is a c/c++ library... Just use Requests
I wish something like this was possible with ffmpeg
The compiler has been thinking about the answer for a few decades ...
Before I found out that statically compiled ffmpeg was a thing, I was trying to compile ffmpeg on a 2010 Mac.
It was a multi-day process and every time like two days in I'd hit some snag.
Not our problem, we just need to write the feature request ;)
But yes, perhaps as a follow-up, we would need to ask for
1. a speed-up of the process or
2. find a way to look up and redistribute existing conversion flows
That would be awesome
Seriously!!
Having a flag in the command line interface that spits out the source code of the program doing the same stuff as your command is pretty cool. It's like lifting the hood and showing you what's going on. This not only helps you get a better grip on how things work but also lets you make changes to fit your needs. You can tweak or add stuff based on what you want, making the whole experience user-friendly. It's all about giving users the power to customize things their way.
It's also just great documentation for a programming library. Like, if you're using libcurl and realize you need to do a range request (or whatever), or "copy as curl" from browser network tab, you can just do it on the command line and add `--libcurl` and find out exactly how to do that with the C library. It's the bee's knees.
100%
I wish this also worked for Gnome settings, network settings, firewall config GUIs, and anything that can do things using CLI commands.
As if the world needed more unsafe C code connected to the internet.
Rust-evangelism aside, I guess one can run the program under ltrace to achieve almost the same result.
As if the world needed more unsafe C code connected to the internet.
Assuming (as usual) that the code generation is solid because of curl’s reputation: why not trust it? It would be pretty bad if the generator could emit memory-unsafe code. (I don’t know.)
For a trivial example, the code just calls curl_easy_init, a bunch of curl_easy_setopt, curl_easy_perform to do the work, and curl_easy_cleanup. (It leaves comments like "CURLOPT_WRITEDATA set to a objectpointer" in a comment block on params for which "You may select to either not use them or implement them yourself" - that's presumably where you are going to write your own unsafe code :-)
libcurl is used in billions of situations.
Fair point about memory allocations in C, but often alt languages rely on other people's code which you'd implicitly trust to do the same thing. So then it becomes an argument of testing and trust. All the same, you trust strangers code or you write your own.
Wait, this option would give you a compilable C program that would replace the need for scripting around the original curl call!?
Yes, it has always been possible with libcurl but you had to write the boilerplate code yourself.
The idea is that you can copy-paste the C code into an existing program or at least use it as a reference to know exactly which libcurl API calls are needed to replicate the curl call.
One convenient thing in browser web developer tools is the ability to copy requests from network tab as either curl commands or even as javascript code. I love to see more this sort of things!
And was curl is such a "standard" to represent a request there's also many tools converting that curl output into native code (Like Go) already, which makes it very fast to reproduce something without manually having to set all the flags. I'm always happy this feature exists without even needing third party extensions in the browser.
Not sure if I like patching more and more parameters into the executable for gimmicks like this. Would be also work when you exchange the executable name instead of addding a parameter. Like ...
curl-as-librucl https://www.example.comAdded in 7.16.1. > Jan 29 2007
so this isn't a new trend...
To compile it you'll need to tell it to link to libcurl, e.g. with -lcurl on gcc:
curl https://ifconfig.me --libcurl ip_fetcher.c
# Output: your ip address, and a file ip_fetcher.c
gcc -o ip_fetcher ip_fetcher.c -lcurl
# Output: no errors, just a file ip_fetcher
./ip_fetcher
# Output: your ip address
code generator that isn't LLM based - how quaint :)
The submission title should be two hyphens (--), not an em-dash and a hyphen (—-)
Someone should buy libcurl.com and make it return the source code to generate a request to itself. Bonus points for setting the same headers and options as the triggering request.
I remember this option was mentioned in a 3 hour video[1] where Daniel Stenberg himself went through most of the curl command line options.
What an awesome idea! That's kind of like a library documentation but alive. Very cool.
Nice. Daniel Stenberg is really careful to details and dev / user experience. This level of polishing is astonishing.
Shameless promotion: Hurl [1] is an Open Source cli using libcurl to run and tests HTTP requests with plain text!
We use libcurl for the super reliability and top features (HTTP/3 for instance) and we've added little features like:
- requests chaining,
- capturing and passing data from a response to another request,
- response tests (JSONPath, XPath, etc...)
There is nice syntax sugar for requesting REST/SOAP/GraphQL APIs but, at the core, it's just libcurl! Using verbose option, you can grep the curl commands for instance. (I'm one of the maintainers)
[1]: https://hurl.dev
I love this! Very nice transition for people to move from commandline to the library when they need to!
I think I saw this first with the ASM bytecode toolkit: https://asm.ow2.io/javadoc/org/objectweb/asm/util/ASMifier.h...
Possibly via the Eclipse extension: https://marketplace.eclipse.org/content/bytecode-outline
Tried this out, it doesn’t seem to work at all with a good few flags - notably —upload-file
It says everything about C when you need a code generator for something like this. This is why development in python or node.js is so much faster.
Useful idea
Can someone give an example use case for this utility? I'm really curious to learn how would you use this?
I use the browser equivalent of this all the time to generate javascript code for requests. It’s very cool to see this for C and hopefully other languages, too.
This is really nice, but a feature that some applications implement that I wish was available in the library is some way of outputting curl command line reference which is equivalent of the requests being made. In-fact, I often find myself, MITM'ing myself using burp-suite and "copy as cURL command" feature for exactly this.
curlconverter.com looks amazing, instant bookmark – thanks!
I also use the browser's 'copy as curl' function quite frequently as it's so convenient, having all auth and encoding headers set to _definitely_ work (instead of messing around with handmade, multi-line curl command lines)
Be aware that online service like this one might log your request which could have sensitive data, I'm not saying it does, but those websites give me the creep
I agree that it's possible, and that the majority of utility websites do use a backend to provide their utility, but it seems curlconverter.com doesn't make any requests to a website to convert and instead does so in javascript.
It would be nice if more sites offered themselves as PWAs that worked when you set "throttling" to "offline" in the dev menu, so that you could ensure that no data is leaving the browser when working with sensitive info.
Maybe that would be a nice browser plugin. Something that blocks any further requests. I guess it would work similarly to ad blockers, only once enabled blocks everything.
True. Luckily it's open source. You can do `npx curlconverter --language go example.com` behind a firewall after downloading the npm module.
I kinda wish the address bar in any browser had an "advanced" popout menu that's basically a curl frontend with all of its bells and whistles. Basically move that functionality from the dev tools.
This is pretty interesting. It's not like HTTP needs an intermediate representation, but since cURL is so ubiquitous, it ends up functioning as one. cURL is popular so people write tools that can export requests as cURL, and it's popular so people write tools that can import it.
The benefit of curl over raw HTTP is the ability to strip away things that dont matter.
Eg an HTTP request should have a client header, but they're typically not relevant to what you're trying to do.
curl is like an HTTP request that specifies only the parts you care about. It's brief, and having access to bash makes it easy to express something like "set this field to a timestamp".
HTTP requests don't need to have any specific headers, and, if anything, curl will only add ones for you.
But specific HTTP requests might. Like cookies, an accept header, or anything.
I've used a similar tool as part of API logging, filtering out the signature on the bearer token... It's useful with request id for dealing with error reporting.
For R users, the httr2 package (a recent rewrite of httr) has a function that translates the copy as curl command to a request:
https://httr2.r-lib.org/reference/curl_translate.html
oh WOW! this is super useful. Thanks for the pointer!
Great Chrome feature! For those who haven't seen it, it also includes copy as PowerShell, fetch, and Node.js commands.
Yeah, that has made my life so much easier when troubleshooting an API endpoint. I can tweak the request params to run against a local instance as well as pipe through jq for formatting etc.
In curlconverter.com clicking on "C" redirects you to the --libcurl option documentation page instead of generating a C snippet. Wouldn't a more user-friendly way be to still generate a C snippet, but to mention that it can be done with the --libcurl option too?
I've used 'copy as curl' a bunch. Often find I have to append --compressed so the command line will provide the uncompressed output.
Postman can also import in "curl format", so yeah, the representation works.
This is such an interesting and true observation. Anytime something isn’t working with an endpoint, first step is “can you get it to work with curl”.