Launch HN: Sorcerer (YC S24) – Weather balloons that collect more data

Sounds cool. How does it differ from CBOR?

JSON BinPack is space-efficient, but what about runtime-efficiency?

When transmitting data over the Internet, time is the bottleneck, making computation essentially free in comparison.

i thought this was an odd sales pitch from the jsonbinpack site, given that a central use-case is IoT, which frequently runs on batteries or power-constrained environments where there's no such thing as "essentially free"

This looks promising! One of the important aspects of protocol buffers, avro etc is how they deal with evolving schemas and backwards/forward compatibility. I don't see anything in the docs addressing that. Is it possible for old services to handle new payloads / new services to handle old payloads or do senders and receivers need to be rewritten each time the schema changes?

Do you have any info on how your system stacks up to msgpack? (https://msgpack.org/index.html)

Asking because we use msgpack in production at work and it can sometimes be a bit slower to encode/decode than is ideal when dealing with real-time data.

From the OP:

Our payload uses a satellite transceiver for communications

Let's definitely talk, we're using protobufs right now. I'll send an email

Why this over a compact, data-specific format? JSON feels like an unnecessary limitation for this company's use case. I am having a hard time believing it is more space-efficient than a purpose-built format.

It surprised me how popular this message got. I love nerding out about binary serialization and space-efficiency and great to see I'm not the only one :)

If you want to get deeper, I published two (publicly available) deep papers studying the current state of JSON-compatible binary serialization that you might enjoy. They study in a lot of detail technologies like Protocol Buffers, CBOR, MessagePack, and others that were mentioned in the thread:

- https://arxiv.org/abs/2201.02089

- https://arxiv.org/abs/2201.03051

Hope they are useful!

Sure thing! They're made of about 300 grams of polyethylene. Towards the end of their lifespan, we can steer them to an area that's easy for us to drive out and pick them up. The payload has a GPS, which lets us track where they are both in the sky and on the ground.

Right now, most weather balloons fall back to Earth and stay where they land unless someone happens across them (since they can't be controlled and only last a couple of hours).

Here's a video of one of our recent recoveries: https://youtu.be/8DWYLG_95V0

How we make it go up and down is the secret sauce :) I'm a hangglider guy, so I'd love to be using thermals, but I can say that's not how we do it right now

I’d love to know, too. If it was me I’d use a small piezo plate on the side of the balloon to do it the same way hot air balloons do. Heat the gas in the balloon = go up, reverse the polarity to cool it = go down. That would be pretty energy inefficient, though, so hopefully their secret method is better.

Loon used pumps and an interior air ballast like blimps do. So clearly there are a few ways.

Not at all. Look at the growth in human buildings in the most at-risk areas and you’ll see why that number is so big now. It’s only slightly due to an increase in severe weather event frequency / severity.

One detail here is that 1981 dollars aren't 2023 dollars, so to compare they need to be adjusted.

Using [0] $3.5 bn in 1981 would have been worth $11.7 bn in 2023.

Another comment [1] noted (but unfortunately didn't cite) that two years later the damage was assessed at $36 bn, or $110 bn in 2023 dollars.

[0] https://www.usinflationcalculator.com/

[1] https://news.ycombinator.com/item?id=41295116

Yeah, it's a shocking number, and it's just for the US. The global estimates for severe weather are even higher [0], and in places with less infrastructure, the costs are usually more heavily weighted toward human life lost.

Obviously what we're doing can't prevent severe weather from happening, but even very small improvements in accuracy and timelines can have a massive beneficial effect when a disaster does happen. My cofounders and I are all from Florida, so hurricanes are the most visceral examples for us. When hurricanes hit, there are always issues along the lines of "we didn't have the right resources in the right places to respond effectively." Those types of issues can be combated with better info.

[0]: https://www.statista.com/statistics/818411/weather-catastrop...

Definitely a bit of cherry picking. Just 2 years later in 1983 the damages were $36 billion, but that wouldn't make quite as scary of a statement for the website.

Every once in a while things go spectacularly wrong up there and we kick ourselves for not doing a b2b saas

[flagged]

Thanks! Let me know if you want to chase ours sometime.

It’s a literal launch!

Thanks :)

Yeah, I don't have anything of substance to say other than it's really cool to see someone doing something innovative in a niche I'd never really thought of.

In reading the documents google's Loon made available before shutting down (and saying that hydrogen airships were the way to go) [https://storage.googleapis.com/x-prod.appspot.com/files/The%...], the bottleneck on staying up longer was helium loss. Lots of things, mechanical failure etc, caused flights to end early. But the ultimate limit seems to be either helium loss or the impact of helium absorption on seals etc causing failure. Helium is a very small molecule, so very hard to keep confined.

However, as they learned from experience, Loon was slowly increasing their upper limit on how long their balloons could stay up.

One of the first ideas we explored was putting an electrolyzer on the balloon to replenish hydrogen over time. Unfortunately right now, for balloons our size and power budget it's just not feasible. And actually, we can get a pretty low leakage rate with our materials which lets us stay aloft for a really long time, but eventually the UV degradation becomes to extreme.

We cruise in the stratosphere until we gather enough solar power to descend to ground level. On the descent and descent we're collecting soundings, which is the data that's useful for all types of forecasts.

Right now we're focused on the stratospheric forecasts because that's what we know really well (and we already have some interested customers). Our data/models are great for all kinds of forecasts, including ground forecasts, and we'll quickly expand beyond the stratosphere.

This is super interesting, I'd never come across ARGO before. Data assimilation is a similar problem for our data, and there currently exist systems for assimilating weather balloon observations into gridded reanalysis data (https://www2.mmm.ucar.edu/wrf/users/). One thing we believe, however, is that the reanalysis step in weather forecasting is unnecessary in the long term, and that future (ML) weather models will eventually opt to generate predictions based on un-assimilated raw data and will get better results in doing so.

That being said, trajectory-based data tooling could be super interesting to us. Let's definitely chat: austin@sorcerer.earth

And re: recovery, we're pretty confident we'll be able to recover the majority of our systems. Being in the air has the advantage that we can choose to 'beach' ourselves in a specific location, rather than the first place we run across land like with the buoys. At his previous company, Alex wrote a prediction engine able to get similar balloon systems to land in a predicted 1kmx1km zone for recovery

We're not hiring right now, but definitely check back in a few months. As for advice, there's almost always a place for talented engineers. https://www.climatetechlist.com/ and https://jobs.climatebase.org/ both aggregate jobs at climatetech cos specifically if that's what you want to pursue.

I was in the same boat. I've been a software engineer for as long as I can remember and always wanted to do more than just build B2B SaaS.

Max, the first engineer at Urban Sky, hit me up and asked if I wanted to build their mission control. At the time, Urban Sky was just a four-person team, so they couldn’t pay me as much, but I jumped at the chance, even though it meant taking about half my usual salary.

Funny enough, my SaaS background actually helped me create mission control software that was way ahead of the curve!

I guess my advice is, find a small company you're passionate about, where you can make a big impact, and be open to taking a pay cut. It helps the company take less of a risk on you, and you get to work on something that really matters. Plus, when you’re solving real problems, things tend to work out, and eventually, you’ll end up making what you should in salary.

What's the project? Maybe we can help!

Can I ask how you handled the regulatory aspect of launching from urban areas in SF? I can imagine the FAA would have given you some trouble?

We actually fall under a weather balloon exemption to the normal FAA rules for unmanned balloon flights. Here’s a quick rundown of the relevant rules and regulations (Part 101.1) for weather balloon flights in the U.S. Our balloons fully comply with all of these:

- Any on-board cellular tracking devices must be set to Airplane mode before takeoff (we don't have any cellular) - Each individual payload box/package must weigh less than 6 pounds (ours is <250g)

- If a payload has a weight-to-size ratio exceeding 3.0 ounces per square inch, it must weigh less than 4 pounds.

  - Calculation: Divide the total payload weight in ounces by the area of its smallest face in square inches.

- The string connecting the payload to the balloon must break under an impact force of no more than 50 pounds (our string is 30g)

- It’s prohibited to design or operate an unmanned free balloon in a way that poses a hazard to people or property.

- Dropping objects from the balloon that could endanger people or property is not allowed.

We don't have to notify the FAA about our operations as long as we meet these criteria. To be super safe, we don't launch near airports or other busy areas.

Speaking to farmers in Central America was a huge impetus for us starting Sorcerer! FAA's part 101.1 regulation is for ballooning is confusing [0], but ultimately very open if your payload is below 6 pounds. Several thousands balloons are launched every day under this regulation and so far as I know there haven't been any incidents outside of a few radiosondes landing on peoples cars

[0](https://www.ecfr.gov/current/title-14/chapter-I/subchapter-F...)

We're not hiring right now, but send me an email (austin@sorcerer.earth). And Max did the website, it's built on webflow

I'm curious too!

Weather soundings play a part in making a forecast but it's not the only thing? Many times per day we collect weather station data, correlate against past forecasts, bias correct, etc.

Forecasts are made of so much input data it's insane, yes balloons matter but it's not the only thing. They are the only decent source of conditions aloft and the jet stream is main controller of our surface weather so it makes sense.

You’re right - Satellite data like GPS occultations and aircraft derived data still make up most of the data by volume. Soundings remain the ideal data source and are weighted very heavily in models, punching way above their volume.

As of right now power constraints mean we maintain tracking throughout the night, but cannot execute altitude maneuvers. We have a solution to this cooking though!

Very interesting, I'd love to hear more about it! In short, yes, we plan to do descents near urban areas if there's a route where we can go down to a safe height and stay away from any airspace. What cities are you looking at right now?

We've deployed a few dozen, and we don't have a public API right now but send me and email: austin@sorcerer.earth

We think balloons are pretty cool too!

So I can't get into exactly how we do our altitude control, but the Google Loon project has a really great explanation of how they made their (very big) balloons go up and down: https://x.company/projects/loon/

Loon made all of their research public after they shut down, and we're obviously heavily inspired by their work. Our systems use a lot of the tech they pioneered, just on a much, much smaller scale (for reference, Loon's balloons were the size of tennis courts) Here's the PDF in case you're interested in checking out the 400+ page writeup: https://storage.googleapis.com/x-prod.appspot.com/files/The%...

I don’t know much about WAPS at all, but generally speaking, we’re only interested in doing weather sensing. As of now, we’re not planning to license the vehicle at all. And generally speaking our platform doesn’t work for payloads that are heavier than ~0.25lb and have more than minimal power requirements.

Very cool! Are you working on some kind of airborne solar platform?

We're actually just going to have kids launch our balloons, it's no problem:

  A person who is 6 years of age or younger who
  intentionally releases, organizes the release of, or
  intentionally causes to be released balloons as 
  prohibited by s.
  379.233 does not violate subsection (4) and is not. 
  subject to
  the penalties specified in subparagraph 1.

Got our start with WSPR pico balloons!

Just saw that - No Florida launches on the horizon luckily

That's cumulative for all data collected like satellites, weather radars, and surface observing systems. Soundings are relatively "lightweight" compared to most sets. The major difficulty is pushing the data through our low-power satellite backhaul - in practice we average ~10B/s

I would be interested in this as well. I hope they start an engineering blog at some point.

For sure, we'd love to do some launches with some local folks. We've done launches with a few different companies in SF at their offices too, send me an email and we can figure something out: austin@sorcerer.earth

(Sorry for the offtopicness but your otherwise fine comment was formatted in HN's 'code' style (https://news.ycombinator.com/formatdoc) rather than as a regular comment. I changed the formatting, but for the future it would be better not to indent your text with spaces unless it's code or something that needs that format.)

The stratosphere is having a renaissance: https://youtu.be/u1rivUSi8p8?t=519

Not yet, but keep an eye on https://news.ycombinator.com/jobs

Our balloons fall under the FAA Part 101 weather balloon exemption, which is based on the weight and purpose (basically it has to be small and has to collect weather data)

Sorcerer (1977)

Bill Brown is a legend! Love Lee Meadows with SBS as well - 767+ days is unreal

https://www.scientificballoonsolutions.com/news/

Sent you an email, thanks!

a little bit of each right now, depending on what the customer wants

Internally yes we use a data warehouse to store the raw observations. We go through a similar process to traditional NWP to produce forecasts except with a few AI steps in between. Unfortunately, the standard currently is to store these as large multi-dimensional data files like grib2/netCDF/zarr.

In case you’re curious, here’s where NOAA stores all their GFS related forecasts: https://registry.opendata.aws/noaa-gfs-bdp-pds/

Yes! It’s quite confusing to distinguish between launches at the office haha

What we're doing wouldn't be possible without the work the Loon team did (and published)

That's the goal. We're all from Florida, so the possibility of reducing the damage caused by hurricanes was a big reason we started the company in the first place.

Our balloons are actually cheap and steerable enough that we plan to fly them into TS/Hurricanes to get data way out in the Atlantic, farther than hurricane hunters can operate.

*inflates

Weather balloons (ours included) are pretty small and don't spend much time at the altitudes where planes are. And in the very, very unlikely case a plane were to hit one, our payload is less than 250g (about the mass of a pigeon).

Reach out if you think we could help at all :)

1. Wind speed/direction, air pressure, temperature, humidity, and solar irradiance. We are considering doing atmospheric composition as well, there are a couple of partners that are interested in GHGs, VOCs, water vapor, etc. We have the weight budget to do it, but we haven't flown any production payloads with those sensors yet.

2. The US National Weather Service actually has a commercial data buy program called MESONET [0], where they buy weather data from both academic and commercial partners. We're in the process of becoming one of their commercial partners now. Once we are, a single balloon will pay for itself in a matter of days with the data it can collect, which will let us scale up the number of systems we have deployed. The data we collect right now also lets us build niche weather forecast products like the stratospheric wind forecast mentioned in the post. Once we have enough balloons up, we can start producing useful weather forecast products at a regional and then global scale.

[0]: https://nationalmesonet.us/

Send us an email! team@sorcerer.earth

We love Yohan!