return to table of content

Think Python, 3rd Edition

bmitc
14 replies
4h49m

How does one do proper concurrency in Python? As in separate "processes" running with encapsulation, fault tolerance, etc. For example, how does one bundle up a TCP client in a "process" such that the process handles failed connections, broken connections, retrieving data at periodic intervals (relatively fast), receiving "messages" from other "processes", etc.?

There's Ray, Pyro, Pykka, Celery, multiprocessing, asyncio, threads, Qt, and more, but all of them have issues. And a lot of it boils down to the GIL, although "processes" that are doing I/O such as TCP and other network communication should ideally reduce the GIL effect, to my understanding (is that right?).

From what I can tell, it's basically one of the worst language choices for systems of this nature, but I am trying to figure out how to do it because I need to.

da39a3ee
6 replies
4h23m

Can you give an example of what you consider a good solution in a different language? That will help us see exactly what you're looking for and how/whether it might be achieved in Python.

bmitc
5 replies
2h59m

Erlang and Elixir. :)

staticautomatic
4 replies
2h17m

Well RabbitMQ is built in Erlang so there you go :)

bmitc
3 replies
2h13m

But RabbitMQ is just a messaging bus, isn't it? That doesn't handle the processes (operating system processes or virtual processes) actually processing the messages and those processes handling fault-tolerance for the things they're connected to.

staticautomatic
2 replies
2h3m

That is correct. Honestly I’m not sure this is a good use case for Python but it’s definitely possible. I’ve used the RabbitMQ + gevent + multiprocessing pattern in the past and it works but I find the code extremely hard to reason about. If I were doing it again from scratch I’d probably choose another language with better concurrency primitives.

bmitc
1 replies
1h44m

Honestly I’m not sure this is a good use case for Python but it’s definitely possible

It definitely isn't, but this is a new role with a rushed timeline in a place dominated by Python.

and it works but I find the code extremely hard to reason about

This is also a major concern of mine.

staticautomatic
0 replies
58m

Rather than handling the multiprocessing and message passing yourself it would be much easier to use celery + gevent and let celery do the work of spinning up processes for executing the tasks.

It may feel limiting but my advice would be to keep all the celery job queue stuff isolated from your server, especially if you are using an async web framework. Have your web server just put all the jobs in the celery queue and let it handle executing them, regardless of whether they’re cpu or io-bound. If you try to optimize too much by doing something like leaning on celery for cpu-bound tasks but letting your web server handle the io-bound ones you’re going to be in for a world of hurt when it comes to both debugging and enforcing the order of execution. Celery has its warts but you’ll at least know where in the system your problem is and have reasonably good control over the pipeline.

jononor
2 replies
3h44m

If you are I/O bound then asyncio or good old fashioned gevent will do great. If you are CPU bound, then use multiprocessing. If you need to accept "jobs" from elsewhere, use RabbitMQ (with or without Celery). If you have mixed CPU/IO workload that fits the worker pattern, then you would do all 3. At the top level you have a RabbitMQ consumer, fetching jobs from a remote queue and then putting these into a multiprocessing queue processed by N=~cpucount processes. And each of these use asyncio/gevent to do their work.

varispeed
0 replies
2h50m

Why some people are extremely averse to RabbitMQ?

I saw that at one company having RabbitMQ / Celery setup - every time a new software engineer comes in, they complain about RabbitMQ and ask why would company use it. The infrastructure was running like this without hiccups for years. At one point company has let go of many experienced engineers and this time one developer found some issue with the code and blamed it on rabbit as it was locking the queue. There were no more senior developers to contest it, so he convinced manager to swap it out for Redis. He took about two months to rewrite it. Surprise, the same issue existed on Redis. The Redis solution works fine, but has its own limitations...

bmitc
0 replies
2h57m

So do you recommend a single Python process running asyncio or multiprocessing or both? Or do people normally split these things amongst several Python processes?

My understanding is that multiprocessing creates multiple interpreters but that it still comes across some GIL issues if all under the same Python process.

I am in general quite comfortable with the actor model, and I would ideally use Erlang/Elixir here, but I can't for various reasons.

Siecje
2 replies
4h22m

Use asyncio. Bind to a socket, set blocking False, then asyncio.run(on_new_connection(sock)).

Inside that coroutine get the loop and await loop.sock_accept(sock)

And then asyncio.create_task(on_connection_data(connection)).

The only gotcha is you need to keep a reference to that task so it doesn't get garbage collected.

staticautomatic
1 replies
2h17m

Get garbage collected by what? Doesn’t Python use reference counting?

d0mine
0 replies
1h59m

asyncio doesn't store strong references to tasks. It is your responsibility to keep the ref: https://docs.python.org/3/library/asyncio-task.html#asyncio....

Consider organizing the code using TaskGroup.

andai
0 replies
4h31m

Something something libuv?

peruvian
12 replies
3h50m

Apologies for changing the subject, but aside from real world experience (which I have and am getting at work), is there a resource of similar quality for more intermediate/advanced Python programmers? I always feel like there's a big chunk of the language or stdlib I do not know.

theptip
1 replies
1h55m

I like https://effectivepython.com/

Also just reading Norvig’s annual Advent of Code implementations usually provides some insight on how to write elegant and concise Python code.

billbrown
0 replies
1h36m

Looks like there's a new edition coming in March. https://www.amazon.com/Effective-Python-Specific-Software-De...

js2
1 replies
1h48m

I learned Python starting with 1.5.2 from the official documentation and think it's a good resource.

https://docs.python.org/3/tutorial/index.html

https://docs.python.org/3/library/index.html

Whenever a new version is released, I read its What's New documentation.

Beyond that, I like to read source code, both for the stdlib and popular third-party packages. This advice generally applies when I'm learning any new language or re-familiarizing myself with one, not just Python.

sireat
0 replies
45m

I really enjoyed Fluent Python a while back as an intermediate book.

Python official docs are not completely horrible, but compared to most other popular languages (Kotlin, Scala, Rust, Go at least), the Python official docs are kind of meh.

I suppose Python docs beat C and C++ which do not have official docs besides the spec. (not counting K&R and Bjarne's books).

Also I guess Javascript does not have official docs (ie MDN is not official)

billfruit
1 replies
57m

Fluent Python is good book at that level, and works as a good reference book while working too.

hobs
0 replies
34m

I have bought this book for every friend learning python for work purposes, really fleshes a lot out that's not taught implicitly. The data model stuff is really useful.

zerkten
0 replies
2h2m

Fluent Python and Effective Python are good books. The former is huge and is really multiple books in one.

rmk
0 replies
53m

Python Modules of the Week (PYMOTW). Great resource to learn the stdlib.

https://pymotw.com/3/

pid-1
0 replies
1h22m

Reading the docs proactively (not just when you need something).

nickpsecurity
0 replies
25m

Humble Bundle had some nice collections on Python for many uses. For in general, I remember that Serious Python and Automate the Boring Stuff with Python were both good.

jjice
0 replies
3h24m

I enjoyed Effective Python. It's a "tips" style book with a good handful of recommendations with use cases and applications.

bikingbismuth
0 replies
3h11m

It’s a bit older, but I learned a lot from “Writing idiomatic Python”. Honorable mention to “the little book of Python antipatterns” as well.

aitchnyu
10 replies
13h11m

Tangential, is there a book to learn typing? I feel I need a hefty book to learn typing. Don't mind if you recommend a Typescript book or other language with similar concepts.

noelwelsh
2 replies
8h39m

You mean types, right? Typing is what you do on a keyboard.

If you want to understand type systems, Types and Programming Languages (https://www.cis.upenn.edu/~bcpierce/tapl/) is the book most people start with. If that is too advanced for you, PLAI (https://www.plai.org/) is a gentle introduction to programming language theory which includes type systems.

eru
1 replies
4h43m

Those books are great; though I'm not sure whether they are good for a beginner? Getting your feet wet by programming in a few reasonably typed languages might be better, before you tackle TAPL?

(By 'reasonably typed' I mean something like Haskell, OCaml, even TypeScript or Facebook's hack. But not Go, C++ or Java.)

jdeisenberg
0 replies
2h58m

If you want something JavaScript-ish with strong type inference, take a look at ReScript (https://rescript-lang.org/)

quickthrower2
0 replies
13h5m

That is a big topic. Learning typing in Go will be easier than Typescript which is orders more sophisticated. In Typescript you can solve Sudoku in the type system! Then there is the theory of type systems. But if it is the day to day just use it, it will make sense. You are just describing what the type of something will be in code rather than let it be figured out at runtime.

magnio
0 replies
12h7m

I can recommend Total Typescript by Matt Pocock, a course and a collection of tutorials. He is writing a book in public at [0] as well.

A bit more advanced, and geared towards library authors, is Type-level Typescript.[1]

[0]: https://github.com/total-typescript/total-typescript-book

[1]: https://type-level-typescript.com/

lysecret
0 replies
11h37m

I learned it through Domain Modeling Made Functional (it uses F# though which looks the same as Ocaml).

h4ch1
0 replies
13h5m

https://www.cis.upenn.edu/~bcpierce/tapl/

This is a comprehensive resource that I use as a reference. Happy reading!

asicsp
0 replies
12h13m

Not a book, but this might help:

"Python Type Challenges" (https://github.com/laike9m/Python-Type-Challenges) — Master Python typing (type hints) with interactive online exercises

NlightNFotis
0 replies
8h39m

Programming with types by Vlad Riscutia fits your bill exactly.

Examples in typescript (so syntax should be familiar compared to e.g OCaml) and teaches you how to model a domain in types and how to think in terms of a type system, instead of diving into the details of how to implement one.

NegativeLatency
0 replies
12h55m

Haskell might be a fun if indirect way to learn more about a type system. It’s a neat middle between academic typing systems and a language you can do real stuff with.

Typescript might be nice because of how it’s a gradual typing system, just try and do a bit more and more with it as you go.

bikingbismuth
6 replies
10h49m

Think Python 2e changed the trajectory of my life. I took a single Java class and hated it so much I gave up on programming. A few years later as a network engineer I had a problem that seemed like it could be scripted and ended up picking up Think Python and fell in love with the language and programming in general.

ambrose2
3 replies
10h21m

Similar for me - I had learned some Java, Matlab, C, Perl here and there but it wasn’t until Think Python 2e that I was gripped and from there read many other books and changed my career to software.

rmbyrro
1 replies
8h52m

Couldn't this result be more attributable to Python itself than the book?

eru
0 replies
4h47m

In the abstract, yes. But I also know that Think Python is a great book.

(I came across Think Python when I was trying to help other people learn how to program. So I did not learn programming from Think Python, and Python is also not my favourite language. (It's also not my least favourite language, either. Far from it.))

el_oni
0 replies
9h23m

What other books did you read that you would recommend?

bemusedthrow75
1 replies
8h53m

This is just the sort of thing I needed to read.

I am considering changing the trajectory of my own life, towards a more community/maker/teacher role, and I have a freelance/small business idea about teaching but I need sort of "soft syllabus" materials.

I am learning Python myself, having just never had a need for it in all of my professional web development life (I've written apps in just about every other web-focussed programming language, including Perl and Ruby).

It looks like the right language to teach general concepts in, and having a book I can draw from will help.

eru
0 replies
4h48m

Think Python is also just a really great book, even for people who don't want to stick with Python long term.

skotobaza
4 replies
8h28m

What are some books for mid to advanced programming in Python? I already know Puthon and programming in general but want to improve my Python skills.

I know only Fluent Python which I'm currently reading, and CPython Internals.

stefanos82
0 replies
7h5m

`Fluent Python` is more than enough; the rest of the language features you will understand them by reading the official reference manual.

nickelpro
0 replies
8h19m

I mean, the docs? And the source code?

At a certain point of expertise, everything after basic journeyman familiarity, there's nothing left but to read code and write code.

eru
0 replies
4h46m

You could also look into more general algorithm and data structure books, or into design books.

https://www.redblobgames.com/ has lots of really nifty articles, too.

__mharrison__
0 replies
3h15m

You might want to consider books that show application of techniques in real world practical code.

For example, Effective Pandas 2 illustrates common patterns for dealing with tabular data. Along the way, it uses comprehensions, lambdas, unpacking, etc. Shows how to use pytest to refactor. Leverage visualization to understand data.

(Disclaimer: I'm the author)

irrational
3 replies
4h30m

Does anyone know how to report issues with the first chapter? I don't see a way to leave feedback.

hoppyhoppy2
1 replies
4h29m
irrational
0 replies
4h26m

Thanks!

xoxxala
0 replies
3h37m

“If you have comments, corrections or suggestions, please send me email at feedback{at}thinkpython{dot}com.“

dod9er
3 replies
9h56m

I love the Jupyter-/Colab-approach and would like to join the journey... and support the project by buying the book (although I don´t need it to follow the Colab?). But without subscribing to the OReilly-learning platform it seems that buying it on amazon is the only option, right ?

wjnc
1 replies
7h57m

Green Tea Press (by Prof. Downey) has a donate button. That's probably a more efficient means of supporting the project than buying the book, considering all the middlemen.

dod9er
0 replies
5h53m

Oh my ..., was skipping too fast through the page and didn´t see this, and also didn´t realize what he exactly is doing with his Textbook Manifesto etc. Ok, the Paypal-Link is a no-brainer for me now. Thanks for pointing out the obvious.

bemusedthrow75
0 replies
8h49m

It's a pre-order even on Amazon. The publication date for the book isn't until later in the year (at least for the edition available in the UK, which will probably be the same):

https://www.whsmith.co.uk/products/think-python-how-to-think...

https://blackwells.co.uk/bookshop/product/Think-Python-by-Al...

(Blackwells think September, incidentally, whereas WHS think the end of July)

Amazon just list books way, way earlier in pre-order than a lot of places.

jph00
2 replies
13h56m

So excited to see this coming! When we used Jupyter Notebooks to publish Practical Deep Learning for Coders, we discussed with Allen the idea of doing the same for some of his books.

Now it's actually happening. :D Even better, he's taken it further by adding cool tools such as a Jupyter-based turtle that shows inline graphics in the notebooks. I strongly suspect this will turn out to be the best way to learn Python programming when it's released.

Oh and I just remembered, we even showed a proof-of-concept of converting some of the 2nd edition of this book into nbdev notebooks: https://github.com/fastai/nbdev_cards/blob/master/01_deck.ip... . That notebook is rendered as this HTML: https://fastai.github.io/nbdev_cards/deck.html

stuaxo
0 replies
10h2m

Having code you can hide like this looks great foe documentation, I've been looking for something like this for a while.

fifilura
0 replies
9h41m

I am proud of you!

I loved Think Bayes and Think Stats, but it felt a bit off when everyone else were using notebooks.

When I learn a new language with e.g. AdventOfCode, my first task is building a jupyter image for it.

febeling
2 replies
2h49m

What is a good book for learning Python when you already now some/many other languages?

d0mine
1 replies
1h58m

Just follow the tutorial (yes, really) https://docs.python.org/3/tutorial/index.html

autoexec
0 replies
20m

It's what I started with, but honestly it's not great. It also heavily overemphasizes the interactive interpreter

BadHumans
2 replies
13h25m

This book gets overlooked in favor of other ones such as Python Crash Course but I really enjoyed reading through Think Python 2e and will read through this version as well. Check out the rest of his books on Green Tea Press.

wyclif
1 replies
12h48m

I also like PCC. What do you think the pros and cons are of PCC v. TP3e? They are really two different types of books in terms of pedagogy.

BadHumans
0 replies
3h14m

Python Crash Course teaches Python while Think Python teachers computer science using Python. You are right that they are two different books but people recommend Python Crash Course for beginners when Think Python is a much better recommendation for them in opinion. If you just need to learn Python then PCC is a great book.

orzig
0 replies
2h41m

The author also blogs his latest Python projects (mostly Bayesian data analysis) on https://www.allendowney.com/blog/ , which I have really enjoyed.

mharig
0 replies
1h36m
lynguist
0 replies
1h59m

I use the 2nd edition when I teach intro to programming. The students uniformly love this book and prefer it over any university provided material.

librasteve
0 replies
9h24m

I love these books … in my case Think Perl6 (now Think Raku) was a real eye opener. A quick scan of the new Python 3rd edition looks like it is quite dumbed down - which is probably appropriate for Python.

I highly recommend the intro to Functional Programming in Raku (chapter 14). https://greenteapress.com/wp/think-perl-6/

fbdab103
0 replies
13h53m

What luck! I just started mentoring someone in Python, and I am a huge fan of Downey. Truly an outstanding educator and renaissance man.

Will definitely keep my eye on this.

elbear
0 replies
12h16m

This is the book that got me started with programming, so I'm grateful to Allen for it.

dang
0 replies
12h6m

Related:

Think Python 2e - https://news.ycombinator.com/item?id=35421096 - April 2023 (30 comments)

Think Python: How to Think Like a Computer Scientist - https://news.ycombinator.com/item?id=1586000 - Aug 2010 (9 comments)

caturopath
0 replies
5h21m

Love Think Python, I have recommended it to so many learners: it balances the various concerns of a new programmer book really well. Allen Downey has a bunch of other books with somewhat similar approaches too https://greenteapress.com/wp/ -- some I do think he might have gone too far in the low-rigor side, but all the ones I've reviewed have been pretty good.

(I was sharing a table at a conference with Allen some time ago and told him how many times I'd recommended or bought people his books, and I think he thought I was bullshitting him.)

brunooliv
0 replies
6h55m

This book ALSO changed my life when I was first learning programming!! Not so much in terms of a pivot but of the way the book was written itself and how some concepts all of a sudden just made sense as a total freshman. Now, ~10-15 years later, seeing this third edition is such a nostalgia and I can't recommend this book enough!!!

asicsp
0 replies
13h27m

I had read the second edition when I was still relatively new to Python. I even spent a few weeks trying to translate the book to Ruby. Found it a great experience, especially the friendly prose and excellent exercises.

F00Fbug
0 replies
4h41m

This has been a great resource - I love this book! For the last 5 years, I've taught an intro to programming class at the college level and I always recommend that my students augment their resources with this book.

Glad to see it evolving!