In a copyright context, public domain is a term which means the set of creative works not under copyright, because the work's copyright has either expired, never existed (such as U.S. government works), or been forfeited1. Generally, no special restrictions apply to the reuse of public domain works, and they can be freely modified and redistributed without limitation.
Open source software, as defined by the Open Source Initiative's Open Source Definition, means software whose source code is available and can be freely modified and distributed. Open source software is typically still under copyright (i.e., typically not in the public domain), but the author grants rights to users via a copyright license.
Open source software and public domain software are by no means the same thing, but there is some overlap. Open source software can be in the public domain, and public domain software can satisfy the open source definition.
The primary differences in definitions are:
Public domain software might not include source code. Since public domain software automatically allows redistribution and modification, it satisfies almost all of the open source criteria automatically, but not quite. It is notably missing the requirement that the program's source code be publicly available. It is therefore possible to have a piece of software in public domain that fails to satisfy the open source definition because its corresponding source code is unavailable.
Open source software can grant a more limited set of rights than public domain. Like public domain works, open source software grants the right to modify and redistribute the work. However, an open source license may impose other restrictions that do not exist on public domain works. For example, the GNU GPL requires copyleft: any derived work that uses GPL-licensed material may only be distributed under the terms of the GPL. Such a restriction does not violate the open source definition, but it is certainly far stricter than the public domain.
Open source software can be released under different licenses by the author. The author of a copyrighted open source program can issue multiple licenses on the work. A common case of this is when an author licenses a program under a copyleft license, but will also issue individual proprietary licenses for a fee. Users can either get the software free of charge under copyleft rules, or pay money to get the software under another license. (See OSS Watch's article "Dual-Licensing as a Business Model" and the Free Software Foundation's "Selling Exceptions to the GNU GPL".) Such a scheme is impossible with public domain software, because it relies on the author licensing the work's copyright in different ways. Since public domain software has no copyright, there is nothing to license.
Open source licenses may grant patent rights. Public domain software does not have any copyright on it, but its reuse may still be restricted by patents, if it contains an implementation of a patented mechanism. If you redistributed such patent-encumbered public domain software, you would not be liable for copyright infringement but may be liable for patent infringement. Some open source licenses, however, expressly grant you patent rights. For example, an author who releases software under the Apache Public License or GNU GPL (version 3) expressly grants recipients the right to use any of the authors' patents that apply to the software, so there is no risk of violating the author's patents.2Not all open source licenses do this, but it is an option for open source software under a license, whereas it is not an option for license-free non-copyrighted public domain software.
1 Note that it is unclear if premature dedication to the public domain is truly possible in most jurisdictions.
2 If the author illegally used someone else's patented mechanism without permission in the software, a downstream redistributor could be legally liable for that infringement.