Both file types can be used to store and transfer routes and way-points.
GDB files are Garmin's own proprietary format. GPX files are, in essence, XML files that can be used across quite a few GPS devices. Garmin do have their own type of GPX file as well as there being a generic GPX format as well. As far as I know the Garmin GPX can still be read by other devices but also has a bit of extra data only Garmin devices and programs (Mapsource and Basecamp) can read.
More recent Garmin models (Nuvis and Zumos) usually use GPX files natively anyway.
HTH.