It is true there is more to it than religion, but the religion, the religious heritage, and in particular, the persecution for a religious heritage, is the foundation of the historical shared identity.

Of course one can be a secular Jew, just as there are secular Christians (or for that matter, Christian atheists).