Few, but ripe. (Carl Friedrich Gauss)

It is always tempting to submit a paper to a prestigious journal, but if the paper makes only a borderline case for publication in this journal, then the net result may be a lengthy process, critical reviews holding the paper to a very high standard, and ultimate rejection of the paper.

For instance, with JAMS, a paper really has to do something that makes referees excited and enthusiastic; a paper which is merely a good, solid application of mostly standard techniques to solve a moderately interesting problem will unfortunately have a rather low probability of being accepted into JAMS, even if it would have been readily published elsewhere. (Conversely, if the result is making people excited and enthusiastic, I do hope that you consider JAMS for your paper. :-) )

Similarly, a journal devoted to research mathematics is unlikely to accept any paper whose primary focus lies in recreational mathematics, physics, philosophy, biology, computer science, or anything else outside the scope of research mathematics.

It is also a good idea to check that the editorial board of the journal you are submitting to contains at least one member who is expert enough in the fields that your paper is in that he or she can judge its quality appropriately and send it to a good referee.   Viewing a sample issue of that journal may also give you a sense as to whether it is a suitable venue for your article.  You can also look at journals which have accepted papers similar to yours in the past, but of course there is no guarantee that they will do the same for your paper; indeed, if your paper is closely modeled on an existing paper in a journal, they may feel that the amount of new material in your submission may not be sufficient to warrant placing it on the same level as the earlier paper.

Generally speaking, it is not recommended to simultaneously submit two unrelated papers to the same journal; there is a possibility that they may somehow get confused with each other (for instance, a report for one paper may accidentally be applied to the other), and editors may not wish to give the impression of overly favouring one particular author in the journal.  Also, if it ends up that the referee reports for one paper are more favorable than for the other (or if a referee makes a direct comparison between the two), it becomes quite likely that the paper with the less favorable reviews will be rejected.  For two closely related papers, I would only recommend submitting to the same journal if it would make sense to have a single referee for both papers (but this can be quite a big request for a referee to accept).

The American Mathematical Society maintains a list of research journals in mathematics.