Few, but ripe. (Carl Friedrich Gauss)

It is always tempting to submit a paper to a prestigious journal, but if the paper makes only a borderline case for publication in this journal, then the net result may be a lengthy process, critical reviews holding the paper to a very high standard, and ultimate rejection of the paper.

For instance, with JAMS, a paper really has to do something that makes referees excited and enthusiastic; a paper which is merely a good, solid application of mostly standard techniques to solve a moderately interesting problem will unfortunately have a rather low probability of being accepted into JAMS, even if it would have been readily published elsewhere. (Conversely, if the result is making people excited and enthusiastic, I do hope that you consider JAMS for your paper. :-) )

Similarly, a journal devoted to research mathematics is unlikely to accept any paper whose primary focus lies in recreational mathematics, physics, philosophy, biology, computer science, or anything else outside the scope of research mathematics.

It is also a good idea to check that the editorial board of the journal you are submitting to contains at least one member who is expert enough in the fields that your paper is in that he or she can judge its quality appropriately and send it to a good referee.   Viewing a sample issue of that journal may also give you a sense as to whether it is a suitable venue for your article.  You can also look at journals which have accepted papers similar to yours in the past, but of course there is no guarantee that they will do the same for your paper; indeed, if your paper is closely modeled on an existing paper in a journal, they may feel that the amount of new material in your submission may not be sufficient to warrant placing it on the same level as the earlier paper.

The American Mathematical Society maintains a list of research journals in mathematics.