Question: May one cut down a fruit tree in order to make room for improvements to their back yard for recreational purposes such as to put in a pool or a basketball court?

Answer: The Torah forbids cutting down fruit trees (Devarim 20:19), which is the strictest application of the concept not to be destructive (see Rambam, Melachim 6:8). It is thus not surprising that the gemara and poskim identify "non-destructive" cases where it is permitted to cut down fruit trees.
The gemara grants permission in the following cases: 1. The tree no longer produces a kav (a relatively small amount) of fruit (Bava Kama 91b-92a). 2. It is worth more for wood than for fruit (see Rashi, ad loc.). 3. It is significantly damaging a more valuable tree (see Tosafot). 4. It is damaging someone else’s property (Bava Batra 26a).
The Rosh (Bava Kama 8:15) learns from the above that one may cut down a tree if needed to use its location, which the Taz (Yoreh Deah 116:6) applies to building a home. Most poskim say this includes expanding a home, at least when the addition is objectively more valuable than the tree (see Chayim Sha’al I:22; Yabia Omer V:12). On the other hand, the gemara tells of an Amora’s son who died because he cut down a fruit tree prematurely, and R. Yehuda Hachasid also warned about it. Therefore, even when it is apparently permitted, some prefer that the work be done by a non-Jew (ibid.) and/or that the tree be transplanted (Chatam Sofer, YD 102).
To what extent can we rely on the Rosh’s thesis that making room for something else is an excuse for cutting down a fruit tree? The Beit Yaakov (140) claims that Tosafot and others disagree with the Rosh. The Meishiv Davar (II:56) adds that it is hard to be certain that after cutting down the tree, the building project will actualize. However, many Acharonim (see Chayim Sha’al I:22; Yabia Omer V, Yoreh Deah 12) strongly reject the Beit Yaakov and adopt the Rosh/Taz leniency.
How important must the need for the spot be? Although the gemara’s cases (e.g., wood worth more, affecting another tree) are not huge benefits, they relate to situations where the tree’s existence is more directly wasteful. In contrast, in the Rosh’s (and your) case, the tree is fully viable, just that it precludes another future use. It is therefore not surprising that some who accept the Rosh say that the need must be substantial. The She’eilat Yaavetz (I:176) relates to a case where a shul is too small and needs to be extended to an area occupied by fruit trees. The Chavot Yair (195), while allowing cutting down a tree that darkens one’s house, forbids doing so to allow for a place for walking around or increasing space and light. Several Acharonim, including important poskim such as the Aruch Hashulchan (YD 116:13) and Yabia Omer (ibid.) adopt this middle-of-the-road approach.
Appraising the cases you raised is tricky. On one hand, building a swimming pool or a basketball court is expensive, so that one erects one only if it is important to him (see Minchat Asher, Devarim 33), in which case the tree should not prevent it. On the other hand, some poskim (see Yabia Omer ibid.) indicate that the value of the change should be an objective one that applies to the average person. Swimming pools and basketball courts are not likely to qualify in that regard (even if we focus on the positive and permitted uses of those facilities). It is hard to ignore the possibility that one who uses honest but faulty judgment could be punished with death (aforementioned gemara; see also Chatam Sofer YD 102; She’eilat Yaavetz ibid. is more extreme). Another factor is that it might be possible, even if less convenient, to build what is desired without cutting down a fruit tree.
Therefore, we suggest the following. If you are willing to professionally, preferably by a non-Jew, transplant the tree, you may do so. Otherwise, we would have difficulty permitting removing the fruit tree unless we were convinced that the need and the lack of alternative were clear.