In a medium saucepan, whisk together the sugar, flour, and salt. Gradually whisk in the milk until smooth.
Cook over medium heat, stirring constantly, until the mixture thickens and comes to a gentle boil. This should take about 10 minutes.
Remove from heat. In a small bowl, lightly beat the egg yolks. Gradually whisk in a small amount of the hot milk mixture to temper the yolks.
Return the yolk mixture to the saucepan, whisking constantly. Cook for an additional 2 minutes, then remove from heat and stir in the vanilla extract.
Layer the bottom of a 2-quart baking dish with vanilla wafers. Top with a layer of sliced bananas, then pour a portion of the pudding over the bananas. Repeat the layers, ending with a layer of pudding.
Optional: If using meringue, beat the egg whites until stiff peaks form, then spread over the top of the pudding. Bake in a preheated 350°F oven for 10-12 minutes, or until the meringue is lightly browned.
Chill the pudding in the refrigerator for at least 2 hours before serving. Top with whipped cream if desired.