Preheat oven to 325 degrees. Prepare 3, 8-inch round cake pans by spraying liberally with non-stick cooking spray.
Sift together cake flour, baking powder and salt in a medium mixing bowl and set aside.
In a stand mixer with the paddle attachment, beat butter and sugar together until smooth and fluffy. Slowly add the eggs and mix well until combined.
Stir in sour cream and vanilla, and mix well until combined.
Scrape the sides of the bowl and add half of the flour mixture. Stir until fully incorporated. Scrape the sides of the bowl again and stir in the milk.
Add the remaining flour and stir until the flour is fully incorporated. Scrape the sides and bottom of the bowl and give it one last stir.
Divide the batter into 3 prepared, round baking pans.
Bake in 325 degree oven for about 30-35 minutes. Cakes are done when tops spring back with a light touch, or a toothpick inserted into the center of the cake comes out clean. Allow to cool completely before turning over onto a plate or baking rack.
Set half of the peaches in a bowl. Sprinkle with brown sugar, cinnamon, nutmeg and a pinch of salt. Stir to coat completely and let sit on the counter for about 20 minutes.
Meanwhile, finely chop remaining peeled peach slices. Place chopped peaches in the refrigerator until ready to use.
Puree the peaches/brown sugar mixture in a food processor or blender on high speed for about 30 seconds, or longer if needed, until the mixture is smooth. Pour the peach puree into a fine mesh strainer and set it over a small bowl. Let sit until all of the peach puree has been strained through to remove any larger pieces of the fruit. Set aside.
Using a stand mixer with the whisk attachment, beat the cream cheese until fluffy and smooth; about 2-3 minutes. Add the peach puree and beat until combined. Stir in the powdered sugar until well combined and smooth. Stir in vanilla and a pinch of salt. Fold in non-dairy whipped topping and chopped peaches just until combined.
Keep frosting in the refrigerator until ready to use.
Place the first cake layer on your serving platter and frost just the top with the peach frosting (about ½" or so thick). Repeat with the next two layers, then frost the sides with an offset spatula (the long and skinny kind). Smooth top and sides until even, then top with sliced sliced fresh peaches for garnish, if desired.