Begin by cutting your loaf of challah or brioche bread into 1-inch cubes. Spread the cubes out on a baking sheet and let them sit at room temperature for about 30 minutes to slightly dry out.

In a large mixing bowl, whisk together eggs, whole milk, heavy cream, granulated sugar, and brown sugar. Add vanilla extract, cinnamon, and nutmeg, mixing until combined.

Gently fold the bread cubes into the custard mixture, ensuring each piece is well-coated. Transfer the mixture to a greased 9x13-inch baking dish.

Cover the dish tightly with plastic wrap, and place it in the refrigerator for at least 4 hours or overnight.

In a separate bowl, mix together flour, brown sugar, cinnamon, and salt. Cut in cold unsalted butter until coarse crumbs form.

Preheat your oven to 350°F (175°C). Spread the streusel topping over the soaked bread mixture. Bake for 45 to 50 minutes.

Let the casserole cool for about 10 minutes before serving. Offer optional toppings like maple syrup or whipped cream.