Preheat oven to 325F with oven rack in middle position.
Grease the bottom and sides of a 2-quart baking dish.
Layer potatoes in the dish, seasoning each layer with a sprinkle of salt and pepper.
Bring heavy cream, milk, and garlic to a simmer in a medium saucepan.
Fish out garlic and discard.
Pour hot milk over potatoes and place on a 13- x 9-inch baking dish. Bake until bubbly and golden brown on top, about 90 minutes.
Remove from oven and grate nutmeg over top just before serving.