In order to do its job correctly, the nut's string groove must slant toward the peghead, down away from the plane of the fingerboard:
By the way, look at that beautiful piece of unbleached bone on this 1925 Gibson A-4 mandolin! Reminds me why I like natural materials.
If the string doesn't "bear" on the leading edge (the edge facing the first fret) it is likely to sound bad. It might rattle or buzz in the string groove, it might just lose some overtones, especially the highs, it might sound fuzzy, and in the worst cases it might play terribly out of tune as well.
In order to provide the downward string pressure, the peghead slants backward at an angle.
One electric guitar manufacturer found a way to conserve neck wood by avoiding the peghead angle:
So, they have to use a metal clip they call a "string tree" to hold the treble strings down at the nut so they don't rattle. Virtually no acoustic maker uses this device.
This next bit was rather tricky for me to photograph, so please bear with me, and if in doubt, rely on the text, such as it is. . .
So, how to diagnose this problem? If you've checked the nut action and determined that the nut is not too low, and your instrument buzzes only on the open string position, you may have a nut bearing problem. This is quite a bit more common than you might imagine.
It's usually easy to spot.
First, just lift the string out of its notch and look VERY closely at the bottom of the string notch. If the string bears cleanly through the notch, there probably is a shiny spot, some discoloration or other mark where the string sits.
In the subsequent close-ups, the peghead is UP and fingerboard DOWN
If the string is bearing midpoint in the nut, it might look just about like this:
Look at this! The dirt and wear show exactly where the string sits in this banjo nut. And, yes, it sounds really "fuzzy" in the open position.
Sometimes you can test for the string bearing point without even taking the string out of the groove. Just shove the string from side to side and notice where it bends:
If you compare these two close-ups you should be able to see that the string bends right at the midpoint of the nut. Maybe I should have a streaming video. OK, maybe not, but it was much easier to see while I actually bent the string back and forth.
Now, I've filed the groove in the nut so that it slopes downhill toward the tuners, and I'll try the same test again:
Can you see that? I've pulled the string much farther over, and it still bears on the front (fret) edge of the nut. It now sounds clear and clean as well.
This little trick will usually work on strings of any gauge. I just used a banjo second string for clarity.
Bear in mind that I'm really exaggerating the string bending here. It's not necessary to bend the string very far at all, just enough to see where it pivots. If you yank really hard on the string, you may cause it to roll around in the bottom of a wide slot. This is not what we're looking for. A too-wide nut slot is not necessarily all that bad. The strings may still bear at the front edge, be supported along the entire slot, be at the right height and spaced evenly.
If your nut has deep grooves that are too wide, and the whole thing looks messy, don't fret. Er, I mean, don't worry about it. The biggest concern is performance. The nut's job is to hold the string at the right place and allow it to draw smoothly when tuned. That's about it. You can be ugly and still do your job!
Here's a quick little drawing illustrating some possible nut slot cross-sections:
"A" is more or less the ideal. About half of the diameter of the string sits below the top of the leading edge of the nut.
"B" is a very deep slot that holds the string in a reasonable way, but may look a bit ugly from the top because the string sits so far below the surface. The walls of this slot may be parallel or they may diverge in a wide "V." Either way, if the bottom of the slot is the same as the bottom of "A" then the nut will work just fine.
"C" is a slot which is too wide, or has a flat bottom. Frequently, this kind of nut slot actually works OK because the string tends to be pulled to the side, and doesn't rattle around in the slot and has some bearing on the bottom of the slot. Many times, though, this wide slot allows too much sideways motion of the string, and will give a symptom of buzzing, rattling, or unclear open string notes.
"D" is the only style of nut slot that actually doesn't work in most cases. The string bears downward and is pinched, causing it to "ping" when tuned up. Because the string is held by the sides of the slot, it tends to wear a path downward and before long the string is sitting too low in the slot, buzzing like crazy on the "open" notes. In fact, this kind of nut can pinch the string so hard that skinny strings (octave "G" on a 12-string, for example) will actually break before they can be tuned to pitch!