Shor's algorithm uses two registers. We describe the state of the algorithm at any time by associating a complex number called an amplitude to each possible pair of values the two registers might take. (This is analagous to describing the state of a probabilistic algorithm by associating a real number - the probability - with each possible deterministic state.) The amplitude is the "arrow" (as Feynman calls it in his book QED) associated with the corresponding event.
The first rule regarding amplitudes is that, if the system were to be fully observed at any time, then the probability of finding a particular outcome (i.e., a particular pair of register values) would be equal to the squared length of its current amplitude. The second rule ("adding arrows") is that, if there are several alternative ways for an outcome (again, a pair of register values) to arise, then the amplitude for the event is the sum of the amplitudes of the alternatives. Since amplitudes can cancel, this is the mechanism for destructive interference, which is in turn what allows the Fourier transform to be computed.
The quantum computation that achieves this is analagous to a probabilistic computation that chooses a random A for the first register and then computes the corresponding value XA mod N for the second register.
For instance, if R was 4, the resulting sequence might be 0,S,0,0,0,S,0,0,0,S,0,0,0,S,0,0,..., where the sequence is Q numbers long and S is approximately 1/sqrt(Q/R).
At this point, we know the ratio D/R - we don't yet know D or R. However, we can easily compute relatively prime A and B such that A/B = D/R. If D is relatively prime to R, then B will be R. Since each non-negative integer smaller than R is roughly equally likely to be D, and at least a fraction proportional to 1/log(log(R)) of these values are relatively prime to R (for any R), we find R with probability proportional to at least 1/log(log(R)).