One general challenge is getting people to act properly. What counts as proper behavior is, of course, a rather contentious matter. However, it seems reasonable to believe that at the most basic level harming others is not proper behavior.
It can be argued that self interest will motivate people to act properly. The stock argument (which is based on Hobbes, Locke, and Smith) is that a rational person will realize that behaving badly is not in his self interest because the consequences to himself will be negative.
Naturally, a person might be tempted to act badly if she thinks she can avoid these consequences, which is why it is rather important to make sure that these consequences are rather difficult to avoid. In addition to this concern, there are also other concerns about self-interest as a regulating factor on bad behavior.
First, for self-interest to be a regulating factor, a person’s self interest must coincide with acting correctly. If a person’s self-interest (or what he believed is his self-interest) goes against acting correctly, then he will be inclined to act incorrectly. Not surprisingly, various philosophers have tried to argue that what is truly in a person’s self interest is to act correctly. While there are some good arguments (such as those presented by Socrates) for this view, there are also good arguments that this is not the case. Naturally, from a purely practical standpoint the trick is to get people to believe that their self-interest coincides with not acting badly.
Second, even if it is assumed that it is in a person’s interest to act correctly this will not motivate a person to act correctly unless a person knows what is in her self-interest. While it is tempting to assume that a person automatically knows what is in her self interest, this need not be the case. After all, a person can think that something is in her best interest, yet be mistaken about this. A person might be misled by his emotions, confused or wrong about the facts (to give but a few examples).
Third, even if it is assumed that a person knows what is in her self-interest and that it is in her self-interest to act correctly, there is still the question of whether the person will chose to act in accord with her self-interest or not. To use a simple example, a person might know that exercising is in her self-interest, but be unable to stick with exercising. Roughly put, a person might have knowledge but lack the will or motivation to act on this knowledge.
Thus, self-interest can play a role in regulating behavior-provided that it in accord with correct behavior, the person has knowledge and the will to act on this knowledge.