To composite a figure onto a new background it is necessary that the figure is in some way separated from its old background. If the figure has not been shot against a green screen then the chances are some degree of rotoscoping is required. If the job is not to be hell then it is advised that the following guidelines are observed:
The 'uber aim' is to have a smoothly roto'd figure that looks right on its new background. This requires artistry as much as anything else: the shape and movement of your target should be observed, absorbed and assimilated into a creatively consistent whole.
Rotoscoping is essentially a drawing activity and as such a certain amount of creative leeway is allowed. It is ok not to roto absolutely all the details. Shoes got shoelaces? get rid of em! You will need good visual judgement in selecting what not to include in the roto.
Begin at the most complex point
The roto shapes should be laid down at the point of the sequence that requires the most number of shapes and points to outline.
Only roto what is necessary
It might be that for a particular job only a small part of the figure needs a roto.
Be consistant with edges
It is ok if your roto is very 'tight' and hugs the figure exactly, it is also ok if it is loose and floats one or two pixels away from the figure. Both can be fixed by a bit of dilate and erode. It is not ok if it is both as this will require that the erode and dilate are animated.
Roto'ing is an arm of animation. The flow of movement must be carefully observed. Check this flow by looking at the animated alpha of the roto seperate to the RGB.
The non-linearity of movement
In the case of a roto made for the swinging arm of a man: a single keyframe is placed at the begining and end of the swing in the expectation that this is all it will need (below).
However, half way through the swing the roto is out of place (below). This is because the swing is an accelerating movement.
The options are to add another keyframe or go into the Curve Editor and edit the animation curve of the shape. The shape of the curve can be edited by changing the length and position of the handles on the two end points. This will re-enge the roto with the arm (below). Movement, especially natural movement, is rarely linear.
Always check on the 'shape' of the animation in the Curve Editor (as above).
Roto node tips
Be economical with points
A shape in a Roto node is usually made with something called a bezier. If you are a digital artist of any worth you should get to know the basics of how the bezier works. A typical beginner mistake is to use too many points, but too many points can be hell to animate. Points should economically describe a shape and not smother it. Tip: A good test of your ability is to write the letter 'S' with only three points.
Transform the entire shape
In the Transform tab of the Roto node, the entire shape can be moved as a whole. For big movements this is preferable to moving each point. The big advantage of this method is that it is easier to edit a move if it is broken down into global shape moves (via the transform tab) and local point moves (switch to the Roto tab to make these moves). In the Transform tab translation, rotation, scale and skew are all separately animatable. Which of these values are to be keyed should be carefully considered (actually I habitually animate all of them whether I need to or not). It is wise to do as much of the roto animation in this tab. 'Tip': the skew parameter is more usefull than you might suppose.
Break down complex shapes into smaller shapes
When making a roto for a human it is easier to manage if their form is broken down so that their legs, arms, torso etc are all covered by separate shapes. The centre points for these shapes should correspond roughly to the position of the joints.
Organise the shapes into folders
Folders may be created inside of the Roto node, into which shapes can be dropped. This is a good way of organizing lots of shapes and also of collectively editing their movement. If this containing folder is selected then all its contained shapes are also selected. The movement of the folder contents may be collectivly animated.
Start at the folder level
Once the shapes have been organized into folders the folder animation should be your first point of consideration. The contained shapes should only be animated if necessary.
Attach the output from a Tracker to a shape or folder
The output of a tracker node maybe be linked to the translation values of the Transform tab of the Roto shape. However, most usually, they are linked to the containing folder.
Dont let the points wander off
The points of the roto should always correspond to the same feature point of the figure throughout the length of the animation e.g. a roto point on the elbow of a figure should remain at this corner throughout. If it doesnt then it will animate wrong.