I wanted to write a long post, but now I have limited time.

Firstly, we will introduce the concept of video compression format and media container format. Your notion of "video format" is slightly general, in the sense that it can mean both compression and container to people. In addition, you stated "avi rmvb mpg" which are file extensions. The file extensions identify the media container format, not video compression.

Now look at what we have in practice. A video clip captured by your video camera, a music video, or a movie; these things each has a moving picture track (which is a series of pictures), and a synchronized audio track. The video compression deals only with the series of pictures, without audio, producing compressed bitstream. We have audio compression format to deal with the audio to produce another compressed bitstream. Both bitstreams are multiplexed together with additional information like timecodes added. This multiplexing process is specified by the media container format.

We define quality here as the video fidelity. The media container doesn't influence the video quality.
Examples of media container:
AVI (avi)
Flash Video (flv)
Matroska (mkv)
MPEG-1 and MPEG-2 (mpg)
MPEG-4 (mp4)
Ogg Container (ogm)
Quicktime (mov)
RealMedia (rmvb)

Examples of video compression:
MPEG-1
MPEG-2
MPEG-4 Part 2 (DivX and XviD should sound more familiar)
MPEG-4 Part 10 (AVC or H.264)
RealVideo (Several versions)
VC-1
WMV (several versions)

At this point, you should understand that when you have a video file, both media container and video compression are used. Inside the media container, video compression is used. The purpose of video compression is to compress video, resulting a smaller byte size; the purpose of media container is to interleave video and audio data, adding of time code to provide a synchronized video and audio playback.

Generally, to produce a video file (with audio), you'll choose 3 formats: video compression, audio compression, and container. You can mix and match any of those formats. However this is not true for some restrictive containers like RealMedia, which can only have RealVideo inside.

Now comes to discussion on video quality. This discussion will come togeteher with file size. For all video compression formats, the bitrate can be set very high for high quality; or the bitrate can be set low for small file size. What is important is the "ratio" between quality and file size. Since the main purpose of video compression is compressing video, a good compression format has to produce good quality at low bitrate (small file). Currently, the best compression format in this sense (good quality at low bitrate) is MPEG-4 AVC (also called H.264). MPEG-4 part 2 (ASP) and RealVideo 10 follows closely. It should be noted that the encoder plays a big role in achieving the high compression with high quality. A good encoder is one that does that. For H.264, x264 (the name of a software encoder) is among the top. For MPEG-4 ASP, XviD and DivX6 are very good.

Enough on video compression, we look at media containers. Since the video quality is not affected by containers, the first thing we look at for choosing a format is the filesize. Recall that containers add timecodes and other stuffs like index. These are called overhead. The lesser the overhead, the smaller will be the filesize. Matroska has the smallest overhead. MPEG-4 2nd place. AVI can be said having 3rd place. Well, AVI does have several variants like OpenDML and traditional. I personally use Matroska; it has advantages on filesize, and seeking response. The only disadvantage I can think of is playing on DivX DVD player is a compatibility issue.

My choice of format is the one making smallest file size for a good quality.
Video: MPEG-4 AVC (encoder: x264)
Audio: MPEG-4 AAC (encoder: neroaacenc)
Container: Matroska (muxer: MKVtoolnix)
This is of course for playback on computer.

One last note, the file extension specifies the container, nothing about video and audio compression format.

This post has been edited by xenon: Aug 7 2006, 09:11 AM

The inefficiency of container doesn't cost much. I tested on one sample:
RMVB: 166.2MB
MKV: 164.8MB

I can't get RealVideo and AAC into AVI using AVI-Mux GUI.

So, another sample, which contain "XVID" + "MP3":
AVI: 40.8MB
MKV: 40.4MB

And I don't like AVI much. Modern compression formats can't be stored in AVI correctly, for example H.264. And the overhead is not as small as MKV. I don't think "ppl choose .avi mostly because of its small size". The overhead of AVI is not small from my view. General purpose containers that I know are just Matroska, AVI and Ogg. Ogg is worst, Matroska is best, in terms of overhead.

Your RMVB most likely contain RealVideo compression that I can't get it into AVI. Use Matroska instead, well, you don't save much anyway: about 1MB for a 120MB file.

It can't work wonders. But when watching VCD (resolution 352x288), I use Lanczos resize filter doubling the dimension in both horizontal and vertical, giving 704x576. Also with some sharpening in the resize filter.
Then, I might try a bit of Unsharp mask.

The filters are applied using ffdshow. This is easy when your video files are in the MPEG family (DivX and XviD are MPEGs).
If you want to watch RealVideo, then to apply filter, you can use AviSynth.