Midget refers to something which is smaller than usual. Most frequently, this is a person who is affected by dwarfism, and in this context it is usually considered to be derogatory. Compare nigger as used of African-Americans.
"Midget" is a general term, however, and is also used for example to denote a class of small objects, as a class of very small sailboats or racing cars.