The museum's current collection grew out of the early collection of Jacob Olson, an early immigrant to Bosque County. Mr. Olson collected the important artifacts of that time and saved them for his own enjoyment. In 1924 he gave his collection to Clifton Lutheran College with the proviso they be housed in a fireproof building and should never leave Bosque County. When the college merged with Texas Lutheran College in 1953, a new home had to be found for the collection.

A local non-profit corporation was formed in 1954, monies were collected and donated and a new building was opened in 1957 on land donated by the former Clifton College Corporation. Since that time three additions have been made to the building increasing the total display space to 8,000 square feet. An important recognition of the museum occurred on October 22, 1982 when King Olaf V of Norway visited the museum to view the Cleng Pearson artifacts.

A log cabin, Jacob Olson's childhood home, was moved to the museum ground in 1985 and received a Texas Historical Marker the following year.