About 1650, James came to Pequot (New London), CT. A house which later became the old Avery Homestead, known as the "Hive of the Averys," and stood at the head of Birch Plain, was built around 1656. Here, the second Captain James Avery, oldest son of the founder, lived with his family from 1671 until his death in 1728.
In June of 1684, the old Blinman edifice at New London, "The unadorned church and watch-tower of the wilderness, was sold to Captain James Avery for six pounds, with the condition that he should remove it in one month's time." According to tradition, it was taken down and its materials were carried by river and sound, and added to the 1656 house at the head of Poquonock Cove.
For over 230 years, successive families of Averys lived and reared their children at the Hive, generation after generation. The venerable old house withstood the storms of winter and the heat of summer; the hand of time gently touched every ancient board.
The descendants of James and his father Christopher began to consider it forever indestructible until one night on July 20, 1894, a spark from an engine passing over the railroad bridge set fire to the time-seasoned roof, and soon, engulfed by flames, the old Avery Homestead fell.
Its destruction filled the hearts of the members of the Avery clan with sorrow and dismay, but soon their grief gave way to plans for memorializing the house by establishing a memorial society and a monument on this spot to show to generations yet unborn that here lived the men of olden times, James Avery and his sons and grandsons, who were of such prominence in the very beginnings of the colony of Connecticut in this section of New London and Groton.
Soon after the destruction of the old "Hive," The Avery Memorial Association was incorporated. The owner of the homestead site, James Dension Avery, gave the land by deed of gift to the new Avery Association.
The granite shaft is surmounted by a bronze bust representing Captain James Avery, a Puritan, Magistrate, military leader and co-commander of the New London area militia who fought in the King Philip's War (1675–1677). The monument was designed by the noted sculptor Bela L. Pratt, an Avery descendant. The shaft rises from the middle of the outline of the old house. Whoever walks around the monument today will see the old stone steps where Averys for generations entered the old homestead. Parts of the chimney still stand, ivy covered now. The monument was dedicated July 20, 1900. Six hundred descendants gathered on that day to honor their illustrious ancestor and the ancient home spot. Dr. Elroy M. Avery, the President of the Association presided; Helen Morgan Avery unveiled the bust; Frank Montgomery Avery delivered the oration:
"We have assembled here today to join in the dedication of a memorial; not of a monument erected to the memory of a great historical event or the accomplishment of some illustrious personal achievement, but of a memorial designed to designate the spot where the roots of a family tree struck deeply and strong enough to take permanent hold in the soil of New England, and whence its branches have spread out far and wide and lusty with the strength of the parent stock. It is to honor the place of our forefathers' refuge and abode, of their struggles and triumphs, their birthplaces and their death place through the early generations, that this shaft has been erected on the site of the first of their homesteads, and that we are here today."