The Downer name suggests links between the families; but the coincidence of addresses on top of that would seem to nail it for sure.

So why would James give his son the middle name Downer? His wife's, Edmund's mother's, maiden name was Jane Hill (though in the 1881 census, the Edmond Hoare head-of-household's "mother" is "Jane Done" , suggesting Jane had remarried after Edmund was born). A guess of course is that one of their (James' or Jane's mothers') maiden names was Downer, that this ~1790 Miss Downer had a brother xxx Downer. Brother xxx Downer in turn had a son Edmund Downer, born, as his 1st cousin Jane was, ~ 1816, and who grew up to marry Ann Saxby in1836.

This would make the Edmund-cum-Saxby the 1st cousin once-removed, of Edmund EWTH's erstwhile father (ie, possibly standing in for a deceased Edward).

A quick check in LDS of Edmund Downer 1820 ± 20 yrs. yields four listings (and a duplicate) in the International Genealogical Index (none in any other LDS source):

And among them, lookee here!

Possibly Harriet was James's older sister? And maybe Ann Saxby was a 2nd wife (if Edmund was older and passed away, Ann might well have moved in with his relatives at #44).

There's also:

.....Or even a third wife?