The whalers used Rosetta Harbour near The Bluff as anchorage for their ships. A certain degree of rivalry developed between the whaling stations with instances of assault between the whalers, and tales of smuggling. 

In April 1851, the fishery comprised 10 or 12 buildings including boat sheds, stables, workshops, and sleeping berths. Amid a background of controversy over the future use and development of Encounter Bay, the government planned to upgrade port facilities in general and in 1854 notified its intention to construct a wharf at The Bluff in the Gazette. The District Council of Encounter Bay, formed in August 1853, was asked to provide a road approach to the wharf, which it completed in July 1855. 

Alexander Ewen, grandfather of George and William, arrived in the area about 1848 from Glasgow in Scotland. A whaler all his working life, and held the position of the last manager of the whaling company at the Bluff from 1871 to 1878.

As the whaling industry was nearing an end, Port Elliot was the major centre for the district, as its port was the outlet for the wool and goods shipped down the Murray River by paddle steamer and barge.

During the early years of European colonisation a cutter fleet supplied much-needed food for the settlers, while other fishers, operating small sailing boats or using lines or nets from the shore, hawked their catch door-to-door. People also simply caught fish to feed themselves and their families.

Two separate groups operated from earliest days: local people who netted from the shore, and individuals or a couple of mates or brothers who fished from small sailing boats. Soon they were joined by a third group, who sailed the cutters around the Gulf of St Vincent, Investigator Strait and Backstairs Passage for a week or so at a time, until the well in the boat was full or all their food had run out. All were at the mercy of the sea and the wind.

The cutters, which sailed from Glenelg to Kangaroo Island and around Spencer Gulf, stayed out for about a week, until the well in the boat was full of fish (whiting for preference) or until supplies ran out. Sometimes these reports were of the loss of boats and lives, because, with no motors, no forecasts and, initially, no lighthouses or beacons, all were at the mercy of the weather.

Port Victor was established in the lee of Granite Island and the township laid out in 1863. It was named by Captain Crozier after his ship the Victor.

One of the first settlers to the area was the Reverend Rigdway Newland with a party of 33, being family, friends and servants. Newland selected land at Encounter Bay following the advice of Governor Gawler. Newland named the southern end of Encounter Bay Yilki, in honour of the native name for the district.

Following favourable reports about the area's suitability for colonisation from explorer Captain Sturt, Colonel William Light arrived at Encounter Bay looking to establish a capital city for the Province of South Australia. Although the first Governor Hindmarsh preferred the Encounter Bay surrounds, Light would recommend the present day location of Adelaide.

In 1921 a French ship named the "Eugene Schneider" was almost wrecked off the coast of Wauraltee as the Captain had mistakenly sailed into the "unofficial" Port Victoria, on the Yorke Peninsula instead of the official Port Victoria (Port Victor/Victor Harbour) on the Fleurieu Peninsula. Having misread the map they ran aground on the reef there, but survived. The ship albeit damaged still managed to deliver their cargo to the Screwpile Jetty on Granite Island before sailing on to Melbourne for repairs.

 The town's name was changed to 'Victor Harbor', to avoid confusion with Port Victoria in the Yorke Peninsula. 

Throughout the years "Victor Harbor" has been spelt both ways in the newspapers of the time so why it was officially annexed as "Harbor" could be as simple as that it was the preferred or chosen spelling in South Australia at the time, although "Harbour" with a u was Crozier's original spelling.

This spelling, found in several geographical names in South Australia, including Outer Harbor, is the result of spelling errors made by an early Surveyor General of South Australia. Interestingly, the Victor Harbor railway station is spelt with the u.

Earliest White Settlement in State

• The Mail (Adelaide, SA : 1912 - 1954) 

• Sat 18 Dec 1926 

When Encounter Bay Nearly Ousted Adelaide 

Adelaide once had a serious rival as the capital of the State. Col. Light and Governor Hindmarsh differed so strongly that the matter was put to the vote of the whole of the landholders of the State.

Col. Light favored Adelaide and Governor Hindmarsh liked Encounter Bay. By 218 votes to 137 Adelaide won. That was in 1836, when there were about 400 people in South Australia. 

The first hotel built in the State— 'The Fountain'— is still standing at Encounter Bay, along with other old buildings rich in historic associations. The crumbling walls of the old whalers' shed lie in the shadow of The Bluff, beyond which is Petrel Cove, made famous by Simpson Newland, in 'Paving the Way.'

A fund of historical data has been gathered by Mr. Robert Thomas Sweetman, one of the oldest living residents of Encounter Bay. Born at Oaklands, near Adelaide, he celebrated his eightieth birthday recently. He went to Encounter Bay when seven years old. Whaling was in full swing in those days, and Mr. Sweetman helped to supply the whalers with meat. 'There were 15 men in the first boat and 13 in what was called the pick up boat," he explained. 

"I cannot recall the names of every one, but among them were Bob Cleeve, Billy Honeyman, Jim and Tom Clark, Jack Jones, Jack Taylor, Jim Long, Alex Ewen, Bill Harris, Jack Gangel, Peter Morgan, Jack Parsons, Jack McCarthy, Jim McDonald, Dan Budie, Frank Buckley, George Bennett, Jack Foster, Rube Earl, Tom Smith, Tom Atrill, Joshua Paterson (cooper), and Jack Hyde (boat builder). 

"The old song that relates how seamen used to 'drink their rum by the keg' was literally true of the whalers. There was no beer in those days, but there was always a keg of Jamaica rum on tap in the whalers' quarters, and they drank it like water. Hanging on nails alongside the keg were tin pannikins, and the visitor was promptly handed a pannikin of red raw rum when he paid a call. It must not be supposed that the whalers were all old lags. Some were certainly ticket of leave men from Tasmania, but many of them were seafaring men. 

"The largest whale caught at Encounter Bay measured 74 ft. 7 in. in length. An attempt was once made to catch a larger one, but it smashed the boat, which sank in 10 fathoms of water about half a mile from The Bluff, where it still lies. Several of the men were smashed up. That is the incident on which Mr. Simpson Newland based the smashing up of Dave Cleeve, the headman, in 'Paving the Way.' " The identity of Bob Cleeve, the original of "David Cleeve" in the book is thus thinly veiled. 

"Bob Cle(e)ve was for a time actually headman in the first whaleboat," stated Mr. Sweetman. "He was equally skilful with the harpoon or gun. On the boat was a queer old gun which the whalers used to fill with any old scraps of lead. This was charged into the side of the whale at the moment the harpoon was buried in its back. Billy Honeyman acted as harpoonman for years. Hobart, Tasmania, where the whalers came from in the first place, was always referred to as Hoba'town. For years Mrs. Malin Rumbelow had the old gun, but I understand it is now in the Adelaide Museum. When the whaleboat was smashed up Cleeve, Jones and Harris started a secondhand shop in Rosina street, Adelaide. 

First Hotel in State 'The manager of the whaling station was Capt. Blenkinsop. He came to an untimely end in 1830. Declaring that there were no breakers round any part of the South Australian coast he could not steer a boat through, he was challenged to take one through the rip at the Murray mouth. He essayed the task, taking four men in a whaleboat. When the craft was halfway through it capsized, and only one man escaped. Blenkinsop's body was afterward recovered. Mr. Cain Jeliff, who started the fishing industry with Mr. Malin Rumbelow, sen., was also drowned between 30 and 40 years ago at the Murray mouth.

'Mr. Sweetman never had a day's schooling, his mother being killed when he was four years old. When nine years of age the young colonist decided to seek his fortune in the world.'

"I could drive a bullock team when I was 10 years old,'' said the old man proudly, 'and could cart wheat, too, although I could not up-end the bags. There is nothing now that I cannot turn my hand to," he added, and his agility proves the truth of his assertion. Mr. Sweetman travelled to Encounter Bay in a bullock dray, the journey occupying four days. The party went through Willunga, which then consisted of a bush inn and one store. The South road was only just surveyed. 

"Encounter Bay was the first white settlement in the State," said Mr. Honeyman. "A portion of it known as Yilki, consisting of about 40 acres, was surveyed before Adelaide. Governor Hindmarsh came down from Adelaide, bringing with him a surveyor named George Moore, an Englishman, and Yilki was surveyed." 

Nowadays Mr. Sweetman owns a large portion of Yilki. The house he is at present living in was built of stringy bark and palings in 1837, and he proudly displayed the title deed, which is dated December 23, 1837.

In this episode curator Dr Adam Paterson and archaeologist/ educator Dr Chris Wilson unpack the history of South Australia’s whaling industry against the backdrop of the SA Maritime Museum’s exhibition 'Leviathan: An astonishing history of whales'.