The Butterfly operation is used to increase the contrast of the peaks in the Hough transform by applying a special convolution kernel (shaped as a butterfly). The convolution kernel is designed to match the shape of a typical Hough peak (bright region in the middle and dark regions on top and bottom).
Since the dimension of the peaks depend on the resolution as well as the location of the peaks in the Hough transform, it is impossible to have a perfect convolution kernel. Currently only a 9x9 convolution kernel is available. The kernel was taken from Krieger Lassen Ph.D Thesis .
This operation has three parameters: the size of the convolution kernel (locked at 9x9 in the present version) and two limits. The convolution creates an image (RealMap) with negative and positive values that can potentially range from negative infinity to positive infinity. To bring this RealMap to a conventional HoughMap (ranging from 0 to 255), a lower and upper limit are set by the user. All values lower than the specified lower limit are set to 0 and all values higher than the specified upper limit to 255. This procedure is called flattening of the RealMap. The limits depend on the values of the convolution kernel. Values of -800 and 800 for the lower and upper limits are typically a good choice for the 9x9 convolution kernel.