ls pics/
Instead of copy-and-past, we interpolate the copied region by solving the Poisson equation. More specifically, Eq. (10) in [Pérez 2003] is used here for seamless cloning. Let $\mathbf{f}$ be the pixel values in the region to paste (which is what we're trying to solve), let $\mathbf{g}$ be the corresponding region in the source image. Instead of letting $\mathbf{f}=\mathbf{g}$, we let $\nabla \mathbf{f} = \nabla \mathbf{g}$, and let the pixel values on the boundary to be the same as those from the target image. In this way, the pixel values $\mathbf{f}$'s distribution is "guided" by the source image, and gradually blends to the target image on the boundary. More rigorous derivations can be found in [Pérez 2003], the main idea is to let the pixel values be the same on the boundary, and find a distribution that resembles the source image and changes to the target image smoothly.
The only technical difficulty is to implement the Laplacian operator. This is a good reference. We first build the matrix $\mathbf{D}$, then build matrix $\mathbf{A}$ by setting $\mathbf{D}$ as block diagonal element, and set the $\mathbf{I}$'s:
Now our equation becomes:
$\mathbf{Af} = \mathbf{Ag}$, inside the region;
$\mathbf{f} = \mathbf{t}$, outside the region.
where $\mathbf{g}$ is the pixel value of the source image, $\mathbf{t}$ is the pixel value of the target image.
Now we are ready to implement our Poisson editting algorithm. First, we load the images and the masks:
Note the mask tells us what region to extract from the source image, when we insert to the target image, we may want to translate it, so we need an offset parameter. In here, I set the offset value directly, I'll talk about how to find the desired value later.
Now we translate the source image according to the offset:
Now we need to generate the matrix $\mathbf{A}$. First, apply our function to get the Laplacian matrix:
We only want to apply the Laplacian operator inside the blending region, so for the outside part, we set it to identity. Note for each row in mat_A, if it takes the Laplacian, then the row will have a "4" on the diagonal and four "-1", so to set it to identity, we want to set the "4" to "1", and the rest to "0":
pwd
from scipy import sparse
from os import path
import cv2
import numpy as np
import matplotlib.pyplot as plt
%pylab inline
from tqdm import tqdm_notebook as tqdm
import sys
sys.path.append('bin/')
from poisson_img_editing import laplacian_matrix, poisson_edit
# setup figure template
figure_template_path = 'bin/'
if figure_template_path not in sys.path:
sys.path.append(figure_template_path)
from importlib import reload
import utils
# force reload of the module
reload(utils)
from utils import std_plot,display_dataframe, embed_pdf_figure, embed_pdf_pages
fontlegend = {'family':'Arial',
'weight' : 'normal',
#'linewidth':0.5,
'size' : 6.5*1}
source = cv2.imread("pics/test1_src.jpg")
target = cv2.imread("pics/test1_target.jpg")
mask = cv2.imread("pics/test1_mask.jpg", cv2.IMREAD_GRAYSCALE)
fig,ax=plt.subplots(1,3,figsize=(15,5))
ax[0].imshow(source[:,:,::-1])
ax[2].imshow(target[:,:,::-1])
ax[1].imshow(mask, cmap='gray')
fig.tight_layout()
embed_pdf_figure()
print('src shape:', source.shape[:-1])
print('target shape:', target.shape[:-1])
print('mask shape:', mask.shape)
permc_spec = ['NATURAL', 'MMD_ATA', 'MMD_AT_PLUS_A', 'COLAMD']
%time fusion_1 = poisson_edit(50, 100,3,permc_spec[0],-2,2,-2,-2,source, target, mask)
fig,ax=plt.subplots(1,3,figsize=(15,5))
ax[0].imshow(source[:,:,::-1])
ax[1].imshow(target[:,:,::-1])
ax[2].imshow(fusion_1[:,:,::-1])
fig.tight_layout()
embed_pdf_figure()
source = cv2.imread("pics/test2_src.png")
target = cv2.imread("pics/test2_target.png")
mask = cv2.imread("pics/test2_mask.png", cv2.IMREAD_GRAYSCALE)
fig,ax=plt.subplots(1,3,figsize=(15,5))
ax[0].imshow(source[:,:,::-1])
ax[1].imshow(mask, cmap='gray')
ax[2].imshow(target[:,:,::-1])
fig.tight_layout()
embed_pdf_figure()
print('src shape:', source.shape[:-1])
print('target shape:', target.shape[:-1])
print('mask shape:', mask.shape)
%time fusion_2 = poisson_edit(150, 150,2,permc_spec[2],4,4,4,3,source, target, mask)
fig,ax=plt.subplots(1,3,figsize=(15,5))
ax[0].imshow(source[:,:,::-1])
ax[1].imshow(target[:,:,::-1])
ax[2].imshow(fusion_2[:,:,::-1])
fig.tight_layout()
embed_pdf_figure()
Reference¶
Patrick Pérez, Michel Gangnet, and Andrew Blake. 2003. Poisson image editing. ACM Trans. Graph. 22, 3 (July 2003), 313-318. DOI: https://doi.org/10.1145/882262.882269