Siddhesh Sali

← All work

ml

Generating Camouflage from a Camera Feed

Read the surroundings from a camera, synthesise a pattern that disappears into them — the machine-learning half of a friend’s thesis on a sheet that could change its own skin.

A rendered human figure on black, its whole surface covered in a green, brown and khaki pattern of vertical broken strips.

Camouflage is usually a decision made months in advance, in a factory, about an average of the places a garment might end up. The premise of this project is that it does not have to be: if a surface can change its own pattern, then the pattern can be computed from what the surface is actually standing in front of, now.

The surface was somebody else’s problem. This was a friend’s thesis in Metallurgical Engineering, and the materials half — making a sheet that can display a pattern at all — was theirs. I took the other half: given a camera pointed at an environment, produce the pattern it should display.

What a place looks like, numerically

A camouflage pattern has to get two things right, and they are separable.

Colour is the easy one. k-means over the frame returns the ten colours that account for most of it, as hex, and that is the palette. It costs nothing and it is the part that fails most visibly if you get it wrong — the eye forgives a wrong shape long before it forgives a wrong green.

Structure is the hard one, and it is where most of the work went. A bamboo grove is vertical and long; cracked ground is angular and closed; scrub is neither. That character lives in the edges — but not in all the edges. Canny returns a confetti of every gradient in the frame, most of it noise at this scale. Holistically-nested edge detection returns the edges a person would draw, which is a different and much smaller set, and it is the difference between a pattern that has the right texture and one that merely has the right colours.

SLIC superpixels do the complementary job — collapsing the frame into a few hundred connected regions of near-constant colour, which is both a cleaner way to read the palette and a rough map of how big the patches in the pattern ought to be.

Two ways to draw the result

The first synthesises the texture directly: a pix2pix model trained on edge-map-to-texture pairs, so an edge map extracted from a new scene produces a plausible surface with that structure. It gives the best-looking results and it is the slow path — a trained model, a GPU, and a pipeline of edge extraction, padding and connected-component labelling in front of it.

The second is a polygon generator, given the extracted palette and a patch size. It knows nothing about the scene beyond those numbers, and it runs in milliseconds. That is what made the live version possible: a webcam frame in, a k-means palette out, a fresh pattern generated per frame, and both streams — what the camera sees and what the sheet would wear — served side by side over Flask.

Watching that run is the moment the idea stops being abstract. Point the camera at a wall, and the pattern goes the colour of the wall.

Does it actually hide anything

The only honest test is to look. The patterns were mapped onto a rigged human model in Blender, rendered, and composited back into photographs of the environments they were generated from. The first image in the gallery has five figures in it, and they are findable — the pattern reads darker and greyer than the grove it is standing in, which is a palette problem, not a structure one.

Nothing here is quantified — there is no detection-rate experiment, no observer study, no measurement of how long anyone took to find the figures. That is the obvious next thing and it is exactly the thing that was not done. What this is, is a pipeline that turns a photograph of a place into a pattern belonging to that place, fast enough to do it live.

Credit where it is due

Very little of the machinery here is mine. The pix2pix implementation is the CycleGAN-and-pix2pix codebase from Zhu, Park, Isola and Efros; the edge detector is a published HED implementation; the polygon generator is camogen by Gael Lederrey, itself a Python port of Ulf Åström’s work. What I built is the pipeline between them, the live path, and the decisions about which signal to take from the scene.

The imagery on this page has borrowed inputs. The source textures and the scene the figures stand in are stock photographs, and the figure itself is a third-party 3D model. The patterns, the pipeline and the composites are mine; the raw material is not, and the exception is recorded in this site’s NOTICE rather than glossed over.

There is no public repository for this one — it was a contribution to somebody else’s thesis, and the write-up belongs to them.

Gallery

0606
A bamboo grove with a wooden boardwalk running through it. Five human figures stand among the stems, surfaced in a green and brown pattern generated from the scene.
Fig. 01Five figures in this frame. Find them, then judge it — that is the only test there is.
A close photograph of a bamboo fence: dozens of dry vertical canes, pale yellow and brown, with dark nodes across them.
Fig. 02The input. One frame of an environment, nothing else.
Three panels of the same bamboo texture: the original, a Canny edge map that is a confetti of short broken lines, and an HED map showing continuous soft vertical strokes.
Fig. 03Canny in the middle, HED on the right. Canny finds every edge; HED finds the ones that describe the place.
A generated pattern of vertical broken strips in four greens and a brown, outlined in black, filling the frame.
Fig. 04The output for that input. Not the bamboo — the statistics of the bamboo.
The same green and brown strip pattern tiled seamlessly across a wide frame, mirrored at the repeats.
Fig. 05Tiled, ready to be a texture. The mirror at the seam is the giveaway, and the thing to fix next.
A dense pattern of irregular pale-brown, dark-brown and grey fragments, generated from a dry-ground photograph.
Fig. 06The same pipeline on a dry, broken-ground texture. Nothing in the method knows what bamboo is.