AI Tech Inspire

Why Continuous Normalizing Flows Invent ‘Half-Dog Half-Cat’ Samples

What happens when a mathematically elegant generative model meets a messy, discontinuous world? That’s the puzzle behind why continuous normalizing flows (CNFs) sometimes produce uncanny “half-dog half-cat” samples. If you’ve trained flows in PyTorch or TensorFlow and inspected the trajectories over time, you might have noticed the strange in-between states. Here’s the key: the problem isn’t your code—it’s topology.

Quick facts from the question

  • CNFs transport a source distribution to a target via a smooth, invertible map (a diffeomorphism) on R^d.
  • Concern: if the real data distribution is topologically disconnected (e.g., cats vs. dogs, with zero mass in between), a diffeomorphic mapping may be too restrictive.
  • Example: lots of dog images and cat images, but no physical “half-dog half-cat” images; the support between modes should be zero.
  • Because the base (e.g., Gaussian) is connected, a diffeomorphism must map it to a connected support, potentially forcing nonzero density in the “in-between” region.
  • This can yield blended or implausible samples when generating from the learned flow.
  • Open question: Is the model incorrectly filling gaps due to its topology assumptions, or is it “learning” unseen intermediate regions?
  • Implication: Knowing the topology of the data distribution seems crucial; interest especially lies in the intermediate region during the flow (0 < t < 1) and at t = 1.

Why CNFs struggle with disconnected supports

Normalizing flows, including CNFs, rely on diffeomorphisms—smooth, invertible mappings with smooth inverses. A fundamental property: diffeomorphisms preserve topological connectedness. If your base distribution is a connected set (a standard Gaussian over R^d is connected), then its image under any diffeomorphism remains connected. In other words, a CNF cannot produce a true “hole” or a perfectly disconnected union of supports. It can make bridges extremely thin, but not eliminate them entirely.

That’s exactly where the “half-dog half-cat” samples come from. A maximum-likelihood-trained flow is incentivized to cover every training data point with nonzero density. If the true data lie on two separated manifolds in pixel space, a diffeomorphic model can only approximate this by creating a low-density “bridge.” That bridge lives in the unobserved, implausible region and sometimes leaks into generations.

Key takeaway: With a connected base and a diffeomorphic map, you cannot get a truly disconnected target support. The model will fill the gap with low—but nonzero—probability mass.

About those “in-between” states during 0 < t < 1

Even if the final distribution t = 1 looks sharp and mostly plausible, the intermediate states 0 < t < 1 typically pass through morphing shapes. That’s not a bug; it’s how the transport works. A CNF defines a continuous path of densities p_t. Mass has to travel smoothly from the base to the data, which means “cat-ness” can morph into “dog-ness” through visually mixed intermediates. If you visualize samples mid-trajectory, you’ll likely see blended images. These are not necessarily what the model believes the world looks like; they are waypoints in a smooth transport constrained by invertibility.

However, the same connectivity limitation also applies at t = 1. Unless the model makes the bridge vanishingly thin, some probability remains in invalid regions. In high dimensions, even a tiny bridge can occasionally be sampled—hence rare but weird half-cat outputs.

Is the model hallucinating? Or “learning the gaps”?

It’s not learning missing truths; it’s respecting its geometric constraints. A CNF trained with maximum likelihood can’t assign exact zero density to the in-between region without breaking invertibility. It’s not discovering magical intermediate classes—it’s forced to connect modes. The result is a good log-likelihood on data points, but an imperfect fit to the true topology of the data support.

This is also why flows sometimes shine in bits-per-dimension metrics but lag behind diffusion models in sample realism for complex, multimodal datasets. The metric rewards covering data well and penalizes missing mass, but it doesn’t strongly penalize sprinkling tiny mass in the wrong places.

Practical paths for developers

  • Use models that can change topology. Score-based diffusion models and SDE-based samplers (e.g., the family behind Stable Diffusion) aren’t constrained by invertible mappings. They can represent disconnected supports without creating bridges. If disconnected modes are expected, this class is often a safer choice.
  • Consider SurVAE flows or stochastic layers. “Surjective/variational” flow frameworks introduce non-invertible or stochastic transformations that allow topology changes while retaining tractable training objectives. They can carve out holes that pure diffeomorphisms can’t.
  • Discrete latents for components. Combine flows (or VAEs) with a discrete latent variable indicating mode/component. Each component can be modeled by its own flow, and the discrete variable handles the disconnection. In practice, this looks like a mixture-of-experts with a categorical latent.
  • Manifold-aware approaches. If data lie near a lower-dimensional manifold, consider manifold flows or injective flows that explicitly reduce dimensionality before applying invertible transforms. This can tighten support and reduce off-manifold bridges.
  • Post-hoc filtering or thresholds. Use a density threshold, a classifier, or a learned out-of-support detector to filter implausible generations. It’s pragmatic and often effective when you must stick with flows.
  • Temperature and truncation. Lowering sampling temperature or truncating latents reduces exploration of low-density regions, at the cost of diversity. This is the quickest knob to turn if you’re getting hybrids.
  • Rethink training objectives. Maximum likelihood pushes to cover all data; it doesn’t push hard against small spurious mass. If you can afford it, hybrid losses (adversarial terms, perceptual terms) or explicit penalties on off-manifold regions can help.

Why it matters for engineering decisions

Topology isn’t an abstract nuisance—it’s a design constraint that shows up in production. If your domain has natural separations (e.g., valid vs. invalid molecules, workable vs. unsafe robot trajectories, faces vs. non-faces), CNFs may allocate nonzero density in forbidden zones. In safety-critical or scientific settings, that’s unacceptable.

Before picking a model family, ask: is the data effectively connected in the representation space you’ll use? If not, you may prefer diffusion-style models or hybrid flow architectures that can change topology. At AI Tech Inspire, we see teams increasingly mix techniques: flows for likelihood and fast inference where topology is tame; diffusion or stochastic layers where reality has hard gaps.

Suggested references and deeper dives

  • Normalizing flows overview: Rezende & Mohamed, “Variational Inference with Normalizing Flows.”
  • CNFs and Neural ODEs: Chen et al., “Neural Ordinary Differential Equations.” Libraries like torchdiffeq play well with PyTorch.
  • Neural Spline Flows: Durkan et al., flexible yet still diffeomorphic.
  • Manifold Flows: Brehmer & Cranmer, learning lower-dimensional manifolds for densities.
  • SurVAE Flows: Nielsen et al., introducing stochastic and surjective layers to change topology.
  • Score-based generative modeling: Song & Ermon series; the SDE formulation highlights why disconnected supports are feasible without invertibility.
  • Flow Matching / Rectified Flow: alternative training of vector fields; check whether your implementation enforces invertibility or allows topology changes.

A mental model to keep

Think of a CNF as a smooth, reversible animation between two images of space. If the starting scene is one connected canvas, the ending scene is still one connected canvas. You can mash parts close or stretch them thin, but you can’t tear the canvas. If your data live on two separate canvases, a pure CNF will quietly draw a thread between them.

“If your domain is topologically disconnected, a diffeomorphic flow will bridge it—often imperceptibly, sometimes visibly. Choose the tool that matches the topology.”

Curious to put this to the test? Try visualizing p_t snapshots during training, sweep sampling temperature, and add a simple out-of-support detector. The patterns you’ll see make the topology constraint impossible to ignore—and that’s exactly the kind of insight that saves debugging weeks later.

Recommended Resources

As an Amazon Associate, I earn from qualifying purchases.