Project Topics
Get Complete Project Material File(s) Now! »
Radiative transfer post-processing
In RamsesRT, new variables are added to U : The three ionization fractions, xHII , xHeII and xHeIII , which are stored as multiples of the gas density, and 4M photon field variables, i.e. 4 variables of photon density and flux vector for each photon package. Although there is no difference in how the variables are stored, we conceptually split U into two parts:
• U h consists of the usual hydrodynamic variables (ρ, u, E, Z), plus the three ionization fractions (xHII , xHeII , xHeIII ).
• U rt consists of the 4M photon variables (Ni , Fi ).
The cell state vector is then U = U h , U rt .
In RT post-processing, hydrodynamics and particle propagation are turned off, so RT is in effect performed inside an otherwise frozen simulation, and can be done by re-starting from an existing Ramses output. Here amr_step is not called at all, but instead a similar recursive routine called rt_step is called from the coarse level.
Listing 3.2 shows the pseudocode for this routine: The lines of execution and their ordering is basically the same as in amr_step. In line 2 the time-step is calculated via one simple Courant condition. There is no worry here about the time-step changing due to finer levels, since the time-step length is purely a function of the Courant factor, cell width and light speed, all of which are constants. In line 3 a buffer state U rt is created for the RT variables. There is no need to buffer the hydro variables, as these are not transported between cells in the RT step. The recursion to finer level photon transport and thermochemistry is done in lines 4-6. Then the current level photon transport (§3.2.1) in line 7, which also partially updates inter-level boundary cells at level − 1. Then the photon injection step (§3.2.2) in line 8, where photons are added to the grid (U rt) via e.g. stellar emission. Note that this also involves some U rt−1 updates due to stars that move across level boundaries in the second RT sub-step at level . The photon transport and injection is made final in
line 9 by writing the state variables. Finally in line 10, non-equilibrium thermochemistry is performed on all level cells. It is important to remember that here not only the RT variables are being updated, but also the hydro variables of temperature (or internal energy) and ionization states. This is the vital point where RT couples with the hydrodynamics and this update is actually what makes it very tricky to implement radiation hydrodynamics in an AMR framework, as will soon come clear.
When smooth RT is used (see §3.2.5), line 9 in Listing 3.2 is skipped, i.e. the new state is not made final before the thermochemistry step. This allows the thermochemistry to use the before-transport-and-injection photon densities and fluxes and also allows it to determine their change rates, (U rt − U rt )/Δt , which are the ‘smooth’ terms that are added to the thermochemistry calculation. The effect of photon transport and injection thus only comes through after the thermochemistry step.
Let us briefly again consider the ordering of commands: Here the transport step can actually be called before the recursion (the two calls to rt_step in lines 5 and 6), since all the time-step lengths are known in advance. This means the transport could in principle be done from the coarsest level up instead of from the finest level down as with the hydrodynamics. However, this would only further complicate the buffer states, as the coarser levels would still have to wait for the finer levels to do those inter-level updates. For simplicity and consistency with the amr_step we keep the finest level down stepping. Once that is set, we still need to create the buffer state before the recursion and do the transport after. As with the hydrodynamics the thermochemistry must be done after the transport. The photon injection step could in principle be anywhere after the recursion, as the U rt update here doesn’t depend on U rt.
Although we do not show it in the pseudocode, adaptive refinement can be performed on the grid via gradient criteria on photon densities and ionization fractions. We demonstrate this in some of the tests described in Chapter 4. The details, e.g. of refinement, the AMR grid bookkeeping and MPI communication is made very easy by the fact that we have simply added new variables to the pre-existing structure of Ramses, so while we have to change some indexing and be careful to make calls in the right places to keep the bookkeeping in order (which are basically the same places as in amr_step), the effort is minimal considering the fact that we can perform radiative transfer in an AMR structure on parallel computers. RT postprocessing works well in RamsesRT, we have thoroughly tested it with the benchmark tests of Iliev et al. (2006a) (see Chapter 4) and used it for some preliminary applications, like the mapping of gas ionization state and Lyα emissivity in simulations of isolated galaxies. We note that RamsesRT can perform in two as well as three dimensions – having the possibility of two dimensional simulations is very convenient for making simple tests and debugging.
Radiation hydrodynamics
Then for on-the-fly radiative transfer in RamsesRT. Here we need to do both RT and hydro-dynamics (and DM propagation, gravity etc.) at the same time. This is made tricky by the combination of a timescale difference and the nested time-stepping in the AMR structure. We describe first the obvious and preferred way of doing things, which alas proves impossible so far to implement in a self-consistent manner. Then we present two alternative coupling setups that do work, but are less optimal speed-wise.
First a mention about the advection of the ionization fractions: These variables are stored as multiples of the gas densities, in the form of passive scalars that are advected with the gas. The gas metallicity is already such a passive scalar that is advected in Ramses, and one can simply add further variables (always multiples of density) that work in the same way. This is also the reason that these ionization fractions belong in U h rather than U rt.
RT subcycling: The preferred but illegal coupling method
In view of the large timescale difference between RT and hydrodynamics, which is due to the large speed of light compared to the maximum wave-speeds in the gas, it appears natural and ideal to subcycle the radiative transfer within each hydrodynamical time-step at the finest hydro-stepping scale. For each max hydrodynamical step, an RT step should be performed over the whole grid, over all levels. The problem is that during the coarse hydro time-step, in between the finer level hydro substeps, there are cells with undefined hydrodynamical states, which makes it non-trivial for RT-thermochemistry to update the temperature and the ionization states.
We can take a concrete look by attempting to insert the RT step inside the AMR step so that it is subcycled between the fine-scale hydrodynamics. There is really only one place where calling the RT step fits, which is after the recursion but before the hydro-solver. It has to be after the recursion because until then the final time-step length for the RT subcycling is not determined. It has to be before the hydro-solver for consistency: The RT step includes thermochemistry, so if it is called after the hydro-solver, thermochemistry happens after hydro-transport at max but before it at coarser levels (where hydro-transport has yet to happen). Thus we arrive at the pseudocode setup shown in Listing 3.3.
It seems simple enough, but here is the problem: At the point that the RT step is called, every cell in the grid exists with two hydrodynamic states; the ‘source’ state U h and the ‘destination’ state U h. Which one of those should be updated via RT thermochemistry? Choosing one but not the other leads to an obvious and severe inconsistency between the source and destination states. If thermochemistry does the update U h → U h, then a gas element which is transported from one cell to it’s neighbour during the following hydro transport is not thermochemically evolved over the time-step, because it originates from U h. If instead the update is done on U h → U h, a gas element which stays still in a cell over the following hydro transport step is not thermochemically evolved over the time-step. One might then just update both states via thermochemistry, i.e. apply it on each cell twice. This does not really make sense for these inter-level intercell boundary cells that have U h U h, as U h doesn’t represent a true state but is rather an intermediate and temporary quantity that exists between well-defined times. Also, it would be really non-trivial to implement: Applying thermochemistry on each of the states also implies transporting the photons through two different states in each cell, which creates alternate time-lines for the radiative transfer! We have thus abandoned this pre-supposed ideal way of RT-hydro coupling and gone for two alternatives which sidestep the problem of undefined gas states. In one method we prohibit hydro-subcycling over the AMR levels, and in the other we prevent the RT subcycling within the fine-scale hydro step by limiting the hydrodynamical time-step down to the RT one.
Hydro-single-stepping with RT subcycling
The only way of retaining the subcycling of the RT steps within the hydro step, while avoiding the problem of ill-defined hydro-states, is to do the hydrodynamical update in a single time-step over the whole AMR grid hierarchy (which is an option in Ramses). An amr_step is performed over the whole grid, on a hydro time-step length which is set by the finest level, and then RT is subcycled over the grid to match the hydro time-step. Here the hydro states are final and well-defined everywhere when the RT is called. We show the pseudocode for amr_step and rt_step in Listings 3.4 and 3.5, and a schematic example of a coarse time-step is shown in Fig. 3.7. The basics of the coarse time-step consist of:
Radiation-hydrodynamics with RamsesRT
Not allowing subcycling of the hydrodynamics is a very severe limitation though, and in effect removes one of the two main advantages of AMR: It is not just about adaptive resolution in space but also in time, and here we are throwing away the time-refinement. In practice this has a devastating effect on the run-time of simulations when the AMR hierarchy spans many levels. If there is only a handful of levels however, this is a good option. In fact, we have run most of the RT benchmark tests with this method, as we only do one level refinement in these tests. For large simulations with lots of refinement levels, the method we describe next is a better option.
RHD with an RT time-step
The other alternative for RamsesRT radiation-hydrodynamics, is to abandon the RT-within-hydro subcycling. This can be achieved by either expanding the RT time-step length to match the hydrodynamical time-step, or vice-versa. We’re not allowed to relax the Courant time-step condition on the RT unless we switch to an implicit RT transport – and we haven’t done that. So the option we take here is to reduce the hydro time-step down to the RT one. Of course if we stick to the physical speed of light, this means reducing the hydro time-step by ∼ 3 orders of magnitude, which is unrealistic in simulations, and likely creates a lot of hydrodynamical diffusion to add. However, if we match this with a speed of light which is reduced by, say, a factor of 100, this only means reducing the hydro time-step by a factor of ten or so, which is starting to become borderline manageable. This is in fact the most feasible way we have found to run large scale RHD simulations – the time increase when using a light speed fraction of 1/100 and three photon packages typically is about a factor of ten, and this can basically be reduced to a factor of two if the light speed fraction is low enough that the hydro time-step doesn’t have to be modified at all. A schematic of this setup is shown in Fig. 3.8 and the pseudocode for the amr_step, which now has been ‘merged’ with the rt_step to include the RT transport, injection and thermochemistry steps, is shown in Listing 3.6.
Table of contents :
1 Introduction
1.1 What do we know about the Universe?
1.2 The frontier – reionization and end of the dark ages
1.3 An upcoming peek into the dark ages
1.4 Cosmological simulations: A complementary tool for the exploration of the Universe
1.5 This PhD thesis: A tool for understanding the early evolution of galaxies
2 On the theory of ionizing radiative transfer in cosmological simulations
2.1 Radiation-hydrodynamics in a cosmological context
2.1.1 Cosmological hydrodynamics
2.1.2 The equation of radiative transfer
2.1.3 The RHD equations
2.1.4 Main challenges of numerical RT and RHD
2.2 Ray-tracing schemes
2.3 Moment-based radiative transfer
2.3.1 Moments of the RT equation
2.3.2 Closing the moment equations
2.4 From Aton to RamsesRT
2.5 One more RHD implementation?
3 Radiation-hydrodynamics with RamsesRT
3.1 Multigroup H+He ionizing radiative transfer
3.2 Solving the RT equations on a grid
3.2.1 (a) Photon transport step
3.2.2 (b) Photon injection step
3.2.3 (c) Thermochemistry step
3.2.4 The RT time-step and the reduced speed of light
3.2.5 Smooth RT
3.3 Cosmological settings
3.4 Putting the RT into RamsesRT
3.4.1 The AMR structure of Ramses
3.4.2 Radiative transfer post-processing
3.4.3 Radiation hydrodynamics
3.5 Thermochemistry
3.5.1 Ramses equilibrium thermochemistry – what’s already there
3.5.2 RamsesRT non-equilibrium thermochemistry
3.5.3 Thermochemistry performance and prospectives
3.6 Stellar UV emission and photon package properties
4 Code tests
4.1 Introduction
4.2 Thermochemistry tests
4.2.1 Ionization convergence at constant temperature and zero ionizing photon flux
4.2.2 Ionization convergence at constant temperature and nonzero ionizing photon flux
4.2.3 Temperature convergence with zero ionizing photon flux
4.2.4 Temperature convergence with nonzero ionizing photon flux
4.2.5 Thermochemistry tests conclusions
4.3 The benchmark RT tests
4.3.1 Il06 test 0: The basic thermochemistry physics
4.3.2 Il06 test 1: Pure hydrogen isothermal Hii region expansion
4.3.3 Il06 test 2: Hii region expansion and the temperature state
4.3.4 Il06 test 3: I-front trapping in a dense clump and the formation of a shadow
4.3.5 Il06 test 4: Multiple sources in a cosmological density field
4.3.6 Il09 test 5: Classical Hii region expansion
4.3.7 Il09 test 6: Hii region expansion in a r−2 density profile
4.3.8 Il09 test 7: Photo-evaporation of a dense clump
4.3.9 Benchmark test conclusions
5 Extended Lyα emission from cold accretion streas
5.1 The paper
5.2 Notes on extended Lyα emission
5.2.1 Resolution convergence
5.2.2 Light speed convergence
5.2.3 UV emission density threshold convergence
5.2.4 Operator splitting and underestimated Lyα emissivities
5.2.5 Does a simple self-shielding approximation suffice?
5.2.6 Are disrupted streams a numerical effect of over-resolved gravity?
5.2.7 Smooth vs non-smooth RT
6 Conclusions and outlook
