The ESP8266 -- What goes where?

The purpose of this page is to look at the addressing of various things that might be called "memory" in the ESP8266. So IO registers will be mostly ignored. 
0x00000000 .... (access causes fault)
0x3ff00000 - 0x3ff000ff IO (dport0)
0x3ffe8000 - 0x3fffbfff	dRAM (for user) 80K
0x3fffc000 - 0x3fffffff dRAM (for ETS) 16K
0x40000000 - 0x40007fff bootrom 32K (immutable)
0x40100000 - 0x40107fff	iRAM section 1 (32K)
0x40108000 - 0x4010bfff	iRAM section 2 (16)
0x4010c000 - 0x4010ffff	iRAM section 3 (16)
0x40200000 - 0x402fffff	SPI flash chip 1024K
0x60000000 - 0x600012xx IO registers
All of my ESP8266 units came with a 32M bit flash chip, (a Winbond W25Q32) giving me 4M bytes of flash memory. This can definitely vary. I am not clear how the SPI interface works and how the limitation posed by the 1M byte window is handled.

Note that the device has 96K of data RAM, only 80K of which are available for user applications. There is also 64K of instruction RAM, partitioned into 3 sections. This is rarely if ever directly accessed by the user.

Where does my application go?

My Makefile first links my code to generate a single elf file, perhaps with a name like "myapp". This has the text image and a symbols table and such. But when I want to write this into flash, my makefile does the following: 
esptool elf2image myapp
This yields two binary image files: 
myapp-0x00000.bin
myapp-0x40000.bin
Then to write this to flash, the Makefile does something like this: 
esptool --port /dev/ttyUSB1 write_flash  0x00000 myapp-0x00000.bin 0x40000 myapp-0x40000.bin
So what is going on with these two sections, and why two of them. The quick answer is that one section gets copied into on-chip iRAM to execute and the other remains in SPI flash to execute "in situ" (running maybe 10x slower, but maybe not depending on how well the caching performs).

There are several participants in making this happen. The linker script, esptool, and the bootloader, just for a start. The linker script is specified by a line like: 

-T/opt/esp-open-sdk/sdk/ld/eagle.app.v6.ld

What about ICACHE_FLASH_ATTR in function declarations?

As you may have guessed, this indicates that the function so tagged should end up in one or the other of the two sections just mentioned. So we see a function declaration like the following (but variables can be designated in this way also). 
static void ICACHE_FLASH_ATTR myfunct ( void )
{
	return 0;
}
Some header file in the SDK (which you can readily track down) defines things like this: 
#define ICACHE_FLASH_ATTR __attribute__((section(".irom0.text")))
#define ICACHE_RAM_ATTR __attribute__((section(".iram0.text")))
In a nutshell, this lets you designate that some item should go into some specified linker section other that the default ".text" or ".data" or whatever. You can just invent section names, but without specifying what to do with them, this is not of much use. That is done in the linker script. The linker script (sdk/ld/eagle.app.v6.ld) is over 200 lines in length. The "MEMORY" declaration at the start of the script defines 4 segments into which our code will go: 
MEMORY
{
  dport0_0_seg :                        org = 0x3FF00000, len = 0x10
  dram0_0_seg :                         org = 0x3FFE8000, len = 0x14000
  iram1_0_seg :                         org = 0x40100000, len = 0x8000
  irom0_0_seg :                         org = 0x40240000, len = 0x3C000
}
Code designated by ICACHE_FLASH will go into the irom0 segment and remain in flash to be executed. This code will end up in the second binary file (myapp-0x40000) and gets places into flash at an offset of 0x40000. Since this remains in flash, any variables must be read only. Note that the linker script limits the size of this segment to 240K, which seems arbitrary, especially if you have a 512K flash (4M bit).

Code not designated (or designated by ICACHE_RAM) will go into the iram0 segment. This will end up in the first binary file (myapp-0x00000) and be placed into flash at offset 0x0. This will be copied into instruction RAM at address 0x40100000 to be executed. Note that this segment is limited in size to 32K. The copy is done by the bootrom before launching the application. Variables in this segment may be read/write.