The tilemap layers are drawn one by one z-sorted with other components such as sprites. The tilemaps are drawn using the tilemap.material.
I think your best bet is to split your tilemap layers into multiple tilemaps (top.tilemap, middle.tilemap and bottom.tilemap), and draw the middle.tilemap (the mask) to a render target. Next you use this render target as an input when you draw top.tilemap. You need a custom render script and custom versions of your tilemap materials.