Let's repack tape faster

--- title: Let's repack tape faster description: Presentation of CERN tape repack architecture disqus: hackmd-gm slideOptions: transition: slide theme: white --- # Let's <img src="https://codimd.web.cern.ch/uploads/upload_ab4df4a384042cbc7e0ac758700e8ee5.png" style="border: none;background: none;box-shadow:none" height="300"> faster ## cheap, future-proof REPACK infrastructure [Julien Leduc](mailto:julien.leduc@cern.ch) --- ## Why REPACK? > “Dear tape operations can you make some room in the tape libraries: more data is coming soon!” > > PS: keep everything you have, we may need to read it back, thanks! > >       The experiments --- ## How to REPACK? We know how to move data from disks to tapes and back. Repack is easy: just move the data from tapes to disks and back (on high density tapes). ---- ### REPACK v2.0 architecture ```graphviz graph hierarchy { nodesep=1 // increases the separation between nodes node [color=Red, fontname=Courier, shape=box] //All nodes will this shape and colour edge [color=Blue, label="10Gb/s"] //All the lines look like this Router [shape=circle] Router--{SwitchDisk} [label="3x40GB/s", fontsize=15, style=bold] Router--{SwitchTape} [label="7x20GB/s", fontsize=15, style=bold] subgraph cluster_level1{ label="Repack disk infrastructure\n3x13 diskservers" color=dodgerblue fontcolor=dodgerblue SwitchDisk disk1 [color=black, shape=cylinder] diskXX [color=black, shape=cylinder] disk20 [color=black, shape=cylinder] disksrv01 disksrvXX--{disk1 diskXX disk20} [label=""] } subgraph cluster_level2{ label="Tape infrastructure\n7*10 tapeservers" color=crimson fontcolor=crimson SwitchTape SwitchTape--{tpsrv01 tpsrvXX} SwitchDisk--{disksrv01 disksrvXX } {rank=same; tpsrv01 tpsrvXX} // Put them on the same level tape [color=black, shape=Msquare] tpsrvXX--tape [label="360MB/s"] } } ``` ---- ### REPACK v2.0 pros and cons - Repack is **just another** *medium* disk instance - Repack disks can be **specifically optimized** - Repack is **yet another** disk instance - **Hard on network** --- ## REPACK v3.0 - Limit the cost of additional hardware: no additional servers, no additional network infrastructure - Remove network bottleneck: repack cache fast and close to tapes #### Use the SSDs in our tapeservers! ---- ### REPACK v3.0 architecture mockup ```graphviz graph hierarchy { nodesep=0.5 // increases the separation between nodes node [color=Red, fontname=Courier, shape=box] //All nodes will this shape and colour edge [color=Blue, label="10Gb/s"] //All the lines look like this Switch--{tpsrv01 tpsrvXX tpsrv70} {rank=same; tpsrv01 tpsrvXX tpsrv70} tape [color=black, shape=Msquare] SSDs [color=black, shape=cylinder] tpsrvXX--SSDs [label="", style=bold] tpsrvXX--tape [label="360MB/s"] } ``` ---- ### REPACK v3.0 architecture in practice **Will it work?** - will this be fast enough? - SSD life expectancy? --- ### Some microbenchmarks: *SPEED* On four independent SSDs: # streams | R speed(MB/s) | W speed (MB/s) --- | --- | --- 1 | 548 | 500 2 | 1040 | 1000 3 | **1580** | **1280** 4 | **1580** | **1280** **We are hitting a bottleneck inside the machine** ---- ### Some microbenchmarks: *SPEED* <img src="https://codimd.web.cern.ch/uploads/upload_72fc95ac7027ad2344cd850ff1c4a407.png" style="border: none;background: none;box-shadow:none" height=600> ---- ### Some microbenchmarks: *SPEED* Unbalanced systems: single CPU systems design. ```graphviz graph hierarchy { nodesep=1 // increases the separation between nodes node [color=Red, fontname=Courier, shape=box] edge [color=Blue, label=""] CPUBUS subgraph cluster_level1{ CPU1 [shape=circle] label="NUMA node 1" color=dodgerblue fontcolor=dodgerblue Memory1 [label="{<f0>Memory|<f1> 32GB}" shape=Mrecord color=black] SATA1 [label="{<f0>SATA|<f1> 4 SSDS}" shape=Mrecord color=black] Ethernet1 [label="{<f0>Ethernet|<f1> 2 NIC@1Gb/s\n 2 NIC@10Gb/s}" shape=Mrecord color=black] HBA1 [label="{<f0>FC HBA|<f1> 1 drive}" shape=Mrecord color=black] CPU1--{Memory1 SATA1 Ethernet1 HBA1} [label=""] } subgraph cluster_level2{ CPU2 [shape=circle] label="NUMA node 2" color=tomato fontcolor=tomato Memory2 [label="{<f0>Memory|<f1> 32GB}" shape=Mrecord color=black] SATA2 [label="{<f0>SATA|<f1> -- }" shape=Mrecord color=black] CPU2--{Memory2 SATA2} [label=""] } CPUBUS--{CPU1 CPU2} [style=bold] } ``` ---- ### Some microbenchmarks: *SPEED* SATA is simplex and in practice: $$ \sum_{i \in SSDs}(ReadSpeed_{i} + WriteSpeed_{i}) = 1.6\ GB/s $$ Speed is not optimal because of unbalanced server SATA topology but it is good enough. <img src="https://i.imgur.com/6DASMR5.gif" style="border: none;background: none;box-shadow:none" height="300"> --- ### Will it age well :wine_glass: or not :fish: ? SSDs are composed of cells that are aging with write cycles. Model | *Samsung MZ7LM960HCHP-00003* --- | --- MTBF | 2 000 000 Hours *(228 years)* TBW | 1 400 TB *(WAF=1)* Infrastructure data aging expectancy: **392 PBW** **Good for a few repacks...** ---  ### REPACK v3.0 architecture ```graphviz graph hierarchy { nodesep=0.5 // increases the separation between nodes node [color=Red, fontname=Courier, shape=box] //All nodes will this shape and colour edge [color=Blue, label="10Gb/s"] //All the lines look like this Router [shape=circle] Router--{Switch1 Switch2} [label="60Gb/s", fontsize=15, style=bold] Switch1--{tpsrv01 tpsrvXX tpsrv40} Switch2--{tpsrv41 tpsrvYY tpsrv70} {rank=same; tpsrv01 tpsrvXX tpsrv40 tpsrv41} // Put them on the same level tape [color=black, shape=Msquare] SSDs [color=black, shape=cylinder] tpsrvXX--SSDs [label="1.6GB/s"] tpsrvXX--tape [label="360MB/s"] } ``` ----  ### REPACK v3.1 architecture ```graphviz graph hierarchy { nodesep=0.5 // increases the separation between nodes node [color=Red, fontname=Courier, shape=box] //All nodes will this shape and colour edge [color=Blue, label="10Gb/s"] //All the lines look like this Router [shape=circle] Router--{Switch1 Switch2} [label="60Gb/s", fontsize=15, style=bold] Switch1--{tpsrv01 tpsrvXX tpsrv35} Switch2--{tpsrv36 tpsrvYY tpsrv70} Switch1--Switch2 [color=crimson, style=bold, label="120Gb/s"] {rank=same; Switch1 Switch2} {rank=same; tpsrv01 tpsrvXX tpsrv35 tpsrv36} // Put them on the same level tape [color=black, shape=Msquare] SSDs [color=black, shape=cylinder] tpsrvXX--SSDs [label="1.6GB/s"] tpsrvXX--tape [label="360MB/s"] } ``` We have 2 stackable [Brocade 7750-48C switches](http://www.brocade.com/content/html/en/configuration-guide/fastiron-08030b-switchstackingguide/GUID-4117E358-1F97-4E3F-85A7-F3082C0CB904.html) --- ### REPACK v2.0 monitoring Lemon and network service traffic: <img src="https://codimd.web.cern.ch/uploads/upload_7e20e767b51972d2a6cd726f99ff73ec.png" style="border: none;background: none;box-shadow:none" height="300"> 5 minute resolution, requires fast `switch:port` :left_right_arrow: `tape drive` mental translation... ---- <style> @import url('https://fonts.googleapis.com/css?family=Metal+Mania'); </style> ### REPACK v3.x monitoring This is System Tap Realtime kernel device drivers metrics per second. Collects **all bandwidth metrics**: - tape drive read/write rate, IO time - SSDs read/write rates - network in/out rates per process per protocol ---- ### REPACK v3.x monitoring Relies on SystemTap instrumentation: - kind of bad situation in SLC6 (wrong headers...) - several broken metrics in CC7, I reported 1 bug to RedHat (fixed in 7.4) - I need to report another minor bug... **Production grade finally!!** ---- ### REPACK v3.x monitoring <a href="https://meter-cta.web.cern.ch/dashboard/db/perf" target="_blank"><img src="https://codimd.web.cern.ch/uploads/upload_ddc52d33a9a67e0fdfa95d2d189c0699.png" style="border: none;background: none;box-shadow:none" height=600></a> --- ## ToDos - [x] Think about the repack architecture - [x] Make sure it works - [x] Draw nice graphs - [ ] Repack 100PB <img src="https://media.giphy.com/media/Pr3ll8LR4ZCgg/giphy.gif" style="border: none;background: none;box-shadow:none" height=100%>