EDM4hep Live Notes ================== Date: January 12, 2021 Indico: https://indico.cern.ch/event/983626/ This is a document for taking notes during EDM4hep meetings. Connected: Joseph, Andre, Tao, Weidong, Teng, Jiaheng, Frank, Birgit, Placido, Sang Yun, Thomas, Xingtao, Valentin, Clement, Brieuc, Gerri, Hyeonja Apologies: ## Introduction and General Points * Happy new Year! ### Key4hep presentation at Epiphany * https://indico.cern.ch/event/934666/contributions/4154229/ * Add to https://key4hep.github.io/key4hep-doc/talks-and-presentations/README.html * Also add a link to the FCC presentation https://indico.cern.ch/event/934666/contributions/4154228/ ### Virtual Chep * papers(!) due February 28, 2021 * Submissions should consist of a paper of 6-10 pages, following the instructions for authors. Paper submission is open until 28 February 2021. * https://indico.cern.ch/event/948465/page/21562-instructions-to-submitters * Contribution: * Key4hep overview * Few paragraphs per contribution? * Delphes parts? * podio+edm4hep! * TODO: share the overleaf from the last CHEP proceedings * TODO: create repository for published documents? * Why not CERN gitlab/DESY stash? * Need cern or desy computing accounts to access non-public repositories * (At least at the moment for CERN gitlab, might change) * Private github repository free now? * Unlimited public/private repositories * See: https://github.com/pricing ## Progress and Discussion ## Podio ### Benchmarking [Thomas] Added some scripts and tools to have almost automatic benchmarking and some results using k4SimDelphes to "generate" data for I/O. Could be made fully automatic with a bit more work and if we want it to be fully automatic. * https://github.com/tmadlener/podio_benchmarking * Example results: https://github.com/tmadlener/podio_benchmarking/tree/master/results/k4SimDelphes/ee_Z_bbbar * Teng: Add to github actions, upload plots as artifacts? * Thomas: takes about 1 hour to run (depending on number of events) * Teng: Can set schedule time in the yaml file. Very useful for validation * Andre: The root peak structure understood? * Thomas: Not yet fully understood/investigated. "working hypothesis" is that root has to resize some internal buffers if an Event/Collection has a certain size, resp. when it hits an Event with a certain size for the first time. ### Plans for multi-threading testing of EDM4hep for CEPC * Using Gaudi-hive functionality to speed up garfield simulation of the driftchamber * Any documentation sugestions for how this can be done? * Valentin: sounds very useful, not much experience in FCC * Andre: What is the status of multi-threading for podio? * Thomas: Reading is safe by design, writing has to be synchronised (e.g. by Gaudi) * Frank: Root or also SIO backend? * Thomas: Difference between standalone readers and what happens in the Gaudi world * Valentin: Inside Gaudi it should work ### Issues #### more fixes to unsigned and shadowed variables * https://github.com/AIDASoft/podio/pull/158 #### c++ concepts * BH: add compile time checks for class behaviours: e.g., movable #### CollectionIterator does not fully satisfy iterator concept * https://github.com/AIDASoft/podio/issues/150 * For const collections, would be useful, e.g. using std::find on the collection #### Idea: True reference collections * https://github.com/AIDASoft/podio/issues/146 #### issue w/ ROOT and (vectors of) non-copyable collections * happens in ROOT 6.22 * PM: there is a patch available in LCG repository * ROOT team is working on a general solution #### What are the different branches in the root file? * Related to use in RDataFrame * Encode more information in the _relation_ branch names? #### Multi-Threading See minutes of https://indico.cern.ch/event/969468/ --> Open issue(s) for further discussion * Thomas and Benedikt discussing and started to work, nothing to report yet #### "event class" in podio * Currently being perceived #### Writing second file with another tree * https://github.com/key4hep/K4FWCore/issues/10 * this problem only happens in the GAUDI framework * if user tries to write an additional ROOT file * to be addressed * Solved ### PRs * https://github.com/AIDASoft/podio/pulls #### Disable collection operator= * https://github.com/AIDASoft/podio/pull/154 * Review: Benedikt ### Meta Data #### Usage of "metadata" for user defined data * need to check if current implementation addresses all use cases * need test use-cases ### Issues following MetaData Developments * cannot write out event data previously read from file * Issue: https://github.com/AIDASoft/podio/issues/103 * Test: https://github.com/AIDASoft/podio/pull/102 * TM: have fix, but break framework core handling of collections * there seems to be a fix in the Gaudi ROOTWriter in FCC-EDM !? * historical reason, fixes and developments should be back ported to PODIO * ... ### EventStore ### Schema Evolution - Version for object descriptions, etc. - Open issue: https://github.com/AIDASoft/podio/issues/86 ### Features * Subset collections? ## LCIOConverters * k4LCIOReader: * https://github.com/key4hep/k4LCIOReader * Moved k4LCIOReader code to key4hep project ## EDM4hep https://github.com/key4hep/EDM4hep/pulls ### EDM4hep tools * Thomas, Clement: * Where to put tools using EDM4hep? In EDM4hep or separate repository? * VV: Would be nice to keep EDM4hep stable, so separate repository. * FG: Concur. Many releases for LCIO just to update tools. ### Issues #### k4FWCore: cannot read collections back * https://github.com/key4hep/k4FWCore/issues/27 ### DD4hep output plugin --> Moved to DD4hep ### Repository is getting too big * github pages branch is getting large * Can they be created on the fly? * Use a different host for the website ### Tracker Hit * Q: Tracker hit input to tracking algorithms? * A: In LCIO different traker hits: Planar and Cylindrical using inheritance. Still needs to be adressed how to do this in EDM4hep. * Q: Is inheritance needed? * Q: What to use for Driftchambers? * Open issue: ### Need review of EDM4hepDelphes output * Output is not stored if no tracks or towers created (?) * Allow also other types to be part of the list * Allow construction in a more general way (VV) * Need to assume there is a complete list of reconstructed particles? * Relations: * FCCSW: Issue with more than one RecoParticle pointing to one MCParticle * In LCIO to Delphes, conversion happens later than in edm4hep, maybe delphes does de-duplication? * TODO: open issue in edm4hep with reproducer (VV) --> Moving to separate repository #### Reworking output handling - TM: better address edge cases - TM + CH: FCC cases, not sure if delphes or output - https://github.com/key4hep/k4SimDelphes/pull/5 - TM: How and what to test? - ### PRs https://github.com/key4hep/EDM4hep/pulls/ ### Release 1.0 * Need: * ~~Plugin~~ * ~~Eventheader~~ * ~~Meta Data (Event / Run Parameters)~~ ## AOB ### Generator infrastructure in FCCSW moved to EDM4hep * VV: show at next key4hep ### Framework integration of k4SimDelphes * On hold until delphes output reworked * Started with updating the Gaudi documentation * New cmake in Gaudi v35 ### Dual Read-out calorimeter for FCC * Special data structure used for that simulation * Tried to use edm4hep, some issues * Present in a future meeting ### Conditions handling in Belle2 * Benedikt, or Martin Ritter ### Feedback from FCC tutorial for snowmass * Common question: What are the different branches in the root file? * ``uproot`` was also used to access the root file * Basically re-implemented event store * Associations * Try to get this into the, e.g., podio repository ### Next meeting: * January 26, 2020