--- slideOptions: transition: slide theme: cern6 slideNumber: true title: Mail & Groupware based on Dovecot & OpenXChange tags: presentation, MALT-Mail, ICRC --- ## Mail & Groupware based on Dovecot & Open-Xchange ###### ICRC Technical Training, March 2023 #### Vincent Brillault, Giacomo Tenaglia --- ### Introduction #### History * 2018: replace Exchange 2010 on-premises with alternatives * 2019-2020: [Kopano Groupware](https://kopano.com/products/groupware/) pilot * Complete web-app feature set (advanced features) * Poor IMAP performances * 2020-2021: [Dovecot](https://dovecot.org/)/[Open-Xchange](https://www.open-xchange.com/) * Great IMAP performances and scalability * No technical show-stoppers * [Roundcube](https://roundcube.net/) as "lightweight" webmail (and for service accounts) * 2021: re-assessed client requirements (Outlook) * Now moving to Exchange Online ---- ### Introduction #### Dovecot/Open-Xchange pilot basic principles * Use Open-Source/Open-Core software * Leverage CERN IT expertise and services * Openstack, Ceph, Openshift, Puppet, ... * Focus on protocols at scale * IMAP, Cal/CardDAV * Get a paid consultancy to validate the pilot * Did not cover anti-spam/anti-malware * Supposed to be covered later on --- ### Architecture #### User-facing overview <!-- .element style="border: 0; box-shadow: none;" --> ---- ### Architecture #### Mail flow (pilot/transition) <!-- .element style="border: 0; box-shadow: none;" --> ---- ### Architecture #### Dovecot cluster <!-- .element style="border: 0; box-shadow: none;" --> ---- ### Architecture #### Open-Xchange cluster <!-- .element style="border: 0; box-shadow: none;" --> ---- ### Architecture highlights #### Hardware * Running on dedicated hardware * 6 quads distributed in 3 different racks * Each node (6x4): 64 cores, 192 GB, 2+4 local SSDs * Using Openstack to split nodes * Local/internal Ceph cluster: * Using 4 SSDs from each node * CephFS: 3x replication across quads * Block storage: 2+2 EC across quads * Total usable space: 80 TB (3 x .5TB used) ---- ### Architecture highlights #### Authentication * Dovecot & Postfix (using Dovecot) * Supporting OAuth, plain (LDAP) & Kerberos * Static password & IP filter within Dovecot cluster * Open-Xchange integrated with SSO (OAuth) * Passing the token directly to Dovecot/Postfix * Also supporting application password (Cal/CardDAV) * Roundcube not integrated with SSO, only LDAP * Out of the box image deployed on Openshift --- ### Clients * Focusing on protocols: * SMTP/IMAP/POP3 for mails * ManageSieve for filter rules (via OX web and TB add-on) * CalDAV/CardDAV for calendar/tasks/contacts * Main supported clients: * Webmails (OX & Roundcube) * Thunderbird * Mac Mail / Calendar --- ### Issues #### Outlook support * Outlook not officially supported * OK-ish IMAP support * Sub-optimal Calendar/Contacts/Tasks support ([CalDav Synchronizer](https://caldavsynchronizer.org/)) ---- ### Issues #### Mobile clients * Mobile clients (smartphones) support is complicated * Autoconfiguration not obvious * Exchange ActiveSync (EAS) [Z-push](https://z-push.org/) pilot * Discontinued: buggy/support black hole * Mail (IMAP/SMTP) OK (K9, iOS Mail, ...) * Calendar/Task/Contact more complicated * iOS OK (native CalDAV client) * Android: [DavX5](https://www.davx5.com/) (paid) ---- ### Issues #### Advanced sharing * No "advanced" support for e.g. secretary * No "send-as"/full delegation feature yet in 2021 * Was supposed to be implemented... * (Basic mail folder / calendar sharing enabled) * Limited number of users per "context" in Open-Xchange * Splitting user base in smaller groups * Need to bake some context / user provisioning logic ---- ### Issues (possibly resolved now) #### Full Text Search Indexing (Solr) * In 2020/early 2021, one supported CE FTS engine: [Apache Solr](https://solr.apache.org/) * Complex system, partially documented (only user at CERN) * Grew to be the largest part of the deployment * Since then, new plugin (not tested at CERN): [fts-flatcurve](https://doc.dovecot.org/3.0/configuration_manual/fts/flatcurve/) * Co-located with dovecot, doesn't need separate cluster * Supposed to become standard CE FTS engine ---- ### (Upcoming) Issues #### Dovecot Director & 2.4 * [Dovecot director](https://doc.dovecot.org/admin_manual/director/dovecotdirector/) * Automatically distributing users over backends * Key to avoid lock issues on shared storage * Ensuring that a user always uses a given backend * Easy to reconfigure & flush * Proxies requests to the right backend * Dovecot 2.4 [removes this feature](https://github.com/dovecot/core/commit/4a187116dc2311804be22724007d357323005358) * Proxying still possible natively * Would require external tool to manage targets * Equivalent offered only in Dovecot Pro --- ### Service management/maintenance * Most actions/changes transparent to users & easy * Dovecot backend: Drain on director * Open-Xchange: Drain on proxy (& apache) * Dovecot Director: Drain on proxy & internal DNS update * Touching some servers required slow DNS updates * Proxies: User session never ending... * Postfix: New connections long after change ---- ### Service management/maintenance * Testing all changes on test cluster * OS updates: usually quarterly & transparent * Includes Postfix, but not dovecot nor OX * Open-Xchange updates * Only did minor updates * Need to check changelogs but no big surprise * Dovecot updates * Always waiting a bit for the community response * Delayed updates more than once due to bugs --- ### Community interaction ###### How we managed to get help when needed * Postfix * Great community (ML, `irc`) and documentation * Dovecot * Quite active community (ML, `irc`) * Great documentation although mixing CE/Pro features * One core developer doing community management * Very reactive (when not on vacation) * Open-Xchange * Very scarce documentation and community support * Source code well-written and documented --- ### Questions --- ### Backup slides ---- ### Dovecot --- Basics * Dovecot: mail storage & access * Written in C, open-source (mostly) * Paid features ("Pro" version): * "Native" Full-Text-Search * Object & Archive Storage * Abuse protection (logins) * (Upcoming) Dovecot Cluster Architecture * Exposes IMAP & POP3 for email access * Multiple backends: can scale horizontally * User-defined rules for incoming mail: Sieve * Supports Kerberos & OAuth ---- ### Dovecot --- Deployment * HAProxy as frontend: load balancer between directors * Director: dynamically assign backend to user (all proto) * Backend: * Using shared CephFS to store mail boxes * Using `mdbox` to optimise file size/number * User should only access one backend, lock-protected * MySQL DB to keep track of sharing between users * Full-Text-Search: using a dedicated SOLR cluster * Should probably use `fts-flatcurve` instead now * Backups: * Mailbox replicated (dsync) to an independent system * Using ZFS snapshot for incremental backups ---- ### Postfix * Postfix: mail submission & transfer agent * Written in C, open-source * Exposes SMTP, with & without authentication * Routing incoming/outgoing emails * Currently only relaying from/to Exchange * Should become main MTA later in project * Will need more complex routing rules * (Almost) stateless, can scale horizontally * MTA & MSA functionalities can be split ---- ### Open-Xchange App Suite --- Basics * Open-Xchange: full Web-UI & Calendaring/Contacts * Written in Java, *open-source* (mostly) * Paid features (licence required, not used): * ~~Mobile Apps,~~ ActiveSync & more * Non-security patch updates, support * Licences for *open-source* parts: * Backend: GPLv2 * Frontend: CC BY-NC-SA (not OSI-approved) * Deemed OK for CERN by CERN legal service (YMMV) * Exposes CalDAV & CardDAV for calendar/tasks & contacts ---- ### Open-Xchange App Suite --- Deployment * SSO integration via OAuth * Tokens directly passed to Dovecot & Postfix * Cluster deployment: 3 VM + 1 DB * Session transparently moved between nodes * Distribution groups hidden from OX * Forcefully resolved as simple contacts * Sharing enabled (mail & calendar) * Not tested in depth * Advanced use cases (e.g. send-as) not yet implemented in 2021 ---- ### Roundcube * Roundcube: basic Web-UI * Written in PHP, open-source * *Basic*: mail + sieve rules + address book * Deployed in containers via OpenShift * Extensible by plugins (not deployed) * CalDAV integration for calendar/task * CardDAV integration for contacts