# JACoW Workflow using Indico This document proposes an implementation of the JACoW paper editing workflow using Indico instead of SPMS. Rather than trying to establish a strict 1-to-1 mapping between Indico and SPMS functionality, the remaining sections will focus on establishing a relationship between the JACoW process and Indico as of version 2.3. We will refer to SPMS functionality whenever possible, however, especially in order to establish a comparison in aspects in which Indico differs from it. There are also direct references to the requirements specification document which was written in 2016. ## User Profiles *(pg. 4 of "Porting SPMS features to Indico" document)* User profiles are fully managed through the JACoW repository. Modifications are synchronized with Indico every time a user logs in. One thing to keep in mind is that Indico's handling of user information is based on **snapshots** of the user's data. Whenever a user is added to an event, Indico takes a snapshot of that persons's information (name, affiliation, etc...) and saves it. Further changes in the user's JACoW profile will not be reflected automatically in that event (only in those created after the change). It's also important to point out that, upon **cloning an event**, those snapshots are also cloned. Hence, user profile information will not be updated automatically. ## E-mail management *(pg. 118 of "Porting SPMS features to Indico" document - "E-mail management")* There isn't a "central" e-mail management page as such in Indico, contrary to SPMS. A similar degree of functionality is achieved through the different management pages which provide listings of Indico users. Those are, among others: * **Registrants** Page - this list provides a powerful filtering mechanism which allows users to be picked using predefined criteria and consequently bulk e-mailed;  * **Contributions** Page - the "authors list" option allows for quick retrieval of users playing specific roles (e.g. "speaker", "author") and to e-mail them directly, using a standard Indico e-mail template dialog;  * **Roles Setup** interface - which allows users playing a specific role to be contacted directly; * **Participant roles** overview - where speakers, authors and users playing other event-defined roles are merged together in a uniform view. Users can be batch-emailed and even sent invitations to register with Indico if they haven't yet; * **Editing Team** ("Editing" module) - allows editors to be contacted;  These are just the major interfaces which mimic what is currently SPMS functionality. ### E-mail logging Every e-mail message sent from within Indico is kept in the Event's general log. More generally, all significant modifications made in an event either by an event manager or a user are preserved in it. The full text of the message can be consulted, as well as the identities of the people it was sent to.  ### Coping with a large number of e-mails One of the many improvements done in the last 3 years was the sending of e-mails through an asynchronous task mechanism (Indico 2.0rc1). This ensures that large numbers of participants can be e-mailed without noticeable interface delays. ### Missing Features | Title | Description | Status | | ---------------------------------------:| ---------------------------------------------------------------------------------------------------------------------- | --------------- | | **Option to CC all event managers** | An option which would forward all e-mails sent by Indico to the event managers. The log might be enough in most cases. | To be discussed | | **Indicate number of e-mail receivers** | We should display how many users will be targeted before any mail-sending operation is confirmed. | To be scheduled - high priority | --- ## Role Management *(pgs. 13 ("Functional Roles") and 117 ("Missing Roles") of "Porting SPMS features to Indico" document)* One of the aspects which were identified as central to an approximation of the Indico and SPMS workflows was the implementation of a role system in Indico. This is reflected in the new "Roles Setup" option in the Management Interface.  These groups can granted privileges across the event, including editing permissions.  Indico also allows these "groups" to be e-mailed based on user-defined templates.  Moreover, the new **Participant Roles** overview allows for a quick glance over the people involved in the event, including not only the aforementioned roles but speakers as well. It also shows which of those have already registered for the event.  ## Participant Registration *(pgs. 95-116 and 129 of "Porting SPMS features to Indico" document - "Registration")* Indico provides a full-fledged **participant registration** module which allows for creation of flexible registrations tailored to the preferences of the event organizers.  The fields present in each registration form can be chosen and configured.  Several different types of fields representing different data types can be used. Some of them represent simple values (e.g. number/text inputs) while others more complex data (e.g. multi-choice fields). * Almost all fields can be made **billable** (paid for) and assigned a "price tag". This price will be used to compute the final conference fee; * There is also the possibility to select a **maximum number** of places for choice, checkbox, yes/no, accommodation and multiple choice fields.  There are several Indico plugins which allow for Electronic Payment of conference fees: * PayPal (official) * SIXPay SaferPay (community) * Stripe (community)  Payments can also be accepted manually (e.g. in the event of bank transfers or waiving of fees).  ### Managing Participants The "List of Registrations" is the place to go for everything participant-related. From there, participant data can be modified, they can be approved (in case moderation is enabled), marked as checked-in, e-mailed and exported to different data formats.  This list also provide a "customization" option which allows for participants to be filtered according to any of the form fields. ### Registration Withdrawal One of the new features in Indico 2.3 is the possibility for participants to **withdraw** their registration.  ### Missing features | Title | Description | Status | | --------------------------------------------:| -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------:| ------------------------------- | | **Conference Receipts** | Customizable receipts and printing to PDF/mailing | :watch: Planned for **3.1** | | **Read-only permissions for Reg. Form** | Roles with read-only permissions may be needed | To be scheduled - low priority | | **Additional Payment methods** | Cheque - still needed? | to be discussed | | **Webhook-based payment API** | To use with external systems. As specified in [#3051](https://github.com/indico/indico/issues/3051) | to be scheduled - high priority | | **Unified List of Registrants (management)** | A list of registrants which is common to all forms and where common actions can be performed | to be scheduled - high priority | | **Waiving Payments and other discounts** | A way to waive payments (different from setting as paid) and to apply discounts - see [#3030](https://github.com/indico/indico/issues/3030) and [#1846](https://github.com/indico/indico/issues/1846) | to be scheduled - high priority | --- ## Programme (Track) Management *(pgs. 24-35 ("Scientific Program Administration") and 117 ("General Features > Classifications") of "Porting SPMS features to Indico" document)* Indico had, since the beginning, a "flat" structure of **tracks** which contrasts with SMPS's concept of "classification" and "sub-classification". From version 2.2 on, Indico introduces the concept of "Track groups", which provide an additional level of nesting aimed at targeting that difference.  ### Track import/export  Tracks (and groups) can be "copied" from event to event using the **Import** feature.  This greatly simplifies setting up new events. Configuration can also be copied for **registration forms**, **event roles** and **call for abstracts**, among others. ### Missing Features --- ## Programme Codes *(pgs. 36-38 ("Program Code Assignment") and 121 ("Session Management") of "Porting SPMS features to Indico" document)* Programme Codes are central to the JACoW workflow and were one of the main points lacking in previous versions of Indico. In version 2.3, a new program code system was put in place.  Codes can be assigned based on several "placeholders" which are specific to each item. For instance, in a contribution, elements such as the title, the day of the week, or even the code of the enclosing track can be used to generate a self-contained ID.  Assignments can be fine-tuned if needed. The assignment operation **has to be repeated** whenever there are new contributions (i.e. it's not automatic).  ### Missing Features | Title | Description | Status | | ------------------------------- | ---------------------------------------------------------------------------------------------------- | --------------- | | **Automatic assignment** | Fully-automatic assignment may be tricky, but we could probably make the operation easier to execute | To be discussed | | **Option to keep codes hidden** | An option to keep programme codes hidden until the programme is "finalized" | To be discussed | --- ## Abstract Management *(pgs. 5-8 ("Home Page as Abstract Owner") and 118 of "Porting SPMS features to Indico" document - "Abstract Management")* Abstract management in Indico is done through the "Call for Abstracts" module, which implements the **submission**, **reviewing** and **displaying** of conference abstracts. Indico's Abstract reviewing features loosely map to SPMS's **Initial Q/A** process. They were substantially improved in order to accomodate the JACoW workflow. ### Contribution types *(pgs. 30 of "Porting SPMS features to Indico" document - "Scientific Program Administration > Presentation Types (Contributions)")*  Contribution types are normally used to distinguished keynote/plenary talks from parallel ones. Indico allows for limitless numbers of those. Since Indico 2.1, they can be set as **Private**, meaning that users cannot select them while **submitting their abstracts** and it will be up to the event managers to do that. This was added in order to mimic the **Editor only** flag present in SPMS. ### Abstract Fields *(pgs. 24-25 of "Porting SPMS features to Indico" document - "Scientific Program Administration > Abstract Attributes")* Custom fields can be added to the abstract submission form.   The following settings can be set on an Abstract Field: * **Title** * **Description** * **Required** - Whether the field is mandatory * **Active** - Whether the field is visible * **Visibility** - Who will be able to see the field: Everyone, Managers/Submitters, Only Managers * **User editable** - Whether the submitter should be able to change the field or that should be reserved to managers (besides others which depend on whether it's a free text field or a choice field) The **Visibility** and **User editable** options were added in Indico 2.1, in order to allow for the JACoW workflow. ### Abstract States Abstracts can be in a number of states: * **Invited** (see below) - "incomplete" abstracts which have to be explicitly created by an invited person; * **Awaiting Review** - the initial state of an abstract once it is submitted by a user; * **Accepted** - final state, a **contribution** is **automatically generated** from the abstract; * **Rejected** - final state, the abstract is **not** copied into a contribution; * **Withdrawn** - final state, the abstract has been withdrawn by the submitter; * **Merged** - final state, the abstract has been merged into another abstract which is specified at judgment time; * **Duplicate** - final state, the abstract is a duplicate of an already existing one. No merging happens.  Contrarily to SPMS, where abstracts and contributions are the same, Indico establishes a difference between the two things. One of the advantages of this approach is the preservation of the original submissions in their original state.  Once an abstract is accepted, the following happens: * A **new contribution** is created, with the same title, content and authors/speakers; * The contribution is assigned the track which was decided at **judgment time**; * If the track is mapped to a **session** (see below), the new contribution is automatically placed in that session. ### Tracks and Sessions Indico and SPMS both have separate concepts of "Track" (or "classification") and "Session". | Indico | SPMS | | ----------- |:------------------ | | Track group | Classification | | Track | Sub-classification | | Session | Session | The relationship between Tracks and Sessions is very loose in Indico, and it is so on purpose. In general, the following rule of thumb can be applied: * **Tracks** are used to classify abstracts and are mostly useful **during the reviewing process**; * **Sessions** are used to group contributions within a timetable. In Indico's case, they can be further split into **session blocks** (a **block** is a group of contiguous contributions) e.g. the "morning" and the "afternoon" blocks. Since Indico 2.1 it is possible to specify a **default session** for a given **track**. All contributions accepted for that track will thus be placed by default in that given session.  ### Session Types **Session Types** are a concept which was introduced in Indico 2.1, aimed at bridging the gap with the JACoW workflow. Previously, Indico only allowed two types of session: "Oral" and "Poster". With this addition, options such as "Plenary" or "Parallel" can be added instead. Every session type can be flagged as a "poster session", in which case the assigned session will behave as such (no formal scheduling of contributions).  ### Invited Abstracts One of the main additions to Indico's abstract feature set is the "invited abstract" workflow, which was non-existant before version 2.3. The event organizer needs to provide only a title and a user (which may or not have an Indico account), after which the user will receive an e-mail notifying them that they can proceed to complete their abstract.  ---  ---  **Invited abstracts** remain in an "Invited" state until they are submitted (completed) by the person in question. On that happens, they transition to their regular "Awaiting review" state.  ### Missing Features | Title | Description | Status | | ------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------- | | **Max. Pages for final paper** | *Clarification needed* | To be discussed | | **Flag to indicate if the type can be invited oral** | *Clarification needed* | To be discussed | | **Sort order** | *Clarification needed* | To be discussed | | **Better filtering criteria** | | To be scheduled - high priority | | **List of Abstracts UI**: option to change tracks and contribution types | This is currently not possible to be done in bulk nor through a "one-click" interface similar to the one in the list of contributions | to be scheduled - high priority | | **Abstract manager role** | There is already an "Editing manager" and a "Paper Manager" | to be scheduled - high priority | ## Peer Reviewing Peer Reviewing is implemented through the **Peer Reviewing** module (formerly called "Paper Reviewing"). This functionality is non present in SPMS and is explained in detail in [Indico's help documentation](https://learn.getindico.io/conferences/peer_reviewing/). It is worth, however, to illustrate the relationship between this module and **Paper Editing**. First of all, the **Paper Editing** module was conceived as a replacement for the **Layout reviewing** option in the **Paper Reviewing** (in the meantime renamed **Peer Reviewing** in order to avoid confusion). Generally, whenever the **Peer Reviewing** module is active, an accepted paper will be needed before **Editing** can start. Once the paper is accepted in Peer Reviewing, and if Editing is on, the user will be invited to submit their paper for edition.  ## Editing *(pgs. 42-77 ("Referee > Editor/Proceedings Administration") and 123 ("Paper Management") of "Porting SPMS features to Indico" document)* The new **Editing** module is the crown jewel of the Indico - JACoW coordination initiative and one of the most relevant developments in Indico in the last few years. It is the first module built with a new mindset of "Indico as a **platform**" on which new services can be developed, rather than a fixed (yet very flexible) general-purpose system trying to accommodate a multitude of workflows. ### Editables An **Editable** represents a piece of information which can be the object of an editing process. As of now, Indico supports three types of editable elements: * Paper * Slides * Poster Each editable type can be turned on or off for a given event.  Each editable has **its own revision timeline**, meaning that e.g. a contribution's paper may be accepted while its slides are not. It also has its own configuration: allowed file types, editing teams and others. ### Editing states Editables can be in the following states during editing: * **Submitted** - normally an "ephemeral" state (vanilla Indico), but not in the case of JACoW, as for instance papers under this state will have to be processed ("distilled") before they move to the next one; * **Ready to Review** - the editable is ready to be assigned and reviewed. In the JACoW workflow, this means that "distilling" has been concluded. A system user called **Indico Bot** will do the distilling automatically and leave the editable in this state; * **Assigned** - the editable has been assigned to an editor ("purple dot"); * **Accepted** - final state - the editable has been accepted ("green dot"); * **Rejected** - final state - the editable has been rejected; * **To be corrected** - the editable needs corrections to be performed **by the author(s)** ("red dot"); * **Requires confirmation** - **the editor** has made modifications to the editable, which **need to be approved** by the author(s) ("yellow dot"); * **Published** - the editable is ready for publishing. Furthermore, the JACoW workflow introduces an additional **in QA** state for papers. Once in this state, an paper will be evaluated by a user linked to the **SCS** (Scientific Secretariat) role.  The user will have the option of "failing" QA (paper goes back to **Assigned** state) or approving it (paper moved to a "Published" state).  The diagram below illustrate the usual transitions between states. Judgments can also be undone, and the paper/editable brought back to "Assigned" state.  ### Tags **Tags** are used to classify editables (papers, slides, etc...) Tags are fully customizable on a per-event level. In the JACoW workflow, they are used to implement the equivalent of the **error** codes which exist in SPMS.  ### File Types One of the key instruments in ensuring the quality of submitted material in JACoW conferences is the possibility which SPMS offers to limit users with regards to the files they submit and how they name them. This makes things more predictable so that further workflows can process the data accordingly. Indico's management of uploaded materials has always been quite limited. In the Peer Reviewing module especially, there are no ways to constrain user input to specific rules. Upon designing this Editing module, we were inspired by SPMS's choice of configuration, although in a slightly more simplified way (which doesn't impact its usage, however). **File Types** can be seen as folders into which submitters upload their editable's files. They can be as simple as a single PDF file, or more complex, like for instance a group of image files with different extensions.  The following fields can be configured: * **Name** - which will be displayed on the interface; * **File name template** - the format which the name of the file to be uploaded should follow; * **Extensions** - allowed file extensions for that file type; * **File required** - whether the file type is mandatory (at least one file required); * **Multiple files** - whether the file type allows for more than one file; * **Publishable** - whether the contents of the type should be displayed publicly once the editable is published. These **file types** will be displayed to the user when trying to submit an editable.  ### "Ready for Review" conditions **Ready for Review** conditions are yet another mechanism inspired by SPMS and conceived in a simplified way. Using this mechanism, organizers can set minimum requirements an editable has to fulfill in order to be considered ready for review. For instance, the presence of source files may not be sufficient, and a PDF may be required as well. These conditions can also be made more complex using "OR" operators.  ### Other configuration options Both **submission** and **editing** can be enable or disabled on a **per-editable basis**.  An **editing team** can be configured for each editable. This team may include users and/or roles.   ### Assigning Editors to Editables Editors can be assigned directly from an editable's list view:  ---  Once someone is assigned, they will be able to judge the editable.  There is also simplified **Get next paper** interface available, which allows papers to be picked according to the types of files they contain. This allows, for instance, a review who only works on LaTeX sources to select only papers written using that system.  ### The Reviewing Timeline The reviewing process itself happens in a timeline view reminiscent of Indico's older Peer Reviewing module. Revisions are represented in different boxes and actions can be performed on the spot with very little menu-diving. Comments can be added by any of the editors set for the give editable type, but only assigned editors can emit a judgment.  Snapshots of the various revisions of the editable can be retrieved. ### Missing Features | Title | Description | Status | | ---------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------- | --------------- | | **Allow customization of sent notification e-mails** | | To be discussed | | **Better/more filtering options** | | To be discussed | | **Poster police** | This can probably be achieved through a set of contribution fields and sending the corresponding information to the JACoW micro-service | To be discussed | ## Other missing features In addition to the those mentioned at the end of each section, these are the missing features with regads to the plan established by the “Porting SPMS features to Indico” document: * **Event-specific rooms** - this would make things easier for events which do not take place at CERN (which is the usual in JACoW), and also provide a way to assign codes to each room (useful also for programme codes); * **Fuzzy user search** - there is probably some room for improvement in Indico; * **Poster assignment** - there is currently no concept of "capacity" in poster sessions (*to be discussed*); * **Mobile app** - this would be a resource-demanding and technically complex endeavor which would require a sizable investment and maintenance effort; * **E-dot board** - Very JACoW-specific, should be added to a plugin, or just fetch information through an API; * **My schedule** option * **Statistics/Reports** - Indico could have some more of those, but JACoW-specific reports could be included in a dedicated plugin; ## Some technical details The **Editing** workflow we propose for JACoW is composed of: * Indico's core **Editing** module and its base functionality; * An additional **micro-service**, and autonomous component which communicates with Indico while providing the JACoW-specific parts of the workflow. The micro-service implements two particular features which would otherwise be too hard to provide within Indico (while maintainge its general-purpose nature): * PDF "distilling" with Ghostscript; * Final QA. ## Additional Software Components Here we list the additional software components which power the JACoW workflow. They are available on the production instance ([`indico.jacow.org`](http://indico.jacow.org)). ### flask-multipass-jacow In order to allow users to use their existing JACoW account passwords and to provide Indico with the data which is already available on the repository, a `flask-multipass` plugin was developed, which integrates with the JACoW Oracle database and uses the account information contained in it to authenticate/identify users. * [GitLab Repo](https://gitlab.cern.ch/indico/flask-multipass-jacow) ### JACoW OpenReferee Implementation Some of the JACoW-specific paper editing workflows are implemented through an additional component which integrates with Indico through the OpenReferee API. OpenReferee is an API which allows for integration between Indico and external workflow engines. * [GitHub Repo](https://github.com/indico/openreferee-jacow) * [OpenReferee specification](https://github.com/indico/openreferee)