Teleworking Tips & Tricks CERN

--- title: Teleworking Tips & Tricks CERN tags: general --- ## Development vscode's remote development is probably the best option to develop remotely on code stored on your work PC ## Relevant ITSSB and ISP particulars https://codimd.web.cern.ch/s/ByAUAEWcL ## Remote Access ### Recommendations In addition to the recommendations below, check [this page](https://security.web.cern.ch/security/recommendations/en/ssh_tunneling.shtml) from the security team. Either use the `ProxyJump` method or `sshuttle`, not both. #### SSH to any CERN host with ProxyJump Put this in your `~/.ssh/config` ``` Host *.cern.ch !lxtunnel.cern.ch !lxplus.cern.ch !aiadm.cern.ch !lxtunnel !lxplus !aiadm !gitlab.cern.ch User YOURCERNUSER ProxyJump lxtunnel.cern.ch Host lxtunnel.cern.ch lxplus.cern.ch aiadm.cern.ch lxtunnel lxplus aiadm User YOURCERNUSER # if your client is configured for Kerberos GSSAPIDelegateCredentials yes GSSAPIAuthentication yes GSSAPITrustDns yes # SOCKS5 proxy for web, see below DynamicForward 8090 ExitOnForwardFailure no ``` The keyword 'GSSAPITrustDns' is not accepted by recent versions of macOS, so Mac users should comment it out. Ideally you'd use Kerberos - `kinit yourcernuser@CERN.CH` on your home machine. If you need any configuration beyond installing the Kerberos client package, look at [the docs](http://linux.web.cern.ch/linux/docs/kerberos-access.shtml) Else: setup ssh key authentication on your work machine (please note that AFS/EOS access won't work with key authentication..). * `ssh-keygen -o -a 100 -t ed25519` to generate a keypair if you don't have one yet; * add the contents of `~/.ssh/id_ed25519.pub` to `~/.ssh/authorized_keys` on your remote machine. If you want it on lxplus: create the file in `~/public`, and symlink from `~/.ssh/authorized_keys` to the one in `~/public`: ```bash ln -s ${HOME}/public/authorized_keys ${HOME}/.ssh/authorized_keys ``` On Windows, ProxyJump may not work. You'll have to replace ProxyJump with [this instead](https://github.com/PowerShell/Win32-OpenSSH/issues/1172#issuecomment-544122987): `ProxyCommand ssh lxtunnel.cern.ch -W %h:%p` ##### How to run VNC session To work on remote applications (emacs, browser within the CERN network, etc), consider connecting through vnc instead of X-forwarding. To do so, ssh into lxplus twice, once to start the VNC server, and once to allow your computer to connect to it: ``$ ssh -C lxplus.cern.ch "vncserver -localhost -fg"`` You will require a password to access your desktops. ``` Password: Verify: Would you like to enter a view-only password (y/n)? n A view-only password is not used New 'lxplusNNN.cern.ch:D (MYUSER)' desktop is lxplusNNN.cern.ch:D ... ``` **Note** that as the VNC password is stored in a file, you should NOT use your CERN credentials. You only have to set it once. You can store the file in private doing: ``` mkdir $HOME/Private/vnc ln -s $HOME/Private/vnc $HOME/.vnc ``` Now that we know on which port VNC server is running, forward port 5900+D (the display number from above) and connect to the same lxplus node, from a different terminal: `$ ssh -L15900:localhost:590D lxplusNNN.cern.ch` And finally, connect to `localhost:15900` using any VNC viewer, such as Remmina on Linux. You will be greeted with a password prompt, where you enter the VNC password you just set. #### SSH Socks Proxy for Web traffic Create a SOCKS5 proxy via `lxtunnel`: ``` ssh -D 8090 youruser@lxtunnel.cern.ch ``` Note for users connecting from French ISPs affected by the IPv6 package dropping issue: add `-4` to the SSH tunneling command to force the use of IPv4. And configure it on your browser. Use a proxy switcher ([firefox](https://addons.mozilla.org/en-US/firefox/addon/switchyomega/), [chrome](https://chrome.google.com/webstore/detail/proxy-switchyomega/padekgcemlokbadohgkifijomclgjgif)) if you want to selectively send only CERN traffic via the proxy. You can also launch a separate dedicated browser process for the proxy ``` google-chrome --proxy-server=socks5://127.0.0.1:8090 firefox -P <my SOCKS profile> # manage profiles with firefox --ProfileManager ``` Configuration without additional proxy switcher- - **Mac - System Wide** Settings -> Network -> Advanced -> Proxies Tab -> Check and configure SOCKS proxy with host as `127.0.0.1` and port `8090`. - **Firefox** (tested on Mac) Firefox -> Preferences -> Network Settings -> Settings button -> Manual proxy configuration SOCKS Host - 127.0.0.1 Port - 8090 Check Proxy DNS when using SOCKS v5 This way any site you open in Firefox is routed through the proxy and you can use another browser for sites that you do not want to get routed through CERN (and improve speed on them). - **Windows 10 - System Wide** Internet Options -> Connections -> LAN Settings -> Check "Use a proxy server for your LAN" -> Advanced -> Enter into Socks: `127.0.0.1` : `8090` and clear out all other addresses/ports. Here is then a convenient AutoHotKey to switch the proxy on and off: https://gist.github.com/mydoghasworms/6069753 #### sshuttle [sshuttle](https://github.com/sshuttle/sshuttle) is a "poor man's VPN" solution which works on macOS and Linux. It uses SSH tunnelling to transparently redirect certain parts of your traffic to the internal network. ::: danger ** Probably too much, better to use the config below** > Install [sshuttle](https://github.com/sshuttle/sshuttle) and then run: > ``` > sshuttle --dns -v --remote <username>@lxtunnel.cern.ch 0.0.0.0/0 > ``` > Some questions on this one: > - forwarding everything through CERN network is a bit too much... Is there a way to specify all the IPv4 ranges for CERN? > There is this: https://landb.cern.ch/landb/portal/cernNetwork ::: This is my config (I saved it in a `cern_proxy` script file): ```shell=sh #!/bin/sh case $1 in connect) sshuttle --dns -vr <USER>@lxtunnel.cern.ch 137.138.0.0/16 128.141.0.0/16 128.142.0.0/16 188.184.0.0/15 --daemon --pidfile /tmp/sshuttle.pid shift ;; disconnect) kill `cat /tmp/sshuttle.pid` shift ;; *) # unknown option ;; esac ``` > - `sshuttle` and IPv6? `--method=tproxy`? ##### sshuttle on macOS ``` sshuttle --dns -v --remote <username>@lxtunnel.cern.ch 128.141.0.0/16 128.142.0.0/16 137.138.0.0/16 185.249.56.0/22 188.184.0.0/15 192.65.196.0/23 192.91.242.0/24 194.12.128.0/18 2001:1458::/32 2001:1459::/32 ``` The above command can be used on macOS to tunnel both IPv4 and IPv6 traffic to CERN IP space via lxtunnel. ##### sshuttle on Arch Linux/Other OS If you're having problems on Arch Linux/Other OS with connection (likely a specific forward): ```bash # ... client_loop: send disconnect: Broken pipe # ... c : fatal: ssh connection to server (pid 1234) exited with returncode 255 ``` Follow this [comment in sshuttle GitHub](https://github.com/sshuttle/sshuttle/issues/150#issuecomment-733052508): ```bash pacman -S iptables-nft nftables systemctl enable nftables systemctl start nftables ``` Or [this other comment](https://github.com/sshuttle/sshuttle/issues/150#issuecomment-358113610) and add a `-x` flag pointing to the host you are using for tunneling, e.g.: ```bash sshuttle --dns -vr <USER>@aiadm.cern.ch -x aiadm.cern.ch 137.138.0.0/16 128.141.0.0/16 128.142.0.0/16 188.184.0.0/15 ``` ##### sshuttle and kerberos By default, kerberos uses UDP for sufficiently small messages, which doesn't play nicely with `sshuttle` as the latter only supports TCP. This might cause `kinit` to time out. The solution is to force TCP mode for keberos even for small messages by editing `/etc/krb5.conf` and adding `udp_preference_limit = 0` under the `[libdefaults]` section. ##### sshuttle troubleshooting If you are experiencing problems resolving CERN internal domains like `foreman.cern.ch` and if your machine is using systemd-resolved, make sure you are running a sshuttle version >= 1.0.5. Older versions have a bug that does not allow a good interaction between sshuttle and systemd-resolved. If the problem still persists, run `# systemctl restart systemd-resolved` ### Windows Terminal Server * Create an RDP connection to `cernts.cern.ch` * Use remmina on Ubuntu or Centos 7, works ok * Use Chrome (or Firefox, or Edge ... just not IE) to run Rundeck jobs etc. In case of performance issues you should log out and log in again. If your Windows menu is not displaying type `logoff` in a command prompt or visit [LogMeOff website](https://remotedesktop.web.cern.ch/remotedesktop/UserTools/LogMeOff.aspx). More info: [KB0006556](https://cern.service-now.com/service-portal?id=kb_article&n=KB0006556) ### Windows Remote desktop gateway Instructions: [KB0006556](https://cern.service-now.com/service-portal?id=kb_article&n=KB0006556) ### VPN Setup https://security.web.cern.ch/security/rules/en/vpn.shtml ### Access to Kubernetes cluster from outside CERN Assuming you have your socks proxy setup (see *Remote Access* above) and kubectl >=1.19 you can set a proxy on your kubernetes cluster config file. Simply add the ```proxy-url``` param in your cluster entry in the config, example: ``` $ cat config - cluster: certificate-authority-data: ... server: https://IPOFYOURCLUSTER:6443 proxy-url: socks5://localhost:8090 $ kubectl get pod ... ``` or set it via an environment variable (will apply to ALL connections) ([ref](https://stefan.midjich.name/technical/kubernetes/kubectl_cheatsheet/)) ``` export HTTPS_PROXY=socks5://localhost:8090 kubectl cluster-info ``` ## Meetings / Conference Calls ### Quick Calls [Vidyo](https://webrtc.vidyo.cern.ch/web/index.html?portal=vidyoportal.cern.ch) offers a user room, you can share that link. ### Indico Alternatively [create an event](https://indico.cern.ch/#create-event:meeting) in Indico and enable the Vidyo room. - Camping in a team room works quite well (mic/video off but audio on), allowing people to "drop in" for a discussion. Though note that you can't be interrupted for a point-to-point call while you are connected to another room. ### Scrum If you are organizing sprint planning meetings with points estimation for each story this online service worked fine for us: https://scrum-poker.org ## What could be improved? What changes can be made to IT services in future to make things easier to work remotely and collaborate ? ### Vidyo - A hand up function would be very useful to say that you'd like to talk without interrupting the person speaking - Notification of an incoming call when you're already connected to a room would be useful ### EDH - When requesting teleworking, the 'delegate my EDH requests' defaults to 'yes if longer than one day'. For teleworking, the person is nominally working so should not need their requests delegated. It is therefore recommended to default this to 'no' when the teleworking option is selected. - why does this option exist at all for teleworking? you are literally working so there should be no need to redirect documents in such a case whatsoever (+1 on this comment) - Note that a single EDH leave request can include both teleworking days (delegation not needed), and other types of leave (delegation potentially needed). Perhaps EDH should simply not redirect documents during telewrking periods? ### Oracle https://cern.service-now.com/service-portal?id=kb_article&n=KB0004111 # Changelog - 2020-03-25: Added `!gitlab.cern.ch` on the list of host that shouldn't be jumped through aiadm, on the ssh configuration --gekaklam - 2021-07-02: Added reference to KB to do remote connections to Oracle databases --grancher