The unison-fsmonitor tool can watch a directory for file changes, and trigger Unison to do it’s sync-thang automatically without the need to call it manually. May 18, 2015 at 1:47 pm I had previously made shortcuts with the Tab key, like the Terminal shortcuts to switch tabs that I mention in the blog post. But I just checked, and despite the fact that the shortcuts show up in System Preferences, the menu items in Terminal still show the original keyboard shortcuts, and the ones that use the Control.
Virtualization technologies such as Docker, Vagrant, and Virtualbox provide new opportunities for pre-built development environment images. But where do your project files go?
I personally prefer keeping my files on my local development machine (a laptop in my case) – it is generally faster to develop in (the IDE is faster) and it means I can blow away the virtualized environment at any time, knowing my master source code is safe.
But how then to get the files into the virtualized environment?
Unison
The technology I wanted to focus on for this post is Unison, a bi-directional file syncing application available on Linux, Windows, and Mac OS X. You can set it up so any change on either file system is automatically replicated to the other side. This means for Magento, local file system edits are copied into the virtualized environment, plus any var/generated code created are automatically copied back to the laptop, for use by my IDE during debugging.
When you start up Unison, it does a full tree walk doing any necessary file copying. The default rule to check files is based on file size and timestamps. (You can use file contents checksum instead, but it runs slower.) After that phase, it watches the file system on the client and server hosts for modification events and triggers an incremental sync operation to push changes for modified files to the other end.
Unison uses ssh to log on to the remote server, making it also possible to use on remote cloud servers securely. It also supports plain sockets.
Documentation for Unison can be found at http://www.cis.upenn.edu/~bcpierce/unison/docs.html. It supports “profiles” to be defined by configuration files on disk. I am going old school here at the moment, and just using command line arguments (in a short shell script) to invoke the command. That is, for Magento 2, I am using the following command. Note that /magento2 is where file files reside inside the virtualized environment in this example.
(Please let me know of any additions or removals to the above list.)
Note the above command includes synchronizing ‘vendor’, which adds quite a bit of startup time (around 30 seconds on my laptop). I was also playing with using the following command to do the sync once on ‘vendor’, then exit.
Then I run that by hand whenever I make a change to the vendor directory. That solved the general startup performance problem of the watch mode as it could exclude the whole vendor directory, which is reasonably large.
Paths
The nice thing is on Windows the pre-compiled binaries I managed to find appear to be self-contained. That is, you don’t need to install Cygwin or similar to run the tool within. There is nothing wrong with Cygwin, but I find I have Cygwin, Git Bash, PowerShell, and CMD prompts open at different times. Cygwin binaries expect paths starting with /cygdrive/c/, Git Bash uses /c/ as prefix, and native Windows utilities use C:. It gets confusing at times.
For Unison, I use paths relative to the project directory, with forward slashes. Following this strategy has not caused any problems to date.
Installation
Unison is an interesting project to me as it feels more like a “traditional” open source project. Spurts of activity at times, no commercial backing, different community members providing binaries for different platforms, various blog posts describing how to install that are out of date, etc.
For example, there are different versions of Unison. You need to make sure the client and the server you are talking to are the same version or else you may get problems. Contacts journal crm 2 1 12. That is, it will start up without error, but after a while things go wrong.
So which version to use? The default that comes with Debian (easily installed via apt-get) is 2.40.102. Unfortunately, the Windows binary download page I found at https://www.irif.fr/~vouillon/unison/ does not have the same version available. I tried 2.40.69 (from back in 2011), but had some problems getting it working – possibly user error, I don’t know. But I found a newer version that did work 2.48.3. (Oh, that is 2.48.3 compiled with OCaml 4.01.0, not the binary compiled with OCaml 4.02.1 which the download page says in incompatible… ah, the joys of open source!)
Fsmonitor 1 123 Movies
Which is the primary reason for me posting this post. Its October 2016 as I write this, and the combination of Linux and Windows binaries that I found that appear to work together are… (drum roll please!):
Fsmonitor 1 123 Movie
Windows Installation
For Windows, there is a site with precompiled binaries. You can go to https://www.irif.fr/~vouillon/unison/ and grab the ZIP from https://www.irif.fr/~vouillon/unison/unison%202.48.3.zip. This ZIP includes the Unison executable, and second executable unison-fsmonitor.exe for watching for file system events (needed for the “watch” mode to work). Put these two binaries in your path, and you are ready to go.
Linux Installation
I could not find a nice version around that I could get to work via apt-get (when I followed the instructions in all the blog posts, they reported 404 errors), but there are binaries available online. So I used the following:
Not as nice as apt-get, but pretty straightforward regardless.
Mac OS X Installation
I have not tried it myself yet, but the Windows download page also has a section for OS X. Just download the ZIP at http://unison-binaries.inria.fr/files/Unison-OS-X-2.48.3.zip.
Conclusions
So far so good. I want to experiment more, but Unison is looking quite promising. It allows me to keep the master files natively on my laptop, it supports bi-directional syncing, has precompiled binaries, and more. I am writing this post in part to share the tool for those who did not know of it, but also asking for any experience of others using this tool.
Referring back to my previous post on the Magento 2 tool chain, Unison may be a good technology for the “file sharing” double-headed arrow for syncing files between the “project source files” and the “full-stack development environment”.
It is interesting also because it is not tied to any specific virtualization technology – it only relies on network access to the environment (ssh or raw sockets can both be used).
PS: Other Technologies
There are a range of alternative approaches that may also be of interest. I have mentioned many of these in previous posts, but I will briefly mention a list here as well in case you want to explore other options further.
- You can run Samba inside the virtualized environment and then mount that from your development environment.
- PHP Storm has built in “copy on write” where whenever it saves a file, it can also save a copy in the remote container.
- WinSCP includes a “copy on write” (keep in sync) mode that will sftp a copy of any files in the local directory to the remote server. This is similar to PHP Storm, but works with any text editor as it just watches the file system for changes.
- Docker 1.12 introduces improved native volume mounting, allowing Docker containers to mount the native file system directly. For Windows, this does not support iNotify events (yet), meaning tools like Grunt and Gulp inside the virtualized environment do not get “file changed” events. For Mac OS X, there are still some performance problems with very large numbers of files.
- There is the docker-sync project that can use Unison or rsync to share files. It is kinda cool in demonstrating the flexibility of Docker – you spin up a separate container mounting the same file system and away you go. It will work with any mounted file system without modification to your other containers.
- Vagrant rsync-auto I was trying recently to pretty good effect. It watches the file system and then performs a sync command to copy local file system changes into the container. If using Vagrant, it is worth looking into.
- There is also rsync of course, but it is only a one-directional replication solution.
What was supposed to be a quick-and-dirty on 2-way file synchronisation for Debian servers using Unison.
Pros | Cons |
---|---|
2-way synchronisation | May need to build it from source code |
Exists in Debian repositories | Not under ‘active’ development |
Runs under Unix/Windows/OSX | More than 2 nodes Star topology recommended (ie, single point of failure) |
Homepage:https://www.cis.upenn.edu/~bcpierce/unison/
GitHub:https://github.com/bcpierce00/unison
Manual:https://www.cis.upenn.edu/~bcpierce/unison/download/releases/stable/unison-manual.html
GitHub:https://github.com/bcpierce00/unison
Manual:https://www.cis.upenn.edu/~bcpierce/unison/download/releases/stable/unison-manual.html
See Also:https://medium.com/@yobi.kim/how-to-install-unison-91c9fb36ac09
Installation
From Debian Repos
Assuming a Debian server, we can install Unison using apt-get.
Currently (2018/09/03) this will install the following version of Unison, depending on the version of Debian you’re running:
- 8 Jessie: v2.40.102
- 9 Stretch: v2.48.3
From source
If a later release than that published in the Debian repositories is required, we can build it from source.Grab the latest release source code from https://github.com/bcpierce00/unison/releases
Make sure we have the required OCaml compiler stuff
Then build it (replacing 2.51.2 with whatever is the latest release)
Hmm…
Looks like neither of the above installation methods provide us with the unison-fsmonitor tool, which would allow us to run Unison kinda as a daemon/service with some tweaks.The unison-fsmonitor tool can watch a directory for file changes, and trigger Unison to do it’s sync-thang automatically without the need to call it manually.
Testing it out locally
It’s possible to synchronise 2 directories on the same machine, so in the name of KISS, let’s try that out first.In this experiment we’ll try to keep the following two directories ‘in sync’:
Running Unison Manually
Configuration
In our Unison ‘profile’, let’s try the following so it knows ‘sup/home/jarrod/.unison/local.prf
Push the big red button
Apparently running Unison should be as simple as calling unison with the name of our profile (in this case ‘local’).
Well, that’s interesting.
Makes sense. Let’s create some test files and run it
FUCKING YES! ?
Fsmonitor 1 123movies
Remove some files, and… Icircuit 1 8 1 download free.
Ok. So it basically does what I want, but only when I tell it to.How can we automate this process?
Watch directories and sync automatically with Unison
Building on our above Unison profile, let’s use the following
/home/jarrod/.unison/local.prf
Now, when we run Unison with our ‘local’ profile, it’ll keep running and watch the ' root’ directories for changes.
This is great!
Where it all comes crashing down ?
The maximum number of files that can be ‘watched’ for changes by in Linux using inotify (which I can only assume unison-fsmonitor is doing) is defined by the fs.inotify.max_user_watches system variable. By default (in Debian land, at least) this number is 8192.
Ok, so usually we are able to change these values - that’s the whole point of having configuration options, right?
Permission fucking what?
It’s interesting to note that I’ve tried this on another “VPS” that runs under OpenVZ on my own hardware,and it appears to be a limitation of this type of virtualisation, not necessarily something that has been overridden/forbidden by a host.
Trying the same command on a KVM virtual machine is a whole ‘nother story.
Fsmonitor 1 123 Drive
Works like a bought one! ?
![Drive Drive](https://miro.medium.com/max/914/1*DUdmJg5PO6NO8AO1AteaHw.png)
To make the change to fs.inotify.max_user_watches permenant, ie not lost between reboots, you can add it to /etc/sysctl.conf.
TL;DR
Swinsian 1 12 0 – music manager and player. So, this could be a viable option ifA) You run your own hardwareB) You aren’t running on some VM under OpenVZ
Unfortunately that’s not the situation I’m in right now ?