osxfs is a new shared file system solution, exclusive to Docker Desktop for Mac.osxfs provides a close-to-native user experience for bind mounting macOS filesystem trees into Docker containers. To this end, osxfs features a number ofunique capabilities as well as differences from a classical Linux file system.
In my extensions code I have a possibility to browse a destination folder for User. I`m using common way to do so: window.cep.fs.showOpenDialog(false, true, 'Select Folder', currentPath, null); It works pretty good during years. But occured starting form the v16.1.2 (v16.1.3 also) this function wor. One of the more advanced tools that’s nice to have in your arsenal of “what to do if things go wrong” is a Terminal command called fsusage. Fix sb ports for arduino el capitan. This will show you what your file system is doing at any.
Case sensitivity
With Docker Desktop for Mac, file systems operate in containers in the same way as theyoperate in macOS. If a file system on macOS is case-insensitive, that behavioris shared by any bind mount from macOS into a container.
On macOS Sierra and lower, the default file system is HFS+. On macOS HighSierra, the default file system is APFS. Both are case-insensitive bydefault but available in case-sensitive and case-insensitive variants.
To get case-sensitive behavior, format the volume used in your bind mount asHFS+ or APFS with case-sensitivity. See theAPFS FAQ.
Reformatting your root partition is not recommended as some Mac software relieson case-insensitivity to function.
Access control
osxfs, and therefore Docker, can access only those file system resources thatthe Docker Desktop for Mac user has access to. osxfs does not run as root. If the macOSuser is an administrator, osxfs inherits those administrator privileges. Weare still evaluating which privileges to drop in the file system process tobalance security and ease-of-use. osxfs performs no additional permissionschecks and enforces no extra access control on accesses made through it. Allprocesses in containers can access the same objects in the same way as theDocker user who started the containers.
Namespaces
Much of the macOS file system that is accessible to the user is also available tocontainers using the -v bind mount syntax. The following command runs a containerfrom an image called r-base and shares the macOS user’s ~/Desktop/ directory as/Desktop in the container.
The user’s ~/Desktop/ directory is now visible in the container as a directoryunder /.
By default, you can share files in /Users/, /Volumes/, /private/, and/tmp directly. To add or remove directory trees that are exported to Docker,use the File sharing tab in Docker preferences -> Preferences ->File sharing. (See Preferences.)
All other pathsused in -v bind mounts are sourced from the Moby Linux VM running the Dockercontainers, so arguments such as -v /var/run/docker.sock:/var/run/docker.sockshould work as expected. If a macOS path is not shared and does not exist in theVM, an attempt to bind mount it fails rather than create it in the VM. Pathsthat already exist in the VM and contain files are reserved by Docker and cannotbe exported from macOS.
See Performance tuning for volume mounts (shared filesystems)to learn about new configuration options available with the Docker 17.04 CE Edge release.
Ownership
Initially, any containerized process that requests ownership metadata of anobject is told that its uid and gid own the object. When any containerizedprocess changes the ownership of a shared file system object, such as by usingthe chown command, the new ownership information is persisted in thecom.docker.owner extended attribute of the object. Subsequent requests forownership metadata return the previously set values. Ownership-based permissionsare only enforced at the macOS file system level with all accessing processesbehaving as the user running Docker. If the user does not have permission toread extended attributes on an object (such as when that object’s permissionsare 0000), osxfs attempts to add an access control list (ACL) entry thatallows the user to read and write extended attributes. If this attempt fails,the object appears to be owned by the process accessing it until the extendedattribute is readable again.
File system events
Most inotify events are supported in bind mounts, and likely dnotify andfanotify (though they have not been tested) are also supported. This meansthat file system events from macOS are sent into containers and trigger anylistening processes there.
The following are supported file system events:
- Creation
- Modification
- Attribute changes
- Deletion
- Directory changes
The following are partially supported file system events:
- Move events trigger
IN_DELETEon the source of the rename andIN_MODIFYon the destination of the rename
The following are unsupported file system events:
- Open
- Access
- Close events
- Unmount events (see Mounts)
Some events may be delivered multiple times. These limitations do not apply toevents between containers, only to those events originating in macOS.
Mounts
The macOS mount structure is not visible in the shared volume, but volumecontents are visible. Volume contents appear in the same file system as the restof the shared file system. Mounting/unmounting macOS volumes that are also bindmounted into containers may result in unexpected behavior in those containers.Unmount events are not supported. Mount export support is planned but is stillunder development.
Symlinks
Symlinks are shared unmodified. This may cause issues when symlinks containpaths that rely on the default case-insensitivity of the default macOS filesystem.
File types
Symlinks, hardlinks, socket files, named pipes, regular files, and directoriesare supported. Socket files and named pipes only transmit between containers andbetween macOS processes -- no transmission across the hypervisor is supported,yet. Character and block device files are not supported.
Extended attributes
Extended attributes are not yet supported.
Technology
Fsc For Machined Parts
osxfs does not use OSXFUSE. osxfs does not run under, inside, orbetween macOS userspace processes and the macOS kernel.
SSH agent forwarding
Docker Desktop for Mac allows you to use the host’s SSH agent inside a container. To do this:
Bind mount the SSH agent socket by adding the following parameter to your
docker runcommand:--mount type=bind,src=/run/host-services/ssh-auth.sock,target=/run/host-services/ssh-auth.sockAdd the
SSH_AUTH_SOCKenvironment variable in your container:-e SSH_AUTH_SOCK='/run/host-services/ssh-auth.sock'
To enable the SSH agent in Docker Compose, add the following flags to your service:
Performance issues, solutions, and roadmap
See Performance tuning for volume mounts (shared filesystems)to learn about new configuration options available with the Docker 17.04 CE Edge release.
With regard to reported performance issues (GitHub issue 77: File access inmounted volumes extremely slow),and a similar thread on Docker Desktop for Mac forums on topic: File access in mountedvolumes extremelyslow,this topic provides an explanation of the issues, recent progress in addressingthem, how the community can help us, and what you can expect in thefuture. This explanation derives from a post about understandingperformanceby David Sheets (@dsheets) on the Docker developmentteam to the forum topic justmentioned. We want to surface it in the documentation for wider reach.
Understanding performance
Perhaps the most important thing to understand is that shared file systemperformance is multi-dimensional. This means that, depending on your workload,you may experience exceptional, adequate, or poor performance with osxfs, thefile system server in Docker Desktop for Mac. File system APIs are very wide (20-40message types) with many intricate semantics involving on-disk state, in-memorycache state, and concurrent access by multiple processes. Additionally, osxfsintegrates a mapping between macOS’s FSEvents API and Linux’s inotify APIwhich is implemented inside of the file system itself, complicating mattersfurther (cache behavior in particular).
At the highest level, there are two dimensions to file system performance:throughput (read/write IO) and latency (roundtrip time). In a traditional filesystem on a modern SSD, applications can generally expect throughput of a fewGB/s. With large sequential IO operations, osxfs can achieve throughput ofaround 250 MB/s which, while not native speed, is not likely to be the bottleneck formost applications which perform acceptably on HDDs.
Latency is the time it takes for a file system call to complete. For instance,the time between a thread issuing write in a container and resuming with thenumber of bytes written. With a classical block-based file system, this latencyis typically under 10μs (microseconds). With osxfs, latency is presentlyaround 130μs for most operations or 13× slower. For workloads which demand manysequential roundtrips, this results in significant observable slowdown.Reducing the latency requires shortening the data path from a Linux system call tomacOS and back again. This requires tuning each component in the data path inturn -- some of which require significant engineering effort. Even if we achievea huge latency reduction of 65μs/roundtrip, we still “only” see a doublingof performance. This is typical of performance engineering, which requiressignificant effort to analyze slowdowns and develop optimized components. Weknow a number of approaches that may reduce the roundtrip time but wehaven’t implemented all those improvements yet (more on this below inWhat you can do).
A second approach to improving performance is to reduce the number ofroundtrips by caching data. Recent versions of Docker Desktop for Mac (17.04 onwards)include caching support that brings significant (2-4×) improvements to manyapplications. Much of the overhead of osxfs arises from the requirement tokeep the container’s and the host’s view of the file system consistent, butfull consistency is not necessary for all applications and relaxing theconstraint opens up a number of opportunities for improved performance.
At present there is support for read caching, with which the container’s viewof the file system can temporarily drift apart from the authoritative view onthe host. Further caching developments, including support for write caching,are planned.A detailed description of the behavior in various caching configurationsis available.
What we are doing
We continue to actively work on increasing caching and on reducing thefile system data path latency. This requires significant analysis of filesystem traces and speculative development of system improvements to try toaddress specific performance issues. Perhaps surprisingly, applicationworkload can have a huge effect on performance. As an example, here are twodifferent use cases contributed on theforum topicand how their performance differs and suffers due to latency, caching, andcoherence:
A rake example (see below) appears to attempt to access 37000+different files that don’t exist on the shared volume. Even with a 2× speedupvia latency reduction this use case still seems “slow”.With caching enabled the performance increases around 3.5×, as described inthe user-guided caching post.We expect to see further performance improvements for rake with a “negative dcache” thatkeeps track of, in the Linux kernel itself, the files that do not exist.However, even this is not sufficient for the first time rake is run on ashared directory. To handle that case, we actually need to develop a Linuxkernel patch which negatively caches all directory entries not in aspecified set -- and this cache must be kept up-to-date in real-time with the macOSfile system state even in the presence of missing macOS FSEvents messages andso must be invalidated if macOS ever reports an event delivery failure.
Running
ember buildin a shared file system results in ember creating manydifferent temporary directories and performing lots of intermediate activitywithin them. An empty ember project is over 300MB. This usage pattern does notrequire coherence between Linux and macOS, and is significantly improved bywrite caching.
These two examples come from performance use cases contributed by users and theyare incredibly helpful in prioritizing aspects of file system performance toimprove. We are developing statistical file system trace analysis toolsto characterize slow-performing workloads more easily to decide what towork on next.
Under development, we have:
A growing performance test suite of real world use cases (more on this belowin What you can do)
Further caching improvements, including negative, structural, and write-backcaching, and lazy cache invalidation.
A Linux kernel patch to reduce data path latency by 2/7 copies and 2/5context switches
Increased macOS integration to reduce the latency between the hypervisor andthe file system server
What you can do
When you report shared file system performance issues, it is most helpful toinclude a minimal Real World reproduction test case that demonstrates poorperformance.
Without a reproduction, it is very difficult for us to analyze your use case anddetermine what improvements would speed it up. When you don’t provide areproduction, one of us needs to figure out the specific softwareyou are using and guess and hope that we have configured it in a typical way ora way that has poor performance. That usually takes 1-4 hours depending on youruse case and once it is done, we must then determine what regular performance islike and what kind of slow-down your use case is experiencing. In some cases, itis not obvious what operation is even slow in your specific developmentworkflow. The additional set-up to reproduce the problem means we have less timeto fix bugs, develop analysis tools, or improve performance. So, includesimple, immediate performance issue reproduction test cases. The rakereproductioncaseby @hirowatari shown in the forums thread is a great example.
This example originally provided:
A version-controlled repository so any changes/improvements to the test casecan be easily tracked.
A Dockerfile which constructs the exact image to run
A command-line invocation of how to start the container
This update bring fixes and improvements to all compatible Macs, as well as specific updates for SDR workflow handling on the Pro Display XDR, as well as multi-stream video editing performance on the 16-inch MacBook Pro. December 10, 2019: Apple releases macOS Catalina 10.15.2Apple has released macOS Catalina 10.15.2 to the public. Latest macos for 2012imac. This upate brings improvements for apps like Music, Stocks, News, and Mail, as well as general improvements to stability and reliability.
A straight-forward way to measure the performance of the use case
A clear explanation (README) of how to run the test case
What you can expect
What Fs For Macos 10
We continue to work toward an optimized shared file system implementationon the Edge channel of Docker Desktop for Mac.
What Fs For Macos Download
You can expect some of the performance improvement work mentioned above to reachthe Edge channel in the coming release cycles.
We plan to eventually open source all of our shared file system components. Atthat time, we would be very happy to collaborate with you on improving theimplementation of osxfs and related software.
We also plan to write up and publish further details of shared file systemperformance analysis and improvement on the Docker blog. Look for or nudge@dsheets about those articles, which should serve as a jumping off point forunderstanding the system, measuring it, or contributing to it.
Wrapping Up
We hope this gives you a rough idea of where osxfs performance is and whereit’s going. We are treating good performance as a top priority feature of thefile system sharing component and we are actively working on improving itthrough a number of different avenues. The osxfs project started in December
- Since the first integration into Docker Desktop for Mac in February 2016, we’veimproved performance by 50x or more for many workloads while achieving nearlycomplete POSIX compliance and without compromising coherence (it is shared andnot simply synced). Of course, in the beginning there was lots of low-hangingfruit and now many of the remaining performance improvements require significantengineering work on custom low-level components.
We appreciate your understanding as we continue development of the product andwork on all dimensions of performance. We want to continue to work with thecommunity on this, so continue to report issues as you find them. We lookforward to collaborating with you on ideas and on the source code itself.