Feed
 

APFS file format read-write compatibility

Avatar Euan Williams
Members pondering an upgrade to Apple's new SSD file format (APFS) which is mandatory for Macs with SSD drives with High Sierra [macOS 10.13.x] can read about read-write between HFS+ drives and APFS drives here.

Quote <<
Devices formatted as Mac OS Extended (HFS+) can be read from and written to by devices formatted as APFS.

Devices formatted as APFS can be read from and written to by:
Other devices formatted as APFS
Devices formatted as Mac OS Extended, if using macOS High Sierra.

For example, a USB storage device formatted as APFS can be read by a Mac using High Sierra, but not by a Mac using Sierra or earlier.
>> Unquote.

If you do NOT have an SSD drive in your Mac your drive format will be unaffected by the High Sierra upgrade. This includes traditional "spinning metal" and "Fusion" drives.

Re: APFS file format read-write compatibility

Avatar Mick Burrell
Thanks Euan - I was asked today what would happen with Fusion drives and my guess was that they'd be treated as spinning platter devices and not affected. I guessed Apple would not bother to format the SSD and platter sections differently. So I guessed correctly :-)

Re: APFS file format read-write compatibility

Avatar Euan Williams
Thanks, Mick. Glad it helped someone, I'm indebted to Doug Cheney for the Fusion Drive reference.

Just wondered what would happen to an old 10.6 Snow Leopard partition if I installed High Sierra on another partition (all on the same SSD). The SSD is now APFS as expected, and doing an option-start to select the Snow Leopard OS, it started just fine.

The only immediate curiosity was that the High Sierra desktop icon wasn't present and the [Snow Leopard] Disc Utility reported the High Sierra partition by its Unix system name rather than its 'given' name.

Too late to investigate further tonight, but it's a positive!

Re: APFS file format read-write compatibility

Avatar Tony Still
Note that Apple has reworded the function of APFS for the High Sierra release. It is now "a new file system architecture designed for all-flash Macs" (App Store description). Hopefully HD and Fusion Drive support will come along later.

The esteemed Howard Oakley reports here that dual-boot using external storage (including SSD) for HS is problematic.

I remain suspicious of Sierra's ability to use APFS volumes formatted by High Sierra. There were incompatible changes made to the formatting of APFS volumes in HS betas that post-date the release of Sierra APFS support. This isn't conclusive but I suggest proceeding with care (and not risking irreplaceable data). I'd love to know what happens if anyone tries this.

Re: APFS file format read-write compatibility

Avatar Euan Williams
Thanks, Tony, useful to have a developer's point of view, and I share your respect for Howard Oakley's long record of technical insights.

Curious thing discovered last night: trying to add to the basic Permissions (System, Wheel, Everyone) with either Admin (-istrators) or Euan (me) "Read & Write" to stop having to sign in every time I wanted to add or remove a basic data file, High Sierra came back with "You do not have sufficient privileges to do that", before the Admin or Euan "person" could be included.

The 'Get Info' panel shows a "Fetching..." notation, but only when running HiSi. Nothing obvious shows itself elsewhere.

The two Sovereign Remedies, Disc Utility (HiSi) and Safe Boot, didn't make a difference, and so far I haven't found anything on-line. Perhaps you can offer a Stillation of wisdom?

Emergency Recovery normally found with Option-Restart seems to have disappeared too

Otherwise no problems found so far. HiSigh...

Re: APFS file format read-write compatibility

Avatar Tony Still
Euan, Not sure what you were trying to do with permissions. It may well depend upon where the file is stored: HS has more restrictive access than Sierra in the continuing process of improving security. I am able to add my (non-admin) user account to the permissions of a file in Applications and give that addition R/W privilege; is that close to what you were trying?

I think the recovery modes are now converging on Cmd-R. Mr Oakley again has something on the subject (here).

Re: APFS file format read-write compatibility

Avatar Euan Williams
Thanks, Tony. Just trying to add "me the User" and /or "Admin" Read & Write privileges to the standard list of System, Wheel, and Everyone.

This has hitherto always been a very simple way to stop having endlessly to type in admin passwords when moving non-App files, and avoids having to do it for every individual file, I will try this, maybe there is an "apply to all files" option in HiSi which would render my method redundant. Full OS Security makes these changes inevitable I suppose.

There is perhaps a preference file somewhere, possibly buried deep within Terminal's satanic realms which (may) have a false setting - but I have no idea what or where.

Howard Oakley's current publishing base now added to my list of "must reads" and many thanks for your suggestions so far.

Re: APFS file format read-write compatibility

Avatar Euan Williams
Perhaps here from Oakley?
Quote:
The simple method of setting this should be through Finder’s Get Info dialog, where the Apply to enclosed items… should be able to propagate that throughout the whole of your Home folder.

If we are to believe Apple’s support note, that is not sufficient. You also need to invoke a sub-command of the diskutil command which Apple has added for this purpose, but failed to document: diskutil resetUserPermissions. Apple does not explain why this is necessary, nor what that command does.

Even then, that may return an error, whose code, -69841, is not recognised as having any meaning by OSStatus.com. The command recommended to fix that clears the user immutable (i.e. locked) flag throughout the whole of the Home folder, unlocking its entire contents, ready for diskutil resetUserPermissions to have another go at doing what the Finder couldn’t.
Unquote.

Many years ago I once tried the "apply to all enclosed files" but some kind of chaos ensued and I have avoided it since. Possibly that was because I may have applied it to the drive, not just my Home folder :)

Re: APFS file format read-write compatibility

Avatar Tony Still
I suspect your are running into macOS's increased security: I believe that Sierra's SIP (System Integrity Protection) has been further extended, further limiting what even Root can do. In Sierra, you can turn this off when booted from the recovery partition. I presume this is still available in HS - I have never had cause to look at them.

I have not had problems with Get Info/Apply to Enclosed Items but I have only used it within a particular account's files. I regard it as unwise to mess with permissions outside of my user directory as Apple generally has good reasons for the default settings and sometimes assumes that they are unmodified. The drive for greater security, which is admirable, will only increase the scope for problems with non-standard configurations.

Re: APFS file format read-write compatibility

Avatar Tony Still
A separate addition to this thread:
There are several issues currently being reported about High Sierra that are being linked to APFS (they may just be latent application errors exposed by HS or other, less concerning, HS bugs). These include Unity (game engine tools here) and Steam (games, here).

At this stage, it is unclear whether there are significant problems with HS: APFS is easy to blame for any and all problems. However, given the magnitude of the APFS change, it may be best to be prudent. I shall not be updating for a while!

Re: APFS file format read-write compatibility

Avatar Euan Williams
Thanks, Tony. While we were under the impression that High Sierra would reformat our SSDs automatically to APFS, on my partitioned SSD it has only reformatted the partition for High Sierra. The others are still HFS+.

This probably explains the odd behaviour re. permissions and the "absence" of the HiSi icon when booted into the very early Snow Leopard partition mentioned in my first post above.

With hand magnifier, hooked pipe and violin I shall pursue this, but suspect that in the end my occasionally useful Snow Leopard will have to leave the stage in favour of a full APFS reformat followed by fresh partitions. We live in interesting times!

Re: APFS file format read-write compatibility

Avatar Tony Still
For those running HS (I haven't dared to yet), there is apparently a 'Supplemental Update' available now on the App Store. This fixes a variety of problems with both the installer and HS apps, including one fairly serious security issue.

More on this here.

Re: APFS file format read-write compatibility

Avatar Euan Williams
Yes, although the list of fixes seems not to touch on permissions and partitions. Have already installed and am checking it for progress. My feeling is that the issue lies in the difference between APFS 'containers' -- which can bulge and contract according to need, and whose behaviour can also be constrained by users -- and the traditional HFS+ partitions which are always precisely constrained.

Checking this issue with David M. his High Sierra behaves quite as one would expect, and there is no issue with 'insufficient privileges', so perhaps it's an installation glitsch with my pre-update installer. Anyone downloading the Hi-Si installer from today on will find the update wrapped in the basic installer.

Here's an extract from the Ars Technica High Sierra review
<quote from p23>
"It’s important to understand the difference between creating a new APFS volume and creating a new partition. The latter function is still available if you hit the Partition button, and it still uses the same pie-chart graphics it has used since El Capitan. You can use this to create new non-APFS partitions on a disk (Boot Camp still needs to work this way), but to do so you’re taking away space from the APFS container and all the volumes it contains."

"To add new APFS partitions to a single disk, use the new Volume buttons instead. This will let you add (and delete) new APFS volumes to the APFS container—those volumes will show up in the Finder and the command line just as multiple HFS+ volumes would, but they all share the same pool of space and are therefore more flexible than before. Adding a new volume doesn’t reduce the amount of space available to existing volumes."

"If you need to make sure a certain amount of disk space is reserved for a given volume, or if you want to prevent any individual volume from taking up too much space on your disk, hit the Size Options button when you create a new APFS volume. Think of the “reserve size” as a minimum—the OS will reserve at least this much space for this volume’s use—and of the “quota size” as a maximum—this volume will take up no more than this amount of space. These sizes are defined in gigabytes by default but can also be measured in kilobytes, megabytes, terabytes, petabytes, and exabytes."
<end>

This from page 26 about APFS internal cloning (might) also have something to do with it, although I'm still working with a "pure" installation without any data transfers:
"First, if you’ve already cloned a folder in Sierra and then you upgrade to High Sierra, that file or folder still takes up space and is stored twice after you upgrade to High Sierra. The upgrade and conversion process (wisely) only touches metadata without touching actual file data, so it won’t proactively delete things that are stored twice. I cloned a 4.66GB folder on a Sierra Mac and then upgraded to High Sierra; that extra 4.66GB was still being stored on the disk. I deleted that folder and re-cloned it in High Sierra—from the user’s perspective, this looks exactly the same when you view the folders and files in Finder—and APFS worked as intended, saving me almost all of that 4.66GB of space."

"Second, the feature only works if you're copying data from one place on an APFS volume to another place on the same volume. If you copy the same folder down from a network share twice, the data will actually be copied and written twice. If you copy the files between two different APFS volumes in the same container, the data will actually be copied and written twice. Making the system work as users expect it will takes precedence over proactively comparing file data and metadata and saving space."

More from 221B Baker Street later.

Re: APFS file format read-write compatibility

Avatar Lionel Ogden
How will Time Machine and other backups work from a SSD Mac to spinning disc external HD. Will it be necessary to use a SSD external drive? The reference link in Euan's first post refers to this, but can it be as simple as it sounds?

Re: APFS file format read-write compatibility

Avatar Tony Still
Lionel - Time Machine remains as HFS+ so supports rotating HDs as well as SSD. TM is file-based, not just a binary image, so it can still backup a drive in APFS format. Presumably TM will migrate to APFS at some point but, wisely and reassuringly, I think, it has not been changed for now.

To reiterate Euan's point: the High Sierra update available from the App Store has been updated to include the Supplemental Update so those who choose to update now will see only the one update. The list of fixes is long (good) but the version number is unchanged (bad). I find it inexplicable why Apple has not called it a .1 release - it's just going to cause confusion.

I shall wait a little longer...

Re: APFS file format read-write compatibility

Avatar Mick Burrell
No confusion for AUGWessex members now though ;-)

Re: APFS file format read-write compatibility

Avatar Euan Williams
News from 221B Baker Street: after conversation with a senior Apple on-line Rep. yesterday, it seems that Apple want to persuade users to keep all their data files in their 'Home' folder as they have found inexperienced users getting in to all sorts of trouble deleting libraries and other key components of macOS.

The move to "hide" the User Library introduced some versions of macOS ago (and since eased) was an attempt to improve this situation. Those of us who prefer not to keep everything in our User folder may experience the permissions hiccup discussed which is "expected behaviour". He hopes to be able to offer a solution by the weekend after researching this.

Re: APFS file format read-write compatibility

Avatar Euan Williams
Apple Support has now confirmed that the System refusal to add an Admin User, or Administrators in the drive Get Info window is “expected behaviour”. System Integrity Protection (SIP) has been beefed up to encourage users to keep data within the user’s Home folder rather than at the top level of the drive, to discourage people inadvertently trashing important System components, and/or leaving components open to malware attacks.

Copying data to the top level of the drive now requires the user’s Admin password, while copying to the user’s folder on that drive or volume doesn’t.

The suggested links are here,
and here.

OSX Dsily has a fairly comprehensive upgrading woes and solutions here.

Re: APFS file format read-write compatibility

Avatar Derek Wright
In a multi partition system with High Sierra installed, only the partition containing the operating system and the application libraries are converted to APFS, other SSD based partitions are not converted.
 
Feed