| Knorrie | hm, after the first batch converting the fun decreases... |
| Knorrie | maybe it was too good to be true :o |
| Knorrie | it's between 5 and 10 minutes now per metadata chunk, and more writing |
| Knorrie | I modified my balance_least_used.py to work on metadata chunks instead and force convert to single, maybe that will help removing the ones with most empty space first and decrease the rewriting into DUP chunks while converting |
| Knorrie | I'll let it run for some time to see what happens. If it becomes a disaster again, I can just throw away all of it again (nice!!) |
| zerocool | well |
| zerocool | i just wanted to say, to those of you that work on btrfs |
| zerocool | that i love you |
| zerocool | for all of your hard work and dedication |
| zerocool | and kindness and bravery |
| Knorrie | \o/ |
| Knorrie | kdave: ^^ :) |
| jidar | how do I promote the snapshot I'm currently running to be the new root? I've also lost access to all of my snapshots when using snapper to rollback to an older commit |
| jidar | basically this issue : https://github.com/openSUSE/snapper/issues/159 |
| jidar | I had to mount subvol=0 to /mnt to be able to even look at the "lost" snapshot subvols |
| jidar | I guess I'm partially wrong there.. mounting the UUID of the device as subvol=.snapshots seems to have fixed it |
| Mo | demfloro: But old kernels are still able to read a zlib9 compressed btrfs? |
| Mo | Hi, could a balance operation make the btrfs faster in any way? I know https://btrfs.wiki.kernel.org/index.php/FAQ#Does_a_balance_operation_make_the_internal_B-trees_better.2Ffaster.3F |
| multicore | Mo: have you tried defragging metadata? |
| Mo | But there is also said, that balance also does "some defragmentation, but not on a file level..." |
| Mo | multicore: What means defragging metadata? I tried to avoid defrag as it would drop all reflinks of my snapshots. |
| Mo | So balance would not drop any reflink? |
| multicore | Mo: read btrfs-filesystem man page.. theres a NOTE section under defrag |
| Mo | multicore: That note about "Directory arguments without -r .."? Nothing said about metadata defragmentation. |
| multicore | "ok" |
| Mo | What does btrfs-maintenance BTRFS_BALANCE_DUSAGE="1 5 10 20 30 40 50" mean? For usage filter I only know single values or ranges. |
| multicore | i don't know what the script does but i'm assuming that it first balances with 1 and then 5 ... |
| Mo | multicore: Why, isn't that the same as only balance with dusage 5? |
| Mo | So you mean 'btrfs filesystem defragment -r /' would not break reflinks and only defrag metadata? Got that from https://wiki.archlinux.org/index.php/Btrfs#Defragmentation This is not so clear in the man page about defragment -r, "only certain internal trees...defragment files recursively in given directories" |
| multicore | "Directory arguments without -r do not defragment" ... WITHOUT |
| Mo | Thanks. |
| Mo | So this is quite safe to try, not dropping reflinks. So I try if that gives me some performance back. |
| multicore | Mo: it'll defrag metadata it's not clear to me if you'll have to do this for every subvol or not |
| multicore | Mo: if you're getting 50KB/s speeds i doubt it'll help |
| multicore | Mo: have you tested your drives that they are in OK working order? |
| multicore | Mo: and do you have quotas in use? |
| Mo | No quotas. scrub is fine. btrfs check is fine. |
| multicore | well i had this one hdd that did 2KB/s with no errors |
| multicore | no ata or smart errors |
| Mo | hdparm -tT is also fine on both. |
| multicore | you'll have to read the whole drive to make sure |
| multicore | tho if the problem is on source end you should see slow scrub speeds |
| Mo | So for defragment without -r you are not sure about subvolumes? So maybe I need that on every subvolume? I just try it on the toplevel sub. |
| multicore | Mo: yeah, i'm not sure about that |
| Mo | Hö, it returns immediately: WARNING: directory specified but recursive mode not requested: /mnt/usb/mobiledata/ WARNING: a directory passed to the defrag ioctl will not process the files |
| multicore | those warnings are because the behaviour with and without -r is confusing |
| Mo | multicore: But without -r it did not do anything. |
| multicore | how do you know that it didn't do anything? |
| Mo | multicore: Defragmenting anything in less than 1s? Having 14GB Metadata on a slow rotating USB device? |
| Mo | multicore: Ok, if so, I did defrag the metadata now. Another scrub running, and speed is fine, 14GB in 2 minutes and running... |
| Mo | Any other idea how to improve the btrfs receive speed? |
| Mo | If not I will need to defrag at least the latest snapshot, ignoring the lost reflinks and lost space. |
| multicore | Mo: defragging metadata shouldn't take too long but <1sec it most likely didn't too anything/much |
| Mo | Running that defrag without -r on my main btrfs takes longer, ok. |
| Mo | I guess the metadata on the backup only btrfs, only receiving snapshots, wasn't defragmented or was already defragmented by the balance operations I did before. |
| multicore | Mo: try doing a btrfs send to null so you'll know if the problem is in sending or receiving side |
| Mo | Nice. |
| Mo | multicore: Like btrfs send /mnt/btrfs-top-lvl/root > /dev/null ? |
| Mo | Send runs smooth with rates from 10MiB/s to 80 MiB/s, nothing weird. So the receiving backup btrfs is the bottleneck. |
| UukGoblin | is it possible to have de-duplicated files that have a header, i.e. a file stored in an archive (with 0 compression)? |
| opty | block level based? |
| UukGoblin | yeah |
| UukGoblin | but the header will not necessarily be a full block in size, unfortunately |
| UukGoblin | so there'd have to be a gap in the block |
| multicore | UukGoblin: full block sized? |
| UukGoblin | multicore, if block is, say, I dunno, 4kB, the header might be 247 bytes, so there'd have to be a 3849-byte gap in the block. Is that possible? |
| opty | i guess it depends on the archiver |
| UukGoblin | no, there will be different sized-headers for sure |
| opty | i mean the ability to align |
| UukGoblin | yeah, no, I mean, I'm asking if the fs can do it :-) |
| multicore | UukGoblin: so in effect you'd want a dedupe block smaller than 4k ? |
| UukGoblin | multicore, well, no, I don't want to de-dupe the header. Let me try to explain on an example. If I have a 1 GB .mkv file, which is then put inside a .rar with 0 compression (stored), let's say rar adds 247 bytes of a header to make a 1GB + 247b .rar file. I'd like to deduplicate all data between the .rar and .mkv |
| UukGoblin | so there'd need to be a non-duplicated block containing the 247 of the header, then possibly that gap I was talking about, followed by 1GB of blocks which are also duplicated on the .mkv |
| form | UukGoblin do you want that be automatically done by a dedupe-run, or do you just want to put the file on the fs that way? |
| Mo | multicore: After complete defragmentation of the last snapshots on the USB backup btrfs, the next send/receive is really faster. But I lost some space dues to dropped reflinks. |
| UukGoblin | form, I don't suppose there's any dedupe that can do it right now, so I'm considering writing one myself |
| UukGoblin | so let's say I'll "put the file on the fs that way" |
| Mo | I get at least around 18,4MiB/s on the UAS drive and 1,72MiB/s on the USB2.0 drive. All better than 50kb/s before. |
| form | and you have single rar files or multichunk? |
| UukGoblin | form, multichunk, actually |
| UukGoblin | (you know where they're from;-) |
| form | ok, i dont know the answer, because you would have to lay down the single rar files "padded" for alignment on the fs. dont know how to do that. but i know how to play rar'ed mkv without extracting :) |
| UukGoblin | there's rarfs on fuse that I know of, but I was wondering if I could improve that with btrfs dedupe |
| form | you can just drop the .rar on vlc. or do it with a unrar|mplayer pipe |
| UukGoblin | yeah, I'll be streaming that via dlna, but rarfs on fuse can sort of do it |
| form | ok |
| multicore | UukGoblin: so you're looking for a sliding window deduper |
| UukGoblin | multicore, yes! |
| UukGoblin | that's exactly right :-) |
| multicore | Zygo: ^ ? |
| Ke | does that rar contain the original data at multiple of 4K offset |
| Ke | if not, nothing can do this for btrfs |
| UukGoblin | Ke, no, assume it doesn't. I know how to do it if it did. |
| UukGoblin | ok, that's an OK answer :-) |
| Zygo | multicore: did somebody say sliding window deduper? ;) |
| Zygo | well, the hard limit is 4K alignment, i.e. the duplicate data must be in units of 4K and aligned to 4K blocks |
| Zygo | bees will slice and dice the extents as required to dedup any 4K-aligned blocks, but there's nothing it can do if one copy of the data is at 4K and the other is at 5K or 7K |
| Zygo | oddly enough I get a lot of random hits on git pack files |
| Zygo | I'd expect to see a 0.025% match rate on those, but it's more like 10-20% |
| Zygo | so who knows what happens in any specific test case |
| Zygo | (these git pack files I'm getting all the hits on come from different git gc runs on the same git repo, so they are storing mostly the same data, but with different deltas inserted in between leading to different file offsets. Still has an insanely high hit rate for not trying to be 4K aligned, though) |
| Zygo | one of these days I should build a tool that tells me what other files share blocks with file X, broken down into special cases like "exactly identical", "Y contains X", "X contains Y", "X and Y share blocks at the same offsets", "X and Y share blocks at different offsets", with a percentage breakdown when there's a mix of results |
| Zygo | and a special case for 'all instances of Y share a common base name, you should rename file X named "#1152562" to "comic_sans.ttf"' |
| Zygo | and then I can go through all my old ext4 lost+found directories and finally dispose of them because I'll have some idea WTF they are :-P |
| redfish | hi, btrfs-prog 4.15 on 4.15.1 (arm64) fails with btrfs: unable to add free space :-17 free-space-cache.c:828: btrfs_add_free_space: BUG_ON `ret == -EEXIST` triggered, value 1. should i report as bug? |
| redfish | this is 'btrfs check'. also it succeeds on another drive (with independent btrfs fs of also 8TB) just fine. |
| raynold | ahh it's a wonderful day |
| Zygo | Knorrie: I am somewhat impressed with your netapp device |
| Zygo | specifically, the part where you can make snapshot/clone devices and then do performance tests on them |
| Zygo | if I try that with an LVM snapshot, everything is 95% slower :-P |
| Knorrie | Zygo: it's pretty incredible stuff yes |
| linduxed | i'm in a liveUSB environment |
| linduxed | when i run "uname -a" i get linux 4.2.2-1-ARCH |
| linduxed | so it's not the newest |
| linduxed | what i'm wondering is whether there's a problem with creating a btrfs filesystem from this USB, considering i'll boot and install the latest kernel right after |
| linduxed | i know that newer kernels make use of btrfs partitions in better ways (whatever that means, i'm not sure) |
| Knorrie | that depends in the btrfs-progs version more than the kernel |
| linduxed | i'll check |
| linduxed | 4.2.1-1 |
| darkling | In general, no, there shouldn't be any problems. |
| darkling | You may have to deal with a couple of minor tweaks once you've made it. |
| linduxed | ok! |
| darkling | (Balance away any extra single chunks is the main one) |