Something is seriously wrong with FreeBSD implementation for PowerPC: it is practically unusable as of now, freezing for more than a minute at every input. Same behavior during installation and with installed system. Pattern is consistent: you have a 1–2 seconds to type something or make a selection in a menu, then everything freezes, after half a minute fans go high speed, that condition lasts for another minute, then fans go back to normal, and system unfreezes. It is not occasional, but happens literally at every window during installation and every text command in the terminal. Installation process is rather buggy and super-fragile too: I had random quits in the middle a number or times; in the beginning when installer boots, sometimes there are numerous USB-related errors, but sometimes they are skipped. I have tried it on two PowerMacs, both reasonably fast (G5 Quad 16 GB RAM SSD, G5 2.3 DC 9 GB RAM SSD) – in MacOS I use them daily for development without any pain. I also tried using HDD instead of SDD – nothing changed. Quad has ATI GPU installed, while 2.3 has NVidia, if that matters. I wanted to have FreeBSD as a second system to work on improvement of Macports support for it (so that I can test my ports not only in MacOS but also in FreeBSD). Hope someone can help to sort this issue out. P. S. I am aware PowerPC is Tier II support, but this is a matter of system simply being usable or not at all. It obviously should not be the case that it takes several minutes to type in login and password with freezes every few letters typed.
Is a -current snapshot any better? Does 12.x work any better?
(In reply to Sergey Fedorov from comment #0) How are the HDD(s) and SSD(s) connected? > … when installer boots, sometimes there are numerous USB-related errors, > but sometimes they are skipped. … In the absence of a photograph, I might assume hardware issues.
Created attachment 242607 [details] An instance of USB errors
(In reply to Graham Perrin from comment #2) I do not think so, since it happens on two different machines which otherwise work perfectly fine. However I add a pic now of USB-related issues. Please notice those are not central to the matter though. Freezing happens regardless.
Created attachment 242608 [details] USB error upon intro screen load This happened 2–3 times out of numerous install attempts. I.e. not consistent, nevertheless did happen on both machines.
I will re-upload pics, compression killed them. Sorry.
Created attachment 242609 [details] USB errors when booting (resolution fixed)
Created attachment 242610 [details] USB error upon intro screen load
Created attachment 242611 [details] Booted system on PMG5 2.3 Installed and booted system on PowerMac G5 2.3 DC. No errors on boot, but it was badly freezing even when entering login/password and every other command.
(In reply to Graham Perrin from comment #2) > How are the HDD(s) and SSD(s) connected? Native SATA bus connection. I.e. not a PCIe card, FW drive or anything else non-default. SSD were a direct replacement for HDDs. The PowerMac where I had FreeBSD installed now has HDD in upper bay which is used for FreeBSD (I moved macOS onto SSD).
(In reply to Warner Losh from comment #1) I don’t have spare DVDs around to check that. Is there a way to make install image recognizable from a FW drive? That works fine for MacOS installation, but I could not get it work for FreeBSD. USB drive might be an option too, late PowerMacs support USB booting, unofficially.
In attempting to install FreeBSD on PPC I found that utilizing the 13.2 DVD iso with a G5 1.8 Ghz single processor and Nvidia FX5200, the install was smooth and rapid, working nicely in text mode (but no success on the graphic side). Moving forward I attempted an install of 13.2 on my favored device, a G5 1.8 Ghz dual processor with an ATI 9600XT card, but the install was painfully slow... marginal graphic support (card is "supported"). So not to be thwarted, I pulled out my last remaining G5, an identical G5 dual 1.8 with the 9600 card and attempted the same install... once again, the process was incredibly slow with the same outcome observed as with the first dual. Just to confirm my suspicion, I wiped the drive on the original G5 single processor and began an install. Once again, with the single, the process was quick and smooth. So basically I found 13.2 very workable on a G5 single. However, it has been my experience and that of others that install on a multiprocessor PPC is exceedingly challenging and as noted by opening poster renders 13.2 essentially unusable on these devices. Good news: Swapping in a 9600Xt card into the G5 single has yielded fully functional FreeBSD on PPC. Bad news: 13.2 would currently seem to be a no go on the multiprocessor devices. Hopeful that this may addressed and remedied to some extent. P.S. all drives standard HDD.
Does 13.1 work on multiprocessor/multicore G5 machines? Unfortunately my FreeBSD G5 died 5 years ago, so I haven't been able to maintain the G5 at all, I've been using only the Power9. I recommend testing 12.x (latest), 13.1, and 13.0 (if 13.1 is slow). That could help narrow down when the slowdown occurred.
(In reply to Justin Hibbits from comment #13) To answer your question I've tried several release on my PowerMac G5 Quad (PowerMac 11,2 with pcie bus) using the DVD drive. Here are the results 12.4 Boots and installs just fine. 13.0 Kernel loads but painfully slow, very similar behaviour to Sergey's issue. Loads normally until "Root mount waiting for: usbus2" then fans are roaring and takes forever to get to the Install menu. 13.1 Pretty much the same as 13.0, except even worse as it simply reboots on its own from time to time. 13.2 Doesn't load at all, Black screen with fans roaring immediately. 14.0 Latest Snapshot, Same observation as 13.2. According to what I've seen, the Toolchain/ABI starting with release 13 has departed from GCC4 which makes it even more red herring. It would be really awesome to get a PPC970/MP functional 64bit release with modern toolchain ! I'd like to help testing/troubleshooting as I fear this is the latest opportunity we have until this platform fall into oblivion. Cheers, Michael
(In reply to Mitch from comment #14) Thank you very much for testing! P. S. It would be great if someone could address this and fix in the current master of FreeBSD. There are many people using PowerPCs, and it remains the only affordable/accessible option for the most end-users as well as developers. Given that other realizations of BSD apparently are fine, it is certainly not something unfixable.
Any update on the matter from PowerPC port maintainers? We have FreeBSD working on G4 but broken on G5, this is very upsetting.
FreeBSD 14 (ppc 32-bit) does not boot on G5 at all. Tries to load in OF, then quits. (I have probably not tried 13.x ppc on G5, not sure if that one works. Initial report was about ppc64 version.)
I can confirm this behaviour. I had a fine running FreeBSD 12 test disk which I nulled to try the latest stuff (14). But the 14 installer (RELEASE and STABLE from 18th of April) is so slow that it is impossible to use.
can also confirm now that 12.4 is installing and running fine on a dual g5 powermac.
(In reply to Sergey Fedorov from comment #15) Unfortunately I think the number of committers with PowerMac G5 hardware supporting FreeBSD is dwindling as the machines age. My only G5 that ran FreeBSD died back in 2018, so I'm unable to do anything significant for it. That said, I'll support as best as I can. Regarding the "USB error upon intro screen load" that's often solved by unplugging/plugging the device (happens all the time with one of my FTDI debuggers), not a problem with FreeBSD itself. The "Root mount waiting for usbus2", unless you have root on USB you can try setting the hw.usb.no_boot_wait tunable; it looks like usbus2 is having enumeration problems (just a guess). For 13.x performance, you can try testing a commit before, and after, c583b02587 (PowerMac timebase sync for G4), which should be a nop for G5, but might not be. Other possibilities are pmap related (superpage support was added in the 13 timeframe, I believe).
If anyone can build a kernel without c583b02587 to test I can try that. Since 12.4 has not even working packages anymore I can't do anything.
(In reply to Denis Ahrens from comment #21) I can try to cross compile a 14 PPC64 Kernel without c583b02587 on my x86_64 server if I can find instructions on how to achieve this. I also have a functional PowerMac 11,2 Quad to test.
(In reply to Mitch from comment #22) You can do something like: git checkout main git checkout -b tracking-branch git revert $HASH make kernel-toolchain TARGET_ARCH=powerpc64 make buildkernel KERNCONF=GENERIC64 TARGET_ARCH=powerpc64 will build it. you can install it to the destdir fo your choice: mkdir $HOME/powerpc-test-kernel make installkernel KERNCONF=GENERIC64 TARGET_ARCH=powerpc64 NO_ROOT=t
(In reply to Justin Hibbits from comment #20) If someone could build the image to try (or if installing from a FireWire drive works, then I can clone DVD image there and just replace the kernel, but I still need the new kernel), I will definitely test it.
Hello everyone, In my attempt to build a Kernel without c583b02587, I was initially confronted to this conflict below mitch@bsd:~/freebsd-src $ git status On branch tracking-branch You are currently reverting commit c583b0258728. (fix conflicts and run "git revert --continue") (use "git revert --skip" to skip this patch) (use "git revert --abort" to cancel the revert operation) Changes to be committed: (use "git restore --staged <file>..." to unstage) modified: sys/conf/files.powerpc modified: sys/powerpc/powermac/macio.c modified: sys/powerpc/powermac/platform_powermac.c Unmerged paths: (use "git restore --staged <file>..." to unstage) (use "git add/rm <file>..." as appropriate to mark resolution) deleted by them: sys/powerpc/powermac/platform_powermac.h deleted by them: sys/powerpc/powermac/tbgpio.c I've simply removed both files mentioned above in the unmerged section and I was able to build a Kernel which I assume was the right thing to do, if not please let me know. Thanks to Warner's instructions I then cross compiled a test kernel against the main branch from the freebsd source tree which appears to be the current 15 revision. You can find it here : https://box.ochem.net/cloud/index.php/s/Q6aCBiFdWrk69rb I haven't test it myself but I assume you could simply rename it to kernel.txz and do an in place replacement for testing purposes. If needed I could also build a Stable/14 version of it. Cheers, Mitch
I tested the kernel from Mitch and sad news is that the problem is still there.
(In reply to Denis Ahrens from comment #26) Other big changes that are PPC970 relevant are in the pmap logic. If you take a look at the git log of sys/powerpc/aim/mmu_oea64.c and sys/powerpc/aim/moea64_native.c, you can try to revert (git revert, or a spiritual revert if there are conflicts) some of them and see what impacts. Some options: e2d6c417 (superpages) a79540111 (micro-optimizations of PVO-PTE logic) bc94b7009 (re-merge ISA3 HPT with moea64 native HPT) -- this one *will* have conflicts. Another shot in the dark is e44ed9d34 (atomics) This is all just a guess based on trimming through `git diff origin/releng/12.4..origin/releng/13.1 sys/powerpc`. There may be other issues, but these are the first I see as possibilities.
Created attachment 251098 [details] timebase sync hack I'm not sure my symptoms exactly match what's described here, but it seems like timebase sync may be broken on G5. ofwcons freezes for about a minute every so often, the fans go full blast at the same time because powermac_thermal is hung. The system is usually (always?) responsive over ssh while this is happening. Anyway, the attached hack seems to help in my case, as least. Only tested on the later PCIe dual and quad G5s. I'm not sure timebase sync is even supported on PCI-X powermacs (definitely not on my Xserve G5 with weird sync via I2C).
(In reply to ben from comment #28) I think your patch is in the right direction, but I think the explicit memory barriers are overkill in part. We likely need to use acq/rel semantics with the atomics, and always do a load_acq() when checking cpu_done. Something like using: atomic_add_rel_int(&cpu_done, 1); (in both cases) while (atomic_load_acq_int(&cpu_done) < mp_ncpus) ; And maybe do the same for tb_ready.
Created attachment 251202 [details] tbsync patch 1 (In reply to Justin Hibbits from comment #29) Okay. Thanks for the pointers. My machine seems happy with the attached patch. Hopefully I understood enough of atomic(9). As you suspected, it didn't work without also changing tb_ready.
Comment on attachment 251202 [details] tbsync patch 1 All the atomic_set_*()s should really be atomic_store_*(), `atomic_set` is a bit-set (OR operation). It's incorrect in the existing code, too. It really doesn't make a significant difference, since the end result is the same (cpu_done should be 0 to start with, so setting nothing doesn't change that, tb_ready should be 0 to start with, so setting 1 bit does the same thing), except in the last instance to try to clear tb_ready. In the tight loop checking if cpu_done is high enough, you don't need the acq barrier in there, because the thread fence occupying the loop does the same thing, and acq places the barrier after the operation, so in the same place anyway. Glad to see this patch overall works for you, though! I'm really surprised those barriers are needed, since they're not needed in the mpc85xx case, which this is derived from.
Created attachment 251226 [details] Kernel Trap
Comment on attachment 251226 [details] Kernel Trap That's an illegal instruction trap, which is odd. You'd need to 'x/i 0xc00000000a1a9c28' to see what the instruction is.
Comment on attachment 251226 [details] Kernel Trap Sorry, I tried to add a comment with this screenshot but it didn't quite work as expected :) So first of all reverting c583b02587 did nothing at all as I'm observing the same symptoms as Ben. I've managed to painfully install the lastest snapshot with a machine continually freezing (running okay for 15 secs then fans running full blast during which the computer is unresponsive for a minute of so). Once the install completed, I did a reboot and I was welcomed by a Fatal Kernel Trap like the one I've posted, sometimes I get to see it and others the machine reboots immediately. I then tried to take an image from the drive with my FreeBSD PC but I can't see any partitions on it (I don't remember if the install wizard defaults to UFS or ZFS). Please let me know if that would be of any use for troubleshooting. I haven't yet tried Ben's patches but if it works for him there is a chance it would work for me as we're using the same HW, I'm super happy that we're making some progress ! Also there is something rather strange I've noticed with recent Images, If I boot them while holding C (cdrom) all I get is a black screen with fans roaring immediately but if I use the option key instead and wait for the watch cursor to turn back to a regular mouse cursor I'm able to boot just fine, I'm guessing that might be caused by a bunch of OF variables. Thanks everyone for your efforts so far !
(In reply to Justin Hibbits from comment #33) Yesterday I've tried Ben's patch and my machine was way happier, unfortunately the same problem still occurs, just less often. I was able to install 14.1 and use it for a few minutes before the machine crashed while idling. Just trying to understand why we'd want to use memory barriers with the powermac_smp_timebase_sync function. Is it because PPC is using a weak memory model which would be problematic in an SMP context like explained below ? https://preshing.com/20120710/memory-barriers-are-like-source-control-operations/ https://preshing.com/20120930/weak-vs-strong-memory-models/ You mentioned that the code is derived from mpc85xx, do we have such a machine in the field which is running a modern release properly ? Also how was FreeBSD handling timebase sync before the introduction of plateform_powermac ? I'll try your suggestions and test it again.
(In reply to Mitch from comment #35) mpc85xx platforms I've personally tested with FreeBSD are the AmigaOne X5000 and AmigaOne A1222. I wrote the mpc85xx timebase sync specifically for those targets, after seeing strange hanging behavior. As for why we want to use memory barriers, we really need just execution barriers (isync), I think, because we need to make sure the timebase is correct before it's unlocked by the BSP. This can only be ensured by using a barrier between setting the timebase and declaring done (atomic_add_int() of cpu_done). But, yes, a weak memory model does mean we need more explicit syncs where on strong models they would be implicit (but pay the penalty on all accesses). We really shouldn't need any syncs for tb_ready, because it can be done lazily. The only sync we should really need is the cpu_done. Before this timebase sync we used a synchronization mechanism at AP launch time, where we simply "hoped" that they were close enough to all get the same timebase. This is obviously problematic. It was changed during the 12-CURRENT time frame. The powermac sync change was done in 2021, though.
(In reply to Justin Hibbits from comment #36) Thanks for your detailed reply, yesterday I've installed git and tried to clone the FreeBSD source tree during which the machine crashed. I've archived /var/crash here, The initial crash was self induced to make sure dumping into /var/crash would work and Vmcore.1 is the Kernel crash dump which was created after my Git clone attempt. https://box.ochem.net/cloud/index.php/s/BEcGjstARB8bz9J
(In reply to Mitch from comment #37) To see what's causing the program interrupt you'd need to examine the kernel at crash time. Though, it's possible there's a compile issue that's generating code for newer powerpc64. Just check objdump on your kernel, for 0xa5081a8 address, and see what's at that address.
I got a Power Mac G5 Model A1047 and tried the patches in attached to this bug and nothing changes, it still unstable. As information, "git checkout 6c346639ba986a285e456099bc2faf1fe17954df" seems run OK on my G5. Next commits are unstable but I don't see any reason by looking at the code changes.
(In reply to Alfredo Dal'Ava Junior from comment #39) If anyone could make an image from the said commit, I will try it on G5.
(In reply to Alfredo Dal'Ava Junior from comment #39) On which branch are you working ? I wasn't able to find commit 6c346639ba986a285e456099bc2faf1fe17954df nor what it does.
(In reply to Mitch from comment #41) This commit is from stable/13 branch, but I think it's only coincidence, if I build the next commit it freezes. Other commits like 25f4ddf, fb0d2551 and 303b7702 are working on this G5 as well. One interesting thing is it works after after a cold boot, but they freeze after a soft reboot. Anyway, here is the the ISO from 6c34663 https://1drv.ms/f/s!ArFAjrJ44JgggQrEVu5Msz8VUagv?e=AupOie
One question on this: has anyone tested a recent 32-bit powerpc on the G5? I had made some changes to the 64-bit trap code a while back, which improved performance significantly on the POWER9, maybe this was collateral damage? If 32-bit works, that reduces the test scope significantly.
(In reply to Justin Hibbits from comment #43) Not yet, but I'll test it out this week and let you know how it performs.
Just thought I'd throw my 2c in, My machine is a Dual G5 Xserve (PPC 970FX x2 @ 2.0GHz, 2GB of ram, PCI-X). I'm using a single plain old SATA HDD. I have only a VGA cable, a USB keyboard and a LAN cable connected. I've been trying to get FreeBSD going and stumbled upon this bug report. I've tried both 13.4 and 14.1 of the ppc64 version and neither of them work properly (freezing for long periods, glacial performance and sometimes USB errors). If I use "kern.smp.disabled=1" at the OK prompt it seems to make both of them boot consistently completely fine, albeit without one of the CPUs doing anything obviously. I've also tried to install the 32bit version to see what happens as suggested in here, and it dies almost immediately once the screen goes black after the boot loader finishes. When specifying boot -v on the 32bit version I get a single line which says "mmu_oea64: bpvo pool entries = 14835, bpvo pool size = 8 MB" but the entries and pool size numbers do change a bit. I left it sit on that screen for a while and nothing ever happened. I tried with smp disabled to see if that did anything but it made no difference. It just sits at that forever. Hopefully this is somewhat helpful for someone.
(In reply to Justin Hibbits from comment #43) Can you tell me what is the commit id of that trap change? I own dual core 970MP and would like to see this working. Also if you have any idea of what else could be a culprit please tell me. I can test. Thanks
(In reply to Chattrapat Sangmanee from comment #46) The relevant changes are 3f24b5056 and a6ca7519f89
So I spend whole day testing this, with very positive results minus some loader annoyances. This is late 2005 G5 pcie with dual core 970MP with 8GB ram After finished install, if i let it autoboot by itself, after loader's "Press enter to boot now or wait 10 secs" it will stuck on black screen forever with fan at max speed. But if i boot it manually with command "boot hd:,\\:tbxi" it will boot normally. Other than that it works flawlessly. It never panic not even once. Even after 100 reboots. Here is what i did: - Apply tbsync patch from comment #30 - Revert commit 3f24b5056 and a6ca7519f89 Also this is not about g5 but maybe related: On my ps3 it is working most of the time but sometimes will panic such as "Data Storage Interrupt" or something like that. I don't know how to trigger it but if it going to panic, it will happen shortly after power on when spawning new process such as while booting, login, or execute command shortly after login. If it going to work fine, it will work for days or weeks no problem at all. I looked at ps3 code and don't see anything wrong so i suspect that ppc code itself is broken somewhere. Maybe this is related or just coincidence? After applying these revert it never panic again but maybe too early to tell. That's why i must ask other people to test this iso: https://drive.google.com/file/d/1svtLlWKUW3l7fGBw_ex24BmbbAMENDw5/view
(In reply to Chattrapat Sangmanee from comment #48) Great work getting it working again. Sorry for breaking things, but can you help a little more narrowing things down? * Drop the tbsync patch from comment 30 * Revert each of the changes one at a time, maybe just one of them is the culprit. Those changes together improved performance on POWER9 with Radix pmap considerably, nearly 10%, so I'd much prefer to fix the problem than to penalize radix. As for ps3, can you get a screenshot or backtrace?
(In reply to Justin Hibbits from comment #49) I will do ps3 first, it is way more easier and faster than g5 to test. (5 min each time) Here is panic screen of generic64-debug and generic64 without debug. https://imgur.com/a/sTL7piT I will do what you ask with g5 later.
Crazy question... how can i install on my ps3? It has the right firmware, but ive struggled with bootstrap.
(In reply to Warner Losh from comment #51) Very easy, You must be on <= 3.15 firmware. Then you put this file into your MBR + FAT32 formatted usb at path /PS3/otheros/otheros.bld (CASE SENSITIVE!!). https://github.com/aomsin2526/ps3-petitboot-kexec-patched/blob/master/otheros.bld Then you go to XMB system settings and use "Install OtherOS" option it will ask to install bootloader using that usb I said earlier. Done, from now on you can enter petitboot using option System Settings -> Default System then select OtherOS to enter it at anytime. If you want to go back to ps3 OS, when power on hold power button until you hear two beeps. Then you put 14.2 iso into your usb then plug it in, petitboot should see it. Done. I recommend you to install it into usb drive instead of internal HDD through. ps3disk driver upstream is very slow (around 5-10MB/s) I did some patch on my own tree (now 30-60MB/s) but it's bit hacky and ugly so it stays in my tree. Also after install don't forget to fix /boot/etc/kboot.conf as well or petitboot will not see it. My yesterday PR on github explains why.
(In reply to Justin Hibbits from comment #49) Here is g5 test results: no tbsync, no revert = panic on boot with mtx_lock() by idle thread... https://i.imgur.com/JUI9bmm.jpeg no tbsync, revert both = work fine tbsync, no revert = work fine no tbsync, revert only a6ca7519f89 = panic on boot just like above Conclusion: 3f24b5056 is the real culprit but tbsync acts as "band-aid" hiding the true cause. I'm not sure what to do in this case, how about apply new code only on little endian mode? I'm sure everyone with modern stuff use that nowadays so it's not too bad to me.
(In reply to Chattrapat Sangmanee from comment #53) I don't think tbsync acts as a "band-aid", because the whole purpose of the tbsync is to synchronize the timebases between processors. Once they're synchronized, everything should be fine. Since everything is fine with the tbsync fix, then it's apparent that the CPUs are not, in fact, correctly synchronized.
Created attachment 258647 [details] Alternative patch to test I think this patch has the essence of the tbsync patch. Can you please test this against stock main and report?
(In reply to Justin Hibbits from comment #55) It's better than without any patch, but the system still hangs periodically. My detector for this (in case you can't hear the fans) is: # d=0; while true; do nd=$(date +%s); expr $nd - $d | egrep -v '^[12]$'; d=$nd; sleep 1; done 1742027912 84 127 127 96 The same thing also happens with my last patch using stuff from atomic(9), but it has not happened in ~12 hours using the patch with mb()s sprayed everywhere. Is it possible that something isn't right with the atomic functions?